I have been following the development of virtual and augmented reality (VR and AR), writing about it here occasionally. I have a VR headset and use it regularly (I am currently playing Myst VR – very nice). I don’t use it exclusively not even for gaming, some games are better on a usual screen and some are better in VR. My only gaming experience with AR is Pokemon on my cellphone (probably like most people). It was fun for a short period but the novelty wore off fast. This is not a knock against AR, I just think the cellphone version is limited.

Now Microsoft is pushing their Mesh network on their Hololens device, which represents “mixed” reality – what’s that? The term “mixed reality” was coined in a 1994 paper about a range of technologies including VR and AR. MR, essentially exists along a spectrum including physical and virtual elements, and the ability to interact with those virtual elements and project yourself into the virtual world.

For background, VR involves total immersion, wearing a headset that occupies your entire vision so you only see the virtual world. AR, rather, overlays digital elements onto the real world, which you still see. The Pokemon version I referenced uses your cellphone’s camera to take realtime video of the world and places digital elements within them – so you only see the Pokemon critters on your screen, like looking through a lens at the virtual world. The far better version of AR would be wearing transparent glasses that display the virtual elements so you see the virtual world overlaid on top of the real world.

Mixed reality, like the Mesh network, can accommodate VR, AR, and regular screen use at the same time. We are at the very beginning of this technology with lots of people making predictions about how they will be used, and that’s what I want to talk about. Currently the Holo lens costs almost $5k, so this is just dipping into the prosumer market. The general consumer market for high-tech devices is generally considered to be sub-$1,000, so we have a ways to go. One of the primary “killer apps” for mixed reality is collaboration, so how widespread the adoption is matters. That’s why it is common now for virtual spaces to be accessible on regular flat monitors, as a bridge to MR adoption.

But let’s say the prices will come down to smartphone or iPad range where everyone will want to have one. How can MR be used, and will people actually want to use MR? This is hard to predict – harder than I think some of the proponents believe. One factor is comfort – will people want to wear something like a hololens for a long time. Playing a game for an hour is one thing, working all day wearing the device is another. Definitely there is fatigue from wearing the heavier VR headsets, and this killed early attempts at building a virtual office. Eyestrain may be a factor for some as well. Further, while standing rather than sitting is good for your health, standing all day can also be tiring. And if you are going to be sitting at your desk, why not just have a monitor in front of you?

These questions also depend somewhat on the specific applications. Let’s deal with what I think is the easiest one first – gaming. MR games are awesome, and I eagerly anticipate full AR games that you can play in the physical world (like Pokemon Go) but will full AR through a headset. I see this as additive, not replacing current platforms. There are some games I play on my phone, some on my desktop, and some in VR – and I suspect there will be some I would rather play in AR.

Closely related to this is entertainment. Mixed reality is already taking off for stage performance, even real-time. This is mixing AR and live action to view on your TV. In VR there are movie shorts (an interesting experience), and you can take virtual tours. This has not caught on yet simply because of the bootstrapping problem – there isn’t enough of an audience to warrant big budget productions. I wonder where this will go. For most of the applications I discuss, I suspect that adoption will be anywhere from niche to common – we will probably see some VR/AR entertainment, it’s more a matter of how much. For some genres MR would be intense – such as a haunted house or horror film in MR. When you watch horror on a screen, you can try to distance yourself. For many in MR it is far more visceral, and you are in it, you can only escape by taking off the headset.

Work applications is the trickier question, but I will start with perhaps the easier one – virtual conferences. We have been living in Zoom hell for the last year due to the pandemic, so many of us have lots of experience with it. I actually like virtual conferences, and even prefer them overall. It’s hard to beat the convenience and efficiency. The question is – would I rather be looking at my screen, or watching the conference while sitting in a virtual theater wearing a headset? I’d have to do it for a while to really know, which again is why these things are so hard to predict. If it is a little tiresome wearing the headset, are the advantages worth it? An MR presenter can see their audience –  I have also given lectures and workshops over Zoom, and I can tell you, this would be huge. Lecturing into the void is hard, even a virtual audience would be preferred.

A virtual presenter would also have more control over the lecture space. Rather than just showing slides or sharing a desktop, you could summon and manipulate 3D objects in the virtual lecture space. For some types of content, this could be incredibly useful. Anything that involves talking about physical objects could benefit.

For education, this might be another game-changer. Right now in the medical school where I work, students are less and less interested in didactic lectures, so we essentially eliminated them. Student now consume content through multi-media, such as podcasts, videos, and online content. When we get them physically together with instructors, it is in small workshops, not for lectures. But I can imagine that an MR lecture on a topic that would be greatly enhanced by 3D demonstrations might be worth going back into the classroom – at least if it’s a virtual classroom they can attend from home. Mainly this will be about optimizing the relationship between the media and the content.

Finally let’s turn to productivity – working in an office.  The Mesh network is being sold as a mechanism for collaboration, because several people can be together virtually and look at the same virtual objects and representations of each other. In other words – they can be in different physical spaces while occupying the same virtual space. This is doable right now, and can be a great option for people who are physically far apart. It is probably better than flying from Tokyo to New York to discuss a new project.

The question is – will it be good enough to replace driving across town for a meeting. And – will it be good enough that you won’t even have a large office space for people to gather every day to work? People will work from their home office by default, and being “at work” means putting on your MR device and being virtually present in the virtual office. I think the answer has to be – it depends, on tons of variables, mostly about the kind of work. If work does not require interacting with physical objects or people, then it is possible to work in a virtual space.

Clearly we are moving in this direction, and the Hololens and Mesh network are somewhat of a milestone on this path, but a small one, not a turning point. The tech is for early adopters, or wealthy corporations who can drop that kind of money just to see how it will work. You might argue there are potential savings, by not having to fly people around and put them up in hotels, but those savings can already be realized through virtual meetings over existing desktops, with video and high-definition monitors. The question is – does MR have sufficient advantages without significant disadvantages to make it work replacing a regular monitor? At this point it’s hard to say. Maybe for some specific applications it might, but we will essentially see over the next couple decades how this technology will be adopted.

I predict (while acknowledging how hard it is to do so) that over that timeframe we will see increased adoption of MR in the workplace, but it will not come close to replacing existing methods. It will take a more mature technology and applications for dominant adoption. For entertainment, meanwhile, it will become one more medium but not replace older media.