Virtual reality (VR) and augmented reality (AR) applications are already here, but they still have not hit the steep part of the curve. We appear to be right at the beginning. We are about to experience the rapid adoption and experimentation with this new technology, and it will be interesting to see what applications become popular, and how people end up using these technologies. This is like predicting prior to the iPhone what smart phone apps will be popular.

For background, VR involves wearing goggles that fill your entire field of vision, so that you appear to occupy an entirely virtual world. When you physically move your head, your virtual perspective changes accordingly, so you can actually look around your virtual world.  Wearable devices like gloves are used to manipulate the virtual world.

Augmented reality is similar, but instead of immersive goggles you wear transparent glasses (like Google Glass) or use a handheld device with a camera (like Pokemon Go) which overlays virtual information onto the real world.

AR is perhaps the more interesting of the two, in that there may be more potential applications, since you are still interacting with the real world. Some enthusiasts predict that within 10 years most people may be walking around with AR goggles, overlaying information on the world, and basically replacing their smartphone for communication and access to information.

It is fun and easy to speculate about possible applications of such technology. It is more difficult to predict which applications people will actually use, and how they will use them. The human element can be hard to predict.

Mental Workload

One important factor to actual use is how easy or demanding a particular application will be. A raw measure of mental difficulty might be helping in thinking about the utility of AR. Psychologists call this mental workload, or cognitive load.

Recently researchers at Drexel University used a system called functional near-infrared spectroscopy, or fNIRS, to measure the frontal lobe activity of subjects while using AR vs a traditional smartphone app.

In the near-infrared spectrum skin and bone are relatively transparent, while hemoglobin will tend to absorb these frequencies. The technology, therefore, can be used as a measure of blood flow from which relative brain activity can be inferred.

The researchers used the technology to measure frontal lobe activity during various tasks. The fNIRS is wearable, so they can walk around with it.

The frontal lobes are involved with high level decision making and directing attention. This are highly cognitively demanding tasks, meaning that the brain has to work hard. Typically when we perform a new or complex task we need to focus a lot of our attention on the task and it has a high mental workload. With practice we actually develop pathways to make the task more automatic, and therefore less cognitively demanding. The activity shifts from the frontal lobes to deeper parts of the brain.

Also, the easier a task is then, of course, the less mental workload is required. What makes a task “easy” can be a complex question, and in fact there are user interface specialists whose sole job is to understand this very question.

Most people have had the experience of using software that is “intuitive” – which means it has a good user interface. It is obvious what buttons do, they are easy to find and identify, it is easy to figure out how to accomplish a specific task, and the screen is thoughtfully arranged, with just the needed information and nothing distracting or extraneous.

You have probably also used software that was probably designed by an engineer and was never touched by an expert in user interface. The electronic medical record system I use at work, unfortunately, fits into this category. It is amazing how much mental work it can be to accomplish a simple task when the user interface is terrible.

Minimizing cognitive load is actually a highly developed art, and one you probably do not notice most of the time. The point, actually, is to make you not notice it. It’s like watching a film from an expert director and editor – the experience can be seamless. Compare that to a B movie from a hack director that can be hard to follow and even jarring at times.

AR and Workload

So – the big question is, is AR likely to be high cognitive load or low cognitive load. This could have a dramatic impact on how it is used. In the current experiment the researchers compared AR to using a handheld device to follow directions around a college campus.

They found that the students using AR had lower cognitive load and higher situational awareness than those using Google Maps on their smartphone. This makes sense as I can imagine it would be easier to simply have direction floating in your visual field, overlaid on the physical space you are trying to navigate.

In fact this is likely to be a killer app for AR. Imagine walking in a city and trying to find a specific address, with virtual arrows pointing the way. You never have to look down – it is like someone leading you the way.

The reduced cognitive load probably comes from not having to shift your attention from the map to your surroundings.

However, the researchers also found that the AR users experienced “cognitive tunneling,” which is a fancy way of saying that they were paying attention to the task of navigating and were less aware of other aspects of their surroundings.

“We observed greater mental capacity reserves for head-mounted display users during ambulatory navigation based on behavioral and neuro-metabolic evidence. However, we also observed evidence that some of the advantages of head-mounted displays are overshadowed by their suboptimal display symbology, which can be overly attention grabbing,” said Ryan McKendrick, PhD, the study’s lead author and now a cognitive scientist at Northrop Grumman Corporation.

This is a potential problem for AR, and may limit applications. The whole point is to use AR in the real world. If it causes people to step out into traffic, that’s a serious problem. Although, that is already a problem for people texting while walking, so I doubt it will prevent people from using AR.

However, the researchers think that the problem may not be inherent to AR but result from the details of the “display symbology.” If correct this means that you could design the AR interface so that it provides the necessary information without distracting the user too much from their surroundings.

I suspect we will see intuitive and horrible user interfaces for AR applications, just like with every other application.

Conclusion

This study, of course, is just one piece of information, but it is interesting. It suggests that AR, if used properly, can provide an effective user interface for some applications that actually decreases cognitive load when compared to using a smartphone.

The study also raises a concern – that AR may be distracting in some settings.

Hopefully the standard that develops for AR interfaces will be high, and will take into consideration cognitive load and situational awareness, minimizing “cognitive tunneling.”

We’ll just have to wait and see, but probably not very long.