Feb 15 2018

The Neuroscience of Virtual Reality

A couple months ago I received my first virtual reality (VR) headset, and have been experimenting with various games and apps since. (Here is my initial review.) As a neuroscientist, it is a fascinating demonstration of how our brains construct our experience of reality.

What I and everyone who has used my gear has experienced is surprise at how visceral VR can be. It’s just a big video, right, so why do our lizard brains react so strongly? The most dramatic example is an app called “The Plank Experience”. In it you take an elevator up to a high floor in a skyscraper. The door opens to reveal a plank going out over the street far below. Everyone so far is frozen at the moment the doors open and they see the chasm below them. Some can walk out onto the virtual plank, but most people hesitate and at least one person bailed and would not do it.

What is interesting is that when I stepped out onto the plank, I completely 100% knew that I was standing on the carpet in my office, totally safe and at no peril at all. However, the part of my brain that knew I was safe was in conflict with a deeper and more primitive part of my brain that was screaming, “Danger, danger.”  It took an effort of will to overcome the fear, but I could not make the fear go away.

Let me describe one other part of the VR experience and then I’ll discuss what is going on neurologically. Motion sickness has been a major challenge for VR. All of the games and apps I have used so far have an option (usually the default) where if you have to move your character in the VR world you do it by teleporting. You use the control to place an X on the floor where you want to go, and then you are instantly in that location. This type of movement does not produce any motion sickness.

However, in some games it is possible to “run” through the world. With any kind of continuous movement of my avatar, even a little, I am instantly motion sick. I essentially cannot tolerate this kind of movement. People apparently differ in this regard, but mostly as a matter of degree. VR developers have not been able to make the problem go away through technical fixes, such as resolution or visual tricks. They essentially just had to provide other ways of moving in the VR world. (Companies are still working on other options, but nothing is out yet.)

Both of these aspects of VR stem from the same neurological phenomena. You can understand them if you understand that your experience of yourself and the world are completely constructed by your brain. You don’t directly experience your sensory input. Rather, that input is filtered, altered, analyzed, compared, and then finally constructed into a seamless experience.

For many sensory constructions there is a threshold effect – your brain essentially makes choices of how to construct sensory streams, and once those choices are made, the construction snaps into place. It is often an all-or-none phenomenon.

This is most apparent with common optical illusions. One class of optical illusion derived from ambiguous stimuli. Are you looking at a rabbit or an old woman? Is the girl spinning to the left or right? Is the box pointing up or down? Your brain will construct the image one way or the other, but not both, depending on assumptions about what is foreground, what is background, orientation, and shading.

Other illusions are pictures that are right at the threshold of our brain’s ability to construct the apparent image. Snow and rocks can instantly turn into a picture of a dalmation, and once you see it, you cannot unsee it. The construction is in place.

Even more interesting is that part of this construction of your perception of reality that your brain does includes a perception that you exist, that you occupy, own, and control your various body parts, and that you are separate from the rest of the universe. These components of your experience are actively constructed, and can be disrupted.

Part of these constructions involve another phenomenon mentioned above – your brain compares the various sensory streams. It also compares them to other processes in the brain, such as your intention to move. For example, your vision indicates your relationship to your surroundings and whether or not you are moving. Vestibular sensation senses movement and orientation to gravity. These two sensory streams should be telling the same story – your brain expects to see and feel the same movement at the same time. When these two sensory streams do not match, you feel motion sick.

That is what is happening dramatically with VR. You see yourself running through the virtual landscape, but your vestibular system feels you are standing still. Teleporting gets around this because you don’t see yourself move, you are just instantly in the new location. Your brain processes smooth movement different than sudden jumps. It’s possible this is due to the fact that we don’t teleport in the real world, and so our brains would not be adapted to experience it. But also, our eyes have both smooth pursuit and sudden “saccadic” movements. Our eyes can jump from one target to the next, and when we do this we don’t become disoriented. Our brains suppress the sudden change. Perhaps our brains treat teleporting in VR like saccadic eye movements.

Turning back to the visceral experience of VR, this is likely also similar to an optical illusion. Just like you cannot unsee your brain’s construction of ambiguous stimuli, you cannot convince the subconscious part of your brain that you are not standing on a plank 40 floors above the street.

The visual system does not only construct what it thinks it is seeing, but then it also connects the resulting construction to other parts of the brain that trigger memory and also emotion. The visual stream connects directly to the limbic system, assigning emotional significance to what you see. You feel love, hunger, anger, or fear automatically based on the visual stimuli.

Further, the more sensory streams that are involved in an experience, the stronger the construction. VR systems, obviously, include sound also. So you hear the wind whistling by your ear, and that contributes to the illusion you are in danger of falling to your death. Blow a small fan in the face of someone walking the virtual plank, and the illusion becomes more powerful still.

I experienced this type of illusion in some of the newer them park rids. The Spiderman ride in Universal is essentially virtual reality, but instead of wearing goggles they use large screens built seemlessly into the environment. They use small movements of the car you are sitting in to simulate the car falling off a building and careening through the city. At one point you are faced with a villain with a large blow torch, and you are met with a blast of warm air which totally sells the illusion.

To illustrate the power of multi-sensory illusions (and the fundamental nature of our brain’s reality construction) there are experiments in which people sit at a table. One arm is on top of the table, while the other is below and covered by a sheet. There is a fake arm on top of the table where the real arm is hidden. All it takes is for someone to touch the fake arm while also touching your real but visually hidden arm – you see and feel the fake arm being touched. The alignment of these two sensory stimuli is enough to convince your brain much of the time that the fake arm is actually part of your body.

Our brains also compare our intention to move with sensory feedback (visual and proprioception) that indicates we actually have moved, and this creates the sensation that we control our body parts. The construction is also easily broken if these two streams do not match, or one is disrupted. This results in what is called “alien hand syndrome” – the sensation that your body part is not under your control.

What all this also means is that we can easily trick the brain into constructing reality so that you occupy and control a virtual avatar, and that you are living in the VR world. It will be real to you – just as real as the plank.

The visual part of VR is now above the waterline – it is good enough to convince your brain that it is real. Incremental advances will be nice, but the technology has arrived. What VR developers are working on now are the other senses. Haptic feedback means giving sensory feedback to match what the person sees. So if you touch a virtual object, the haptic glove you are wearing gives the sensation of touch to match. Others are working on ways to fool the vestibular system to sense the movement you are seeing. This would fix the motion sickness problem, but it remains to be seen if they will succeed. One company, for example, is working on a system in which you control your movement by moving your head and body. Tilt your head forward to move forward. I will be interested to see how well this works.

The bottom line of all this is that – our brains construct our experience of reality, and it is possible to hack that construction to fool the brain into believing that you are operating in a virtual reality. The experience, as primitive as it is, is already profound, and will only get more so.

16 responses so far