We mostly take our vision for granted. I am not referring to how much we appreciate having good vision, for those who do, but rather we tend to be unaware of how much of a neurological feat simple vision is. In a way, we evolved not to appreciate this – the experience of good vision evolved to be seamless, and to hide all the massive processing necessary to make it so. Neuroscientists, however, have been making a lot of progress reverse engineering how our brains process vision, uncovering new layers of complexity.

A recent study, for example, looked at the persistence of visual images in the brain given dynamic stimuli. How does the brain maintain a constant representation of an object in the world when the image of that object might be dramatically and rapidly changing? Again, we tend to think of our own vision like a video camera, passively recording what is out there. But actually vision is a complex hierarchical constructive process. Imagine, for example, driving in a car. There are other cars on the road, and there are objects on the side of the road you are passing by. The visual image falling on your retina is changing dynamically, and your brain has to keep up with all this change while maintaining a sense of stability.

How much processing power this takes becomes apparent to many people with traumatic brain injury (TBI). This can cause a disrupti0n in the connections among neurons of the brain, slowing down processing speed. People with TBI often experience disorientation when driving, moving quickly, or even turning around. Their brain cannot keep up with the rapidly changing visual input. This is a common symptom because visual processing is one of the most intense that happens in the brain, so it is the first to be affected (the ability to focus and shift attention being another).

Think about what has to happen in the brain. The raw image of a familiar object falls on your retina, and then is conveyed with slight processing to the primary visual cortex. There are then successively “deep” layers of visual cortex adding all sorts of processing, enhancing the outline, processing color, accounting for shadows, constructing a three-dimensional interpretation of the 2D image, then matching the image to a memory, enhancing the match, and assigning significance to the object. The brain also has to tweak the timing of everything, to account for its own processing speed, so that the world is in sync. If there is audio it then compares the visual and audio stream to sync them up and cross-check them. It also compares visual information with vestibular information in real time.

This much processing is literally overwhelming, so the brain, as it evolved, also incorporated efficiency mechanisms. This was the point of the current study – they looked at the different level of visual process while subjects viewed a changing image. They found that an image will persist at certain levels for several hundred milliseconds, even after the stimulus is gone. This is obviously one tiny piece of the puzzle, but can be added to others. For example, how does the brain keep track of an image as it moves through different lighting and shadows? Research suggests it does this by average images over time. This is consistent with the finding that images persist – they would have to in order to be combined with new information, creating a seamless flowing image (rather than a confusing morphing mess). For example, we don’t perceive an object as changing color when it moves through shadows, because our brain adjusts the color we perceive to account for the shadow effect. Our brains maintain stability of perception through the dynamic change.

Putting it together it seems that once the visual cortex thinks it has identified an object in the world, it doesn’t bother reconstructing the same object over and over again (which would take a lot of processing), it just maintains it, averaging it out over time, and updating the movements of the object.  This is not only more efficient, but maintains the illusion of smooth continuity. Of course, this system also lends itself to many types of optical illusions, which are always fun, partly because they break the real illusion, that our vision is a passive and objective representation of external reality. Optical illusions reveal the processing, which is why neuroscientists use them as a method of exploring visual processing.

All this also reminds me of computer processing, because the same basic demands of efficiency and smoothness of experience exist. For those who remember the old days of video conferencing, there was a constant annoyance of digital distortion and slowness of image processing. Sending video data is demanding, and overwhelmed earlier networks. Programmers came up with one hack that worked fairly well. Instead of sending the entire image of each frame, the software would only send data for those pixels that changed from one frame to the next. Therefore the image of your face would be updated as you talked, while the static background would persist. This worked really well, although if you moved too much you would smear out your background as you overwhelmed the data throughput. Live video images are much better today, both because of the fact that most people have greater upload and download speeds, but also because the video compression algorithms have continued to get more sophisticated and efficient.

Our brain’s visual processing is similar, but has had a few hundred million years of evolutionary tweaking.