Dec 23 2010

The Uncanny Valley

I recently saw the movie Tron – the sequel that just came out last week. Not a bad sequel for a Disney movie – the plot and characters were a bit thin, but there was some nice eye-candy. The movie did try one thing that few have before it – it tried to accurately simulate a realistic person in CG. They needed a young Jeff Bridges so they created him in CG. It was pretty good, but just slightly off. I especially noticed it around the mouth when he spoke – it was creepy.

Despite advances in computer graphics and animation, it is not yet possible to create a convincing human. That’s partly why so many CG movies feature bugs, toys, robots, and dragons – they look seamless. But the human ability to discriminate human expression is remarkable, and so subtle that even the slightest imperfections are noticeable and tend to provoke an emotional response.

This phenomenon has been dubbed the “uncanny valley.” The term refers to a map of emotional acceptance on the vertical axis and accuracy of human simulation on the horizontal axis. As simulations appear more human we tend to accept them more, until you get close to realistic but not quite – then acceptance plummets into the uncanny valley where acceptance turns into revulsion.

This is all still a bit speculative, but accords well with subjective experience. The idea is that we recognize cartoons as fake. The big eyes and pouting expression can still evoke and emotional response – if things act like agents we tend to treat them as if they were alive. But we know that they are not human because they are cartoony. But when you get close enough to human, being a bit off provokes revulsion. The speculation is because this triggers evolved revulsion against corpses and unhealthy or disfigured people.

For this reason many modern CG movies, if they portray people, stay clear of the uncanny valley and look deliberately cartoony.

One recent exception was The Polar Express, staring Tom Hanks. This movie was smack in the middle of the uncanny valley. Many viewers also noticed a closely related aspect of the CG being slightly off – the characters often had a “dead” expression on their face. This is a specific aspect of the human ability to discriminate faces – there is something about a human face that looks alive. We know when the lights are on and somebody is home.

Recently researchers have explored this phenomenon. They took realistic dolls and then found volunteers who looked similar to the dolls. They then used computer software to morph images of the dolls and the people, so there was a continuum of images between the two (you can see a couple of examples at the link above). Then they had subjects look at the morphing video and determine the point at which the resulting image looked “alive.” They found that this turning point was about two-thirds of the way from the doll image to the human image. They also found that the eyes were the most important feature in determining “aliveness.”

Again – this fits with subjective experience. It’s interesting to look at the morphing videos – to see a person’s face slowly becomeĀ  “dead” and mask-like.

The deeper context of all of this is that our brains are hardwired to pay close attention to what is an agent (an actor in the world capable of motivation), what is human, what is healthy, and what is alive. Our visual processing is organized on these concepts – we actually process visual information for agents along different visual pathways than for non-agents or inanimate objects. This visual processing is also tied to the emotional centers in our brain, which is why we can have powerful emotional responses to purely visual stimuli.

What I find interesting is that, according to our hardwiring, the concept of “agent” does not have 100% overlap with the concept of “alive.” It makes sense it would not completely overlap with “human” – as there can be important non-human agents in the world, like that predator who is about to eat us. But the rules for determining agency are not tied to the rules for determining what is alive. Agents are anything that seems to have a will of their own, even if they do not possess any other features of being alive. This is probably why we can emotionally assign agency to things that are clearly non-living, like toys and cartoons. This may also be why we feel glitchy software is deliberately trying to frustrate us, for example.

The even deeper neurological context here is that our brains, in many respects, are organized thematically, in terms of how they process and assign importance to information. Far from being passive recorders of information, our brains subconsciously make many choices about which information is important and what it means. Emotions are our automatic reactions to this subconscious processing. Therefore it must have been evolutionarily important for our ancestors to automatically have a feel for which things in their environment were likely to be agents (erring hugely on the side of false positives), which things were alive, and which things were tainted and should be avoided.

The reaction of revulsion to the dead-eyes of Tom Hanks in the Polar Express likely results from this hardwiring.

Finally, I wonder how long it will take before CG is advanced enough that it can represent a realistic human that can fool the average human. My sense is that we are still off in level of detail by an order of magnitude, at least. Getting the mouth to move realistically will probably be the last hurdle. Until then we are stuck in the uncanny valley.

17 responses so far