What is reality? And does anybody care? These questions permeate the work of science-fiction writer Philip K. Dick, whose characters' memories and identities are frequently products of drugs and other technologies — implying that recollections, as well as appearances, should never be trusted. Dick's themes have been catnip to film-makers — Blade Runner, Total Recall and Minority Report are all based on his stories. The latest example, A Scanner Darkly, directed by Richard Linklater, will be released this July.

Life like: a filming technique in which normal footage is transformed into cartoons is being exploited by neuroscientists to assess how the brain responds to more or less realistic scenes. Credit: © WARNER BROS. PICTURES 2006

Neuroscientists often ask the same questions. And one of the tools that they are using to address them is ‘rotoscoping’, a filming technique Linklater uses in Scanner and some of his earlier work to blur the distinction between reality and animation. Our notions of authenticity, brain scientists are finding, depend as much on emotional and psychological plausibility as they do on physical accuracy — and the brain will swallow almost anything, provided it comes in the form of a story.

“Our ability to think of others as having minds is very promiscuous, and applies itself across a wide range of entities,” says neuroscientist Rebecca Saxe of the Massachusetts Institute of Technology. Watch a crude animation of a big square tracking a small one, for example, and the word ‘chasing’ springs to mind. Art exploits this promiscuity, creating emotional impact from strings of abstract symbols such as the lines of letters in novels, or from flickering images on screens.

Studies of brain activity using functional magnetic resonance imaging (fMRI) show that the brain distinguishes movements that might seem to have intentions from other movements. Seeing mechanical motion, such as that of a pendulum, involves a different part of the brain from seeing biological motion, even if it's only that of a cartoon arm.

But how do neuroscientists know whether our responses to such lab-based fictions reflect the real world? “I was interested in whether people attributed intentions to cartoon people in the same way that they comprehend the intentions of real people, but finding or creating videos with cartoon and real people doing the exact same thing seemed very difficult,” says neuroscientist and keen filmgoer Raymond Mar of the University of Toronto in Canada. Mar struck lucky; he discovered that Linklater's earlier rotoscope movie, Waking Life (2001) provided just such material. In rotoscoping, conventionally shot footage is transformed into cartoonish animation using a combination of human animators and computers; Waking Life uses the technique as a way of presenting a constantly shifting world of dreams, whereas A Scanner Darkly uses it to evoke the estranged world of drug use. In the extras on the first movie's DVD, Mar found clips of the video footage that had been transformed into animation, ranging from mellow scenes of pillow talk between Julie Delpy and Ethan Hawke to a man tipping petrol over himself.

Fact or fiction?

Mar and his colleagues showed both cartoon and video clips to subjects in an MRI scanner. They found that two areas of the cortex previously associated with detecting intention, the superior temporal sulcus and the temporal parietal junction (TPJ) fired more strongly in response to the video footage than to the cartoons1. Cartoon sequences, on the other hand, produced more activity in an area called the bilateral orbitofrontal cortex, which responds to rewarding stimuli. Mar's team speculate that this region may have been more tickled by the trippy cartoon footage than the more mundane video; it's certainly the case that watching the rotoscoped dreamscapes is a peculiarly rich experience.

“It's an elegant but challenging study,” says Saxe. “It's hard to imagine a more minimal contrast that more effectively manipulates this dimension. But the findings are mysterious.” The mystery is what the patterns of brain activity in the two treatments mean. It may be that the lower activity in the TPJ stimulated by the cartoons reflects the fact that the animations are easier to process into ideas about intention than the ‘real life’ footage. Mar, however, plumps for an alternative, although not necessarily contradictory explanation — that the more detailed video footage contains more cues of intention, ticking more of the brain's boxes. His next plan is to show subjects both a real, live hand poking into the scanner and a video of the same event, and see how their brains respond.

Mind twisters: cartoon animation and real video footage seem to activate different brain areas. Credit: © WARNER BROS. PICTURES 2006

“It suggests a template-matching mechanism,” adds neuroscientist Kevin Pelphrey of Duke University in Durham, North Carolina. “It may be that the more realistic stimuli portray more intentionality, which these brain regions prefer.” But more realistic doesn't necessarily just equal more convincing. The best graphics and robots risk toppling into what, in 1970, the Japanese roboticist Masahiro Mori dubbed the uncanny valley, where their almost-but-not-quite realness becomes creepy and repellent.

People believe everything, and one must expend effort to disbelieve. Richard Gerrig

Earlier studies have indicated that the brain's mind-reading areas work harder if they believe they are perceiving a real person. Chris Frith of University College London and his colleagues found that when a person played the game known as paper, scissors, stone, another area associated with attributing intention, the anterior paracingulate cortex, fired more strongly if the subject thought they were playing a person rather than a computer — even if he or she was in fact playing a computer2. What we perceive can depend on what we believe; imagining a thing produces brain activity very similar to the genuine experience.

As well as tapping into pre-existing biases, film and animation mould the brain, says Frith. “The brain responds to cultural effects, and the conventions of realism are constantly changing,” he says. What counts as real, according to these conventions, depends on technologies of representation; as technology has changed, so has our perception of the hallmarks of reality. People once believed newsreels only if they were in black and white. Currently, the mark of authenticity — exploited, for example, in Paul Greengrass's Bloody Sunday and United 93 — is wobbly handheld footage. The coarseness of video stock, as used in The Blair Witch Project, is a similar signifier of reality.

Our understanding of the social brain is still rudimentary, Frith adds. “We know quite a lot about which regions are involved, but almost nothing about what they're actually doing, and what neuronal computations are involved.” What's needed, he says, are imaging studies than can reveal the timing of activity more precisely, showing how different brain regions interact, and theoretical analyses of how the brain solves social problems. “Ultimately, in a science-fictiony way, I'd like robots that can do theory of mind and attribute intention.” This would be beyond what even Dick imagined —in his novel Do Androids Dream of Electric Sheep?, filmed as Blade Runner, the Voight–Kampff empathy test is a way of spotting replicants' emotional deficiencies.

Easily deluded

The mental state that arises when we interact with unreality is complex. We get involved to the extent that, say, we cry when Bambi's mother dies, but not so involved that we walk out of the cinema and strike up a conversation with the nearest rabbit. Whatever the explanation is, says psychologist Richard Gerrig of the State University of New York, Stony Brook, it isn't the much-touted suspension of disbelief, because disbelief is not the default. “People believe everything, and one must expend effort to disbelieve,” says Gerrig.

The brain, it seems, has a default setting of credulity, and a keen appetite for consuming and producing stories. Narrative is a crucial tool in our efforts to understand the world and some brain areas seem specialized for processing it3. Information presented in narrative form lowers our critical faculties, and experiments show that the more deeply people become immersed in a story, the easier it is to sway their attitudes towards those advocated in that story4. This resonates with A Scanner Darkly, when an undercover cop becomes so engrossed in his ‘fictional’ identity as a drug dealer that his police persona begins to pursue his criminal one. Resisting our susceptibility to stories is a useful skill in a media- and advertising-saturated world, says Gerrig. “We need to get kids and adults to construct disbelief. Because people don't know about this tendency, it puts them at risk.”

“It's not important whether you label something as fiction or non-fiction,” Mar agrees. “The true distinction is between narrative and non-narrative expository forms that don't draw you into their world.” It also looks as if the ability to lose yourself in a fictional world might reflect your ability to navigate the genuine social world. Mar and his colleagues have found that the more time a person spends reading fiction the greater his or her empathy and social skills; for readers of expository non-fiction (such as, to pick an example at random, science journalism) the correlation is negative5. I thought it would be best to keep back that particular piece of reality until the end.