Seeing is believing — the visual interface

Tell all the truth but tell it slant
Success in circuit lies,
Too bright for our infirm delight
The truth's superb suprise;

As lightning to the children eased
With explanation kind,
The truth must dazzle gradually
Or every man be blind.

— Emily Dickinson

At its most basic level, the act of seeing involves acquiring an enormous amount of data about the light levels in our surroundings. Our visual system is faced with the Herculean challenge of interpreting this data to construct a single coherent and consistent view of the world out of the infinite possible interpretations — and of doing so quickly. It accomplishes this task by applying a variety of rules and heuristics to eliminate various possibilities and hone in on a single choice.

While our visual system creates a relatively rich and detailed picture of the world, other animals can use rule sets that create a coarser interpretation. For example, flies only extract a subset of information when they process the images from their eyes. Their visual system doesn't seem to construct surfaces from the data, but simply calculates how fast the visual field is expanding to determine whether or not to land. Likewise, frogs don't really detect flies per se; they simply see small, moving black spots of about the right size and respond. David Marr, a neuroscientist and pioneer in the field, argued that these visual systems don't reflect the objective world, describing them as subjective, but useful. "It is extremely unlikely that the fly has any explicit representation of the visual world around him — no true conception of a surface, for example, but just a few triggers and some specifically fly-centered parameters," he wrote, concluding that such a perceptual system "does not really represent the visual world about it". By contrast, he claimed, evolution had endowed humans with a visual system that could create "a true description of what is there."

Donald Hoffman is a cognitive scientist at the University of California, Irvine, where he studies our visual system, trying to understand the rules it uses and how those rules evolved. Together with Justin Mark and Brian Marion, he took on the challenge of testing Marr's claim. In order to do this, they simulated evolutionary competition between different perceptual systems. They created a digital world with patches of territory that had varying amounts of resources in them. Creatures in this world used one of three possible perceptual systems to determine how rich a patch was. Truthful perception told the creature exactly how many resources were in a patch, but it also took the most energy. Critical realism used less energy but gave the creature only a rough idea of the richness of a patch by labelling it with one of three colors. The final perceptive system was the interface approach; like critical realism, this labelled territories with a colour, but the colour depended on how useful the patch was (i.e., its fitness contribution) rather than how many resources it had. Since fitness is context-dependent, more resources in a patch wouldn't necessarily mean a higher fitness contribution; for example, the ideal patch might have an intermediate amount of resources. The team let digital organisms with the different perceptual systems compete against each other to see which would do best. They found that, as long as there was a cost to getting information, the interface model of perception was the most successful. In other words, evolutionary pressures selected for rules to construct a picture of the world that is useful regardless of whether or not it's true.

What does it mean to perceive the world in a "useful" rather than "truthful" way? The authors suggested an analogy with the desktop metaphor familiar from computers. The icons used on computers — shapes like papers or folders — don't reflect the actual nature of files and folders, which are really just collections of bits on a hard drive. Likewise, the computer isn't really a desktop and the actual mechanics of deleting a file have nothing to do with putting it in the trash. These metaphors are wildly inaccurate representations of the "truth", but they are an extremely useful way for users to interact with the computer. They provide a comprehensible interface which we can manipulate, which is precisely how Professor Hoffman's team proposes the "interface" perceptual model mediates our interaction with the world. Natural selection, they propose, has favoured the evolution of senses that cast the world through lenses that reveal usefulness, not truth.

liner

Another way to understand this is to think about what happens when you put on sunglasses. Although the glasses give everything you see a brown (or grey) tint, you quickly adapt and disregard it. Despite the coloured tint, your mind constructs white surfaces as white, not light brown or grey, which is what a light sensor would detect (the "objective truth"). This is also why photos taken indoors often have a reddish cast. The camera faithfully records the light that strikes it (the "truth"), which may have a red/orange colour depending on the kind of bulb used; by contrast, our eyes automatically adjust under the same conditions to produce the "natural" colours we expect to see:

Research by a team of scientists in The Netherlands supports this interpretation of our visual system. They presented participants with a three-dimensional wireframe box on a screen and told them to reposition the furthest face (by moving it back and forth) until the shape looked like a cube rather than a flat surface or a long hallway. They repeated the experiment with boxes of different sizes and with the subjects seated closer to and further from the screen. If our visual system constructs an honest representation of the world, the team reasoned, people would account for how distance would affect perspective. They found precisely the opposite. The subjects seemed to disregard their distance from the screen and uniformly positioned the furthest face according to an idealized "template" of a cube. "The bottom line is that all observers prefer a template over veridicality [i.e., truth]," they wrote. "How little the true perspective influences the observer's settings is illustrated [below]."

liner

Ideas from physics support this viewpoint, since they don't provide any good reason to divide the world into the discrete objects that our perceptual system prefers. So it seems that our visual system may not have evolved to give us any more truthful a view of the world than flies or frogs get — simply a more detailed one!

Further reading
Hoffman, Donald Visual Intelligence. New York: W.W. Norton & Company: 1998.
Hoffman, D. and Singh, M. Computational evolutionary perception. Perception 41:1073-1091.
Mark, J., Marion, B., and Hoffman, D. (2010) Natural selection and veridical perceptions. Journal of Theoretical Biology 266: 504-515.
Pont, S.C, Nefs, H. T., van Doorn A. J., Wijntjes M.W.A., te Pas, S.F., de Ridder, H. and Koenderink J.J. (2012) Depth in Box Spaces. Seeing and Perceiving 25:339-349.

Image credits
The figure-ground illusion is based on an image from Wikimedia Commons by Bryan Dersken.