Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

The integration of motion and disparity cues to depth in dorsal visual cortex

Abstract

Humans exploit a range of visual depth cues to estimate three-dimensional structure. For example, the slant of a nearby tabletop can be judged by combining information from binocular disparity, texture and perspective. Behavioral tests show humans combine cues near-optimally, a feat that could depend on discriminating the outputs from cue-specific mechanisms or on fusing signals into a common representation. Although fusion is computationally attractive, it poses a substantial challenge, requiring the integration of quantitatively different signals. We used functional magnetic resonance imaging (fMRI) to provide evidence that dorsal visual area V3B/KO meets this challenge. Specifically, we found that fMRI responses are more discriminable when two cues (binocular disparity and relative motion) concurrently signal depth, and that information provided by one cue is diagnostic of depth indicated by the other. This suggests a cortical node important when perceiving depth, and highlights computations based on fusion in the dorsal stream.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Schematic illustrations of cue fusion and ideal observer discrimination.
Figure 2: Stimulus illustration and psychophysical results.
Figure 3: Representative flat maps showing the left and right visual regions of interest from one participant.
Figure 4: Results for the quadratic summation test.
Figure 5: Results for tests of congruency and transfer between cues.
Figure 6: fMRI decoding data from V3B/KO adjacent to results from simulations.

Similar content being viewed by others

References

  1. Dosher, B.A., Sperling, G. & Wurst, S.A. Tradeoffs between stereopsis and proximity luminance covariance as determinants of perceived 3D structure. Vision Res. 26, 973–990 (1986).

    Article  CAS  Google Scholar 

  2. Bülthoff, H.H. & Mallot, H.A. Integration of depth modules – stereo and shading. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 5, 1749–1758 (1988).

    Article  Google Scholar 

  3. Landy, M.S., Maloney, L.T., Johnston, E.B. & Young, M. Measurement and modeling of depth cue combination – in defense of weak fusion. Vision Res. 35, 389–412 (1995).

    Article  CAS  Google Scholar 

  4. Clark, J.J. & Yuille, A.L. Data Fusion for Sensory Information Processing Systems (Kluwer Academic, 1990).

  5. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    Article  CAS  Google Scholar 

  6. Knill, D.C. & Saunders, J.A. Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558 (2003).

    Article  Google Scholar 

  7. Tsutsui, K., Sakata, H., Naganuma, T. & Taira, M. Neural correlates for perception of 3D surface orientation from texture gradient. Science 298, 409–412 (2002).

    Article  CAS  Google Scholar 

  8. Nadler, J.W., Angelaki, D.E. & DeAngelis, G.C. A neural representation of depth from motion parallax in macaque visual cortex. Nature 452, 642–645 (2008).

    Article  CAS  Google Scholar 

  9. Liu, Y., Vogels, R. & Orban, G.A. Convergence of depth from texture and depth from disparity in macaque inferior temporal cortex. J. Neurosci. 24, 3795–3800 (2004).

    Article  CAS  Google Scholar 

  10. Gu, Y., Angelaki, D.E. & DeAngelis, G.C. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11, 1201–1210 (2008).

    Article  CAS  Google Scholar 

  11. Morgan, M.L., DeAngelis, G.C. & Angelaki, D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008).

    Article  CAS  Google Scholar 

  12. Rogers, B. & Graham, M. Motion parallax as an independent cue for depth perception. Perception 8, 125–134 (1979).

    Article  CAS  Google Scholar 

  13. Bradshaw, M.F. & Rogers, B.J. The interaction of binocular disparity and motion parallax in the computation of depth. Vision Res. 36, 3457–3468 (1996).

    Article  CAS  Google Scholar 

  14. Nawrot, M. & Blake, R. Neural integration of information specifying structure from stereopsis and motion. Science 244, 716–718 (1989).

    Article  CAS  Google Scholar 

  15. Poom, L. & Borjesson, E. Perceptual depth synthesis in the visual system as revealed by selective adaptation. J. Exp. Psychol. Hum. Percept. Perform. 25, 504–517 (1999).

    Article  CAS  Google Scholar 

  16. Domini, F., Caudek, C. & Tassinari, H. Stereo and motion information are not independently processed by the visual system. Vision Res. 46, 1707–1723 (2006).

    Article  Google Scholar 

  17. DeAngelis, G.C. & Newsome, W.T. Organization of disparity-selective neurons in macaque area MT. J. Neurosci. 19, 1398–1415 (1999).

    Article  CAS  Google Scholar 

  18. Preston, T.J., Li, S., Kourtzi, Z. & Welchman, A.E. Multivoxel pattern selectivity for perceptually relevant binocular disparities in the human brain. J. Neurosci. 28, 11315–11327 (2008).

    Article  CAS  Google Scholar 

  19. Nandy, A.S. & Tjan, B.S. Efficient integration across spatial frequencies for letter identification in foveal and peripheral vision. J. Vis. 8, 3 (2008).

    Article  Google Scholar 

  20. Hillis, J.M., Ernst, M.O., Banks, M.S. & Landy, M.S. Combining sensory information: mandatory fusion within, but not between, senses. Science 298, 1627–1630 (2002).

    Article  CAS  Google Scholar 

  21. Körding, K.P. et al. Causal inference in multisensory perception. PLoS ONE 2, e943 (2007).

    Article  Google Scholar 

  22. Popple, A.V., Smallman, H.S. & Findlay, J.M. The area of spatial integration for initial horizontal disparity vergence. Vision Res. 38, 319–326 (1998).

    Article  CAS  Google Scholar 

  23. Preston, T.J., Kourtzi, Z. & Welchman, A.E. Adaptive estimation of three-dimensional structure in the human brain. J. Neurosci. 29, 1688–1698 (2009).

    Article  CAS  Google Scholar 

  24. Tyler, C.W., Likova, L.T., Kontsevich, L.L. & Wade, A.R. The specificity of cortical region KO to depth structure. Neuroimage 30, 228–238 (2006).

    Article  Google Scholar 

  25. Meredith, M.A. & Stein, B.E. Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391 (1983).

    Article  CAS  Google Scholar 

  26. Avillac, M., Ben Hamed, S. & Duhamel, J.R. Multisensory integration in the ventral intraparietal area of the macaque monkey. J. Neurosci. 27, 1922–1932 (2007).

    Article  CAS  Google Scholar 

  27. Stanford, T.R., Quessy, S. & Stein, B.E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005).

    Article  CAS  Google Scholar 

  28. Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).

    Article  CAS  Google Scholar 

  29. Orban, G.A., Janssen, P. & Vogels, R. Extracting 3D structure from disparity. Trends Neurosci. 29, 466–473 (2006).

    Article  CAS  Google Scholar 

  30. Parker, A.J. Binocular depth perception and the cerebral cortex. Nat. Rev. Neurosci. 8, 379–391 (2007).

    Article  CAS  Google Scholar 

  31. Backus, B.T., Fleet, D.J., Parker, A.J. & Heeger, D.J. Human cortical activity correlates with stereoscopic depth perception. J. Neurophysiol. 86, 2054–2068 (2001).

    Article  CAS  Google Scholar 

  32. Chandrasekaran, C., Canon, V., Dahmen, J.C., Kourtzi, Z. & Welchman, A.E. Neural correlates of disparity-defined shape discrimination in the human brain. J. Neurophysiol. 97, 1553–1565 (2007).

    Article  Google Scholar 

  33. Orban, G.A., Sunaert, S., Todd, J.T., Van Hecke, P. & Marchal, G. Human cortical regions involved in extracting depth from motion. Neuron 24, 929–940 (1999).

    Article  CAS  Google Scholar 

  34. Murray, S.O., Olshausen, B.A. & Woods, D.L. Processing shape, motion and three-dimensional shape-from-motion in the human cortex. Cereb. Cortex 13, 508–516 (2003).

    Article  Google Scholar 

  35. Paradis, A.L. et al. Visual perception of motion and 3-D structure from motion: an fMRI study. Cereb. Cortex 10, 772–783 (2000).

    Article  CAS  Google Scholar 

  36. Sereno, M.E., Trinath, T., Augath, M. & Logothetis, N.K. Three-dimensional shape representation in monkey cortex. Neuron 33, 635–652 (2002).

    Article  CAS  Google Scholar 

  37. Durand, J.B. et al. Anterior regions of monkey parietal cortex process visual 3D shape. Neuron 55, 493–505 (2007).

    Article  CAS  Google Scholar 

  38. Peuskens, H. et al. Attention to 3-D shape, 3-D motion, and texture in 3-D structure from motion displays. J. Cogn. Neurosci. 16, 665–682 (2004).

    Article  Google Scholar 

  39. Orban, G.A. et al. Similarities and differences in motion processing between the human and macaque brain: evidence from fMRI. Neuropsychologia 41, 1757–1768 (2003).

    Article  Google Scholar 

  40. Shmuel, A., Chaimow, D., Raddatz, G., Ugurbil, K. & Yacoub, E. Mechanisms underlying decoding at 7 T: ocular dominance columns, broad structures, and macroscopic blood vessels in V1 convey information on the stimulated eye. Neuroimage 49, 1957–1964 (2010).

    Article  Google Scholar 

  41. Kriegeskorte, N., Cusack, R. & Bandettini, P. How does an fMRI voxel sample the neuronal activity pattern: compact-kernel or complex spatiotemporal filter? Neuroimage 49, 1965–1976 (2010).

    Article  Google Scholar 

  42. Op de Beeck, H.P. Against hyperacuity in brain reading: spatial smoothing does not hurt multivariate fMRI analyses? Neuroimage 49, 1943–1948 (2010).

    Article  Google Scholar 

  43. Freeman, J., Brouwer, G.J., Heeger, D.J. & Merriam, E.P. Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804 (2011).

    Article  CAS  Google Scholar 

  44. Welchman, A.E., Deubelius, A., Conrad, V., Bülthoff, H.H. & Kourtzi, Z. 3D shape perception from combined depth cues in human visual cortex. Nat. Neurosci. 8, 820–827 (2005).

    Article  CAS  Google Scholar 

  45. Tjan, B.S., Lestou, V. & Kourtzi, Z. Uncertainty and invariance in the human visual cortex. J. Neurophysiol. 96, 1556–1568 (2006).

    Article  Google Scholar 

  46. Dupont, P. et al. The kinetic occipital region in human visual cortex. Cereb. Cortex 7, 283–292 (1997).

    Article  CAS  Google Scholar 

  47. Serences, J.T. & Boynton, G.M. The representation of behavioral choice for motion in human visual cortex. J. Neurosci. 27, 12893–12899 (2007).

    Article  CAS  Google Scholar 

  48. De Martino, F. et al. Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns. Neuroimage 43, 44–58 (2008).

    Article  Google Scholar 

  49. Kamitani, Y. & Tong, F. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685 (2005).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank B. Tjan, R. Fleming and A. Glennerster for discussions on the project. The work was supported by fellowships to A.E.W. from the Wellcome Trust (095183/Z/10/Z) and Biotechnology and Biological Sciences Research Council (C520620) and to H.B. from the Japan Society for the Promotion of Science (H22,290).

Author information

Authors and Affiliations

Authors

Contributions

H.B. collected data, programmed stimuli, performed the analysis, wrote the simulations and prepared the work for publication; T.J.P. collected data, programmed stimuli and performed preliminary analysis; A.M. wrote MVPA analysis tools; A.E.W. originated and designed the study, collected data, performed and guided analysis, wrote the simulations, prepared the work for publication and wrote the paper.

Corresponding author

Correspondence to Andrew E Welchman.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–7 (PDF 4828 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ban, H., Preston, T., Meeson, A. et al. The integration of motion and disparity cues to depth in dorsal visual cortex. Nat Neurosci 15, 636–643 (2012). https://doi.org/10.1038/nn.3046

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.3046

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing