Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Neural correlates of reliability-based cue weighting during multisensory integration

Nature Neuroscience volume 15, pages 146–154 (2012)Cite this article

Subjects

This article has been updated

Abstract

Integration of multiple sensory cues is essential for precise and accurate perception and behavioral performance, yet the reliability of sensory signals can vary across modalities and viewing conditions. Human observers typically employ the optimal strategy of weighting each cue in proportion to its reliability, but the neural basis of this computation remains poorly understood. We trained monkeys to perform a heading discrimination task from visual and vestibular cues, varying cue reliability randomly. The monkeys appropriately placed greater weight on the more reliable cue, and population decoding of neural responses in the dorsal medial superior temporal area closely predicted behavioral cue weighting, including modest deviations from optimality. We found that the mathematical combination of visual and vestibular inputs by single neurons is generally consistent with recent theories of optimal probabilistic computation in neural circuits. These results provide direct evidence for a neural mechanism mediating a simple and widespread form of statistical inference.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Cue-conflict configuration and example behavioral session.
Figure 2: Average behavioral performance.
Figure 3: Example MSTd neuron showing a correlate of trial-by-trial cue reweighting.
Figure 4: Likelihood-based decoding approach used to simulate behavioral performance based on MSTd activity.
Figure 5: Visual-vestibular congruency and average MSTd tuning curves.
Figure 6: Population decoding results and comparison with monkey behavior.
Figure 7: Goodness-of-fit of linear weighted sum model and distribution of vestibular and visual neural weights.
Figure 8: Comparison of optimal and actual (fitted) neural weights.

Change history

  • 03 April 2013

    In the version of this article initially published, there were typographical errors in the numerators of equations (12) and (13). The terms μcomb in equation (12) and PSEvis in equation (13) were preceded by plus signs; the correct equations contain minus signs in those locations. The errors have been corrected in the HTML and PDF versions of the article.

References

  1. Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).

    Article  CAS  Google Scholar 

  2. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    Article  CAS  Google Scholar 

  3. Hillis, J.M., Watt, S.J., Landy, M.S. & Banks, M.S. Slant from texture and disparity cues: optimal cue combination. J. Vis. 4, 967–992 (2004).

    PubMed  Google Scholar 

  4. Jacobs, R.A. Optimal integration of texture and motion cues to depth. Vision Res. 39, 3621–3629 (1999).

    Article  CAS  Google Scholar 

  5. Knill, D.C. & Saunders, J.A. Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558 (2003).

    Article  Google Scholar 

  6. Landy, M.S., Maloney, L.T., Johnston, E.B. & Young, M. Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35, 389–412 (1995).

    Article  CAS  Google Scholar 

  7. van Beers, R.J., Wolpert, D.M. & Haggard, P. When feeling is more important than seeing in sensorimotor adaptation. Curr. Biol. 12, 834–837 (2002).

    Article  CAS  Google Scholar 

  8. Young, M.J., Landy, M.S. & Maloney, L.T. A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Res. 33, 2685–2696 (1993).

    Article  CAS  Google Scholar 

  9. Clark, J.J. & Yuille, A.L. Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Boston, 1990).

  10. Cochran, W.G. Problems arising in the analysis of a series of similar experiments. J. R. Stat. Soc. 4 (suppl.) 102–118 (1937).

    Google Scholar 

  11. Gu, Y., Angelaki, D.E. & DeAngelis, G.C. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11, 1201–1210 (2008).

    Article  CAS  Google Scholar 

  12. Gu, Y., DeAngelis, G.C. & Angelaki, D.E. A functional link between area MSTd and heading perception based on vestibular signals. Nat. Neurosci. 10, 1038–1047 (2007).

    Article  CAS  Google Scholar 

  13. Fetsch, C.R., Turner, A.H., DeAngelis, G.C. & Angelaki, D.E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601–15612 (2009).

    Article  CAS  Google Scholar 

  14. Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).

    Article  CAS  Google Scholar 

  15. Landy, M.S., Banks, M.S. & Knill, D.C. Ideal-observer models of cue integration. in Sensory Cue Integration (eds. Trommershäuser, J., Kording, K.P. & Landy, M.S.) 5–29 (Oxford University Press, New York, 2011).

  16. Knill, D.C. & Richards, W. Perception as Bayesian Inference (Cambridge University Press, New York, 1996).

  17. Gu, Y., Watkins, P.V., Angelaki, D.E. & DeAngelis, G.C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73–85 (2006).

    Article  CAS  Google Scholar 

  18. Battaglia, P.W., Jacobs, R.A. & Aslin, R.N. Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391–1397 (2003).

    Article  Google Scholar 

  19. Rosas, P., Wagemans, J., Ernst, M.O. & Wichmann, F.A. Texture and haptic cues in slant discrimination: reliability-based cue weighting without statistically optimal cue combination. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 22, 801–809 (2005).

    Article  Google Scholar 

  20. Duffy, C.J. MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816–1827 (1998).

    Article  CAS  Google Scholar 

  21. Takahashi, K. et al. Multimodal coding of three-dimensional rotation and translation in area MSTd: comparison of visual and vestibular selectivity. J. Neurosci. 27, 9742–9756 (2007).

    Article  CAS  Google Scholar 

  22. Dayan, P. & Abbott, L.F. Theoretical Neuroscience (MIT press, Cambridge, Massachusetts, 2001).

  23. Foldiak, P. The 'ideal homunculus': statistical inference from neural population responses. in Computation and Neural Systems (eds. F.H. Eeckman & J.M. Bower) 55–60 (Kluwer Academic Publishers, Norwell, Massachusetts, 1993).

  24. Sanger, T.D. Probability density estimation for the interpretation of neural population codes. J. Neurophysiol. 76, 2790–2793 (1996).

    Article  CAS  Google Scholar 

  25. Morgan, M.L., DeAngelis, G.C. & Angelaki, D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008).

    Article  CAS  Google Scholar 

  26. Britten, K.H., Shadlen, M.N., Newsome, W.T. & Movshon, J.A. Responses of neurons in macaque MT to stochastic motion signals. Vis. Neurosci. 10, 1157–1169 (1993).

    Article  CAS  Google Scholar 

  27. Heuer, H.W. & Britten, K.H. Linear responses to stochastic motion signals in area MST. J. Neurophysiol. 98, 1115–1124 (2007).

    Article  Google Scholar 

  28. Ohshiro, T., Angelaki, D.E. & DeAngelis, G.C. A normalization model of multisensory integration. Nat. Neurosci. 14, 775–782 (2011).

    Article  CAS  Google Scholar 

  29. Knill, D.C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004).

    Article  CAS  Google Scholar 

  30. Stein, B.E. & Meredith, M.A. The Merging of the Senses (MIT Press, Cambridge, Massachusetts, 1993).

  31. Bremmer, F., Klam, F., Duhamel, J.R., Ben Hamed, S. & Graf, W. Visual-vestibular interactive responses in the macaque ventral intraparietal area (VIP). Eur. J. Neurosci. 16, 1569–1586 (2002).

    Article  Google Scholar 

  32. Chen, A., DeAngelis, G.C. & Angelaki, D.E. Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex. J. Neurosci. 31, 12036–12052 (2011).

    Article  CAS  Google Scholar 

  33. Fukushima, K. Corticovestibular interactions: anatomy, electrophysiology and functional considerations. Exp. Brain Res. 117, 1–16 (1997).

    Article  CAS  Google Scholar 

  34. Xiao, Q., Barborica, A. & Ferrera, V.P. Radial motion bias in macaque frontal eye field. Vis. Neurosci. 23, 49–60 (2006).

    Article  Google Scholar 

  35. Lewis, J.W. & Van Essen, D.C. Corticocortical connections of visual, sensorimotor and multimodal processing areas in the parietal lobe of the macaque monkey. J. Comp. Neurol. 428, 112–137 (2000).

    Article  CAS  Google Scholar 

  36. Stanton, G.B., Bruce, C.J. & Goldberg, M.E. Topography of projections to posterior cortical areas from the macaque frontal eye fields. J. Comp. Neurol. 353, 291–305 (1995).

    Article  CAS  Google Scholar 

  37. Felleman, D.J. & Van Essen, D.C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991).

    Article  CAS  Google Scholar 

  38. Maunsell, J.H. & van Essen, D.C. The connections of the middle temporal visual area (MT) and their relationship to a cortical hierarchy in the macaque monkey. J. Neurosci. 3, 2563–2586 (1983).

    Article  CAS  Google Scholar 

  39. Lappe, M., Bremmer, F., Pekel, M., Thiele, A. & Hoffmann, K.P. Optic flow processing in monkey STS: a theoretical and experimental approach. J. Neurosci. 16, 6265–6285 (1996).

    Article  CAS  Google Scholar 

  40. Fukushima, K. Extraction of visual motion and optic flow. Neural Netw. 21, 774–785 (2008).

    Article  Google Scholar 

  41. Perrone, J.A. & Stone, L.S. Emulating the visual receptive-field properties of MST neurons with a template model of heading estimation. J. Neurosci. 18, 5958–5975 (1998).

    Article  CAS  Google Scholar 

  42. Knill, D.C. Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J. Vis. 7, 1–24 (2007).

    PubMed  Google Scholar 

  43. Cheng, K., Shettleworth, S.J., Huttenlocher, J. & Rieser, J.J. Bayesian integration of spatial information. Psychol. Bull. 133, 625–637 (2007).

    Article  Google Scholar 

  44. Körding, K.P. et al. Causal inference in multisensory perception. PLoS ONE 2, e943 (2007).

    Article  Google Scholar 

  45. Komatsu, H. & Wurtz, R.H. Relation of cortical areas MT and MST to pursuit eye movements. I. Localization and visual properties of neurons. J. Neurophysiol. 60, 580–603 (1988).

    Article  CAS  Google Scholar 

  46. Tanaka, K. & Saito, H. Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clustered in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neurophysiol. 62, 626–641 (1989).

    Article  CAS  Google Scholar 

  47. Fetsch, C.R. et al. Spatiotemporal properties of vestibular responses in area MSTd. J. Neurophysiol. 104, 1506–1522 (2010).

    Article  Google Scholar 

  48. Wichmann, F.A. & Hill, N.J. The psychometric function. I. Fitting, sampling, and goodness of fit. Percept. Psychophys. 63, 1293–1313 (2001).

    Article  CAS  Google Scholar 

  49. Jazayeri, M. & Movshon, J.A. Optimal representation of sensory information by neural populations. Nat. Neurosci. 9, 690–696 (2006).

    Article  CAS  Google Scholar 

  50. Gu, Y., Fetsch, C.R., Adeyemo, B., DeAngelis, G.C. & Angelaki, D.E. Decoding of MSTd population activity accounts for variations in the precision of heading perception. Neuron 66, 596–609 (2010).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank A. Turner, J. Arand, H. Schoknecht, B. Adeyemo and J. Lin for animal care and technical help, and V. Rao, S. Nelson and M. Jazayeri for discussions and comments on the manuscript. We are indebted to Y. Guan and S. Liu for generously contributing data to this project. This work was supported by grants from the US National Institutes of Health (EY019087 to D.E.A. and EY016178 to G.C.D.) and a US National Institutes of Health Institutional National Research Service Award (5-T32-EY13360-07 supporting C.R.F.). A.P. was supported by grants from the National Science Foundation (BCS0446730), Multidisciplinary University Research Initiative (N00014-07-1-0937) and the James McDonnell Foundation.

Author information

Author notes

  1. Gregory C DeAngelis and Dora E Angelaki: These authors contributed equally to this work.

Authors and Affiliations

  1. Department of Anatomy and Neurobiology, Washington University School of Medicine, Saint Louis, Missouri, USA

    Christopher R Fetsch, Gregory C DeAngelis & Dora E Angelaki

  2. Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, USA

    Alexandre Pouget & Gregory C DeAngelis

  3. Department of Basic Neuroscience, University of Geneva, Geneva, Switzerland

    Alexandre Pouget

  4. Department of Neuroscience, Baylor College of Medicine, Houston, Texas, USA

    Dora E Angelaki

Authors
  1. Christopher R Fetsch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandre Pouget
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gregory C DeAngelis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dora E Angelaki
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.R.F., G.C.D. and D.E.A. conceived the study and designed the analyses. C.R.F. performed the experiments and analyzed the data. A.P. derived equations 6–8 and consulted on all theoretical aspects of the work. All of the authors wrote the paper.

Corresponding author

Correspondence to Christopher R Fetsch.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–6 and Supplementary Analysis (PDF 1508 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fetsch, C., Pouget, A., DeAngelis, G. et al. Neural correlates of reliability-based cue weighting during multisensory integration. Nat Neurosci 15, 146–154 (2012). https://doi.org/10.1038/nn.2983

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.2983

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing