Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

A normalization model of multisensory integration

Abstract

Responses of neurons that integrate multiple sensory inputs are traditionally characterized in terms of a set of empirical principles. However, a simple computational framework that accounts for these empirical features of multisensory integration has not been established. We propose that divisive normalization, acting at the stage of multisensory integration, can account for many of the empirical principles of multisensory integration shown by single neurons, such as the principle of inverse effectiveness and the spatial principle. This model, which uses a simple functional operation (normalization) for which there is considerable experimental support, also accounts for the recent observation that the mathematical rule by which multisensory neurons combine their inputs changes with cue reliability. The normalization model, which makes a strong testable prediction regarding cross-modal suppression, may therefore provide a simple unifying computational account of the important features of multisensory integration by neurons.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Schematic illustration of the normalization model of multisensory integration.
Figure 2: Normalization accounts for the principle of the inverse effectiveness.
Figure 3: Normalization and the spatial principle of multisensory enhancement.
Figure 4: Multisensory suppression in unisensory neurons.
Figure 5: Interactions among within-modality inputs.
Figure 6: Normalization accounts for apparent changes in the multisensory combination rule with cue reliability.

References

  1. Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).

    Article  CAS  Google Scholar 

  2. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    Article  CAS  Google Scholar 

  3. Fetsch, C.R., Turner, A.H., DeAngelis, G.C. & Angelaki, D.E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601–15612 (2009).

    Article  CAS  Google Scholar 

  4. Stein, B.E. & Stanford, T.R. Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9, 255–266 (2008).

    Article  CAS  Google Scholar 

  5. Meredith, M.A. & Stein, B.E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640–662 (1986).

    Article  CAS  Google Scholar 

  6. Perrault, T.J. Jr., Vaughan, J.W., Stein, B.E. & Wallace, M.T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J. Neurophysiol. 90, 4022–4026 (2003).

    Article  Google Scholar 

  7. Stanford, T.R., Quessy, S. & Stein, B.E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005).

    Article  CAS  Google Scholar 

  8. Meredith, M.A., Nemitz, J.W. & Stein, B.E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229 (1987).

    Article  CAS  Google Scholar 

  9. Meredith, M.A. & Stein, B.E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J. Neurophysiol. 75, 1843–1857 (1996).

    Article  CAS  Google Scholar 

  10. Kadunce, D.C., Vaughan, J.W., Wallace, M.T., Benedek, G. & Stein, B.E. Mechanisms of within- and cross-modality suppression in the superior colliculus. J. Neurophysiol. 78, 2834–2847 (1997).

    Article  CAS  Google Scholar 

  11. Morgan, M.L., DeAngelis, G.C. & Angelaki, D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008).

    Article  CAS  Google Scholar 

  12. Duffy, C.J. MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816–1827 (1998).

    Article  CAS  Google Scholar 

  13. Gu, Y., Watkins, P.V., Angelaki, D.E. & DeAngelis, G.C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73–85 (2006).

    Article  CAS  Google Scholar 

  14. Gu, Y., Angelaki, D.E. & DeAngelis, G.C. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11, 1201–1210 (2008).

    Article  CAS  Google Scholar 

  15. Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).

    Article  CAS  Google Scholar 

  16. Heeger, D.J. Normalization of cell responses in cat striate cortex. Vis. Neurosci. 9, 181–197 (1992).

    Article  CAS  Google Scholar 

  17. Carandini, M., Heeger, D.J. & Movshon, J.A. Linearity and normalization in simple cells of the macaque primary visual cortex. J. Neurosci. 17, 8621–8644 (1997).

    Article  CAS  Google Scholar 

  18. Busse, L., Wade, A.R. & Carandini, M. Representation of concurrent stimuli by population activity in visual cortex. Neuron 64, 931–942 (2009).

    Article  CAS  Google Scholar 

  19. Simoncelli, E.P. & Heeger, D.J. A model of neuronal responses in visual area MT. Vision Res. 38, 743–761 (1998).

    Article  CAS  Google Scholar 

  20. Reynolds, J.H. & Heeger, D.J. The normalization model of attention. Neuron 61, 168–185 (2009).

    Article  CAS  Google Scholar 

  21. Priebe, N.J. & Ferster, D. Mechanisms underlying cross-orientation suppression in cat visual cortex. Nat. Neurosci. 9, 552–561 (2006).

    Article  CAS  Google Scholar 

  22. Abbott, L.F., Varela, J.A., Sen, K. & Nelson, S.B. Synaptic depression and cortical gain control. Science 275, 220–224 (1997).

    Article  CAS  Google Scholar 

  23. Priebe, N.J., Mechler, F., Carandini, M. & Ferster, D. The contribution of spike threshold to the dichotomy of cortical simple and complex cells. Nat. Neurosci. 7, 1113–1122 (2004).

    Article  CAS  Google Scholar 

  24. Perrault, T.J. Jr., Vaughan, J.W., Stein, B.E. & Wallace, M.T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575–2586 (2005).

    Article  Google Scholar 

  25. Heeger, D.J. Half-squaring in responses of cat striate cells. Vis. Neurosci. 9, 427–443 (1992).

    Article  CAS  Google Scholar 

  26. Alvarado, J.C., Vaughan, J.W., Stanford, T.R. & Stein, B.E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).

    Article  Google Scholar 

  27. Meredith, M.A. & Stein, B.E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).

    Article  CAS  Google Scholar 

  28. Magosso, E., Cuppini, C., Serino, A., Di Pellegrino, G. & Ursino, M. A theoretical study of multisensory integration in the superior colliculus by a neural network model. Neural Netw. 21, 817–829 (2008).

    Article  Google Scholar 

  29. Ursino, M., Cuppini, C., Magosso, E., Serino, A. & di Pellegrino, G. Multisensory integration in the superior colliculus: a neural network model. J. Comput. Neurosci. 26, 55–73 (2009).

    Article  Google Scholar 

  30. Avillac, M., Ben Hamed, S. & Duhamel, J.R. Multisensory integration in the ventral intraparietal area of the macaque monkey. J. Neurosci. 27, 1922–1932 (2007).

    Article  CAS  Google Scholar 

  31. DeAngelis, G.C., Robson, J.G., Ohzawa, I. & Freeman, R.D. Organization of suppression in receptive fields of neurons in cat visual cortex. J. Neurophysiol. 68, 144–163 (1992).

    Article  CAS  Google Scholar 

  32. Morrone, M.C., Burr, D.C. & Maffei, L. Functional implications of cross-orientation inhibition of cortical visual cells. I. Neurophysiological evidence. Proc. R. Soc. Lond. B Biol. Sci. 216, 335–354 (1982).

    Article  CAS  Google Scholar 

  33. Britten, K.H. & Heuer, H.W. Spatial summation in the receptive fields of MT neurons. J. Neurosci. 19, 5074–5084 (1999).

    Article  CAS  Google Scholar 

  34. Recanzone, G.H., Wurtz, R.H. & Schwarz, U. Responses of MT and MST neurons to one and two moving objects in the receptive field. J. Neurophysiol. 78, 2904–2915 (1997).

    Article  CAS  Google Scholar 

  35. Gu, Y., Fetsch, C.R., Adeyemo, B., DeAngelis, G.C. & Angelaki, D.E. Decoding of MSTd population activity accounts for variations in the precision of heading perception. Neuron 66, 596–609 (2010).

    Article  CAS  Google Scholar 

  36. Kouh, M. & Poggio, T. A canonical neural circuit for cortical nonlinear operations. Neural Comput. 20, 1427–1451 (2008).

    Article  Google Scholar 

  37. Alvarado, J.C., Rowland, B.A., Stanford, T.R. & Stein, B.E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13–23 (2008).

    Article  CAS  Google Scholar 

  38. Cuppini, C., Ursino, M., Magosso, E., Rowland, B.A. & Stein, B.E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).

    PubMed  PubMed Central  Google Scholar 

  39. Patton, P.E. & Anastasio, T.J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783–810 (2003).

    Article  Google Scholar 

  40. Rowland, B.A., Stanford, T.R. & Stein, B.E. A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception 36, 1431–1443 (2007).

    Article  Google Scholar 

  41. Wallace, M.T. & Stein, B.E. Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71, 429–432 (1994).

    Article  CAS  Google Scholar 

  42. Jiang, W., Wallace, M.T., Jiang, H., Vaughan, J.W. & Stein, B.E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001).

    Article  CAS  Google Scholar 

  43. Alvarado, J.C., Stanford, T.R., Rowland, B.A., Vaughan, J.W. & Stein, B.E. Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. J. Neurosci. 29, 6580–6592 (2009).

    Article  CAS  Google Scholar 

  44. Tolhurst, D.J. & Heeger, D.J. Comparison of contrast-normalization and threshold models of the responses of simple cells in cat striate cortex. Vis. Neurosci. 14, 293–309 (1997).

    Article  CAS  Google Scholar 

  45. Olsen, S.R., Bhandawat, V. & Wilson, R.I. Divisive normalization in olfactory population codes. Neuron 66, 287–299 (2010).

    Article  CAS  Google Scholar 

  46. Sugihara, T., Diltz, M.D., Averbeck, B.B. & Romanski, L.M. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J. Neurosci. 26, 11138–11147 (2006).

    Article  CAS  Google Scholar 

  47. Albrecht, D.G. & Hamilton, D.B. Striate cortex of monkey and cat: contrast response function. J. Neurophysiol. 48, 217–237 (1982).

    Article  CAS  Google Scholar 

  48. Britten, K.H. & Newsome, W.T. Tuning bandwidths for near-threshold stimuli in area MT. J. Neurophysiol. 80, 762–770 (1998).

    Article  CAS  Google Scholar 

  49. Heuer, H.W. & Britten, K.H. Linear responses to stochastic motion signals in area MST. J. Neurophysiol. 98, 1115–1124 (2007).

    Article  Google Scholar 

  50. Gu, Y., DeAngelis, G.C. & Angelaki, D.E. A functional link between area MSTd and heading perception based on vestibular signals. Nat. Neurosci. 10, 1038–1047 (2007).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We would like to thank R. Jacobs, A. Pouget, J. Drugowitsch, D. Barany, A. Anzai, T. Sanada, R. Sasaki and H. Kim for helpful discussions and comments on the manuscript. This work was supported by US National Institutes of Health R01 grants EY016178 to G.C.D. and EY019087 to D.E.A.

Author information

Authors and Affiliations

Authors

Contributions

T.O. and G.C.D. conceived the original model design. T.O. performed all model simulations and data analyses. T.O., D.E.A. and G.C.D. refined the model design and its predictions. T.O., D.E.A. and G.C.D. wrote and edited the manuscript.

Corresponding author

Correspondence to Gregory C DeAngelis.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–8 (PDF 770 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ohshiro, T., Angelaki, D. & DeAngelis, G. A normalization model of multisensory integration. Nat Neurosci 14, 775–782 (2011). https://doi.org/10.1038/nn.2815

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.2815

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing