Abstract
Responses of neurons that integrate multiple sensory inputs are traditionally characterized in terms of a set of empirical principles. However, a simple computational framework that accounts for these empirical features of multisensory integration has not been established. We propose that divisive normalization, acting at the stage of multisensory integration, can account for many of the empirical principles of multisensory integration shown by single neurons, such as the principle of inverse effectiveness and the spatial principle. This model, which uses a simple functional operation (normalization) for which there is considerable experimental support, also accounts for the recent observation that the mathematical rule by which multisensory neurons combine their inputs changes with cue reliability. The normalization model, which makes a strong testable prediction regarding cross-modal suppression, may therefore provide a simple unifying computational account of the important features of multisensory integration by neurons.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
References
Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).
Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).
Fetsch, C.R., Turner, A.H., DeAngelis, G.C. & Angelaki, D.E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601–15612 (2009).
Stein, B.E. & Stanford, T.R. Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9, 255–266 (2008).
Meredith, M.A. & Stein, B.E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640–662 (1986).
Perrault, T.J. Jr., Vaughan, J.W., Stein, B.E. & Wallace, M.T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J. Neurophysiol. 90, 4022–4026 (2003).
Stanford, T.R., Quessy, S. & Stein, B.E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005).
Meredith, M.A., Nemitz, J.W. & Stein, B.E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229 (1987).
Meredith, M.A. & Stein, B.E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J. Neurophysiol. 75, 1843–1857 (1996).
Kadunce, D.C., Vaughan, J.W., Wallace, M.T., Benedek, G. & Stein, B.E. Mechanisms of within- and cross-modality suppression in the superior colliculus. J. Neurophysiol. 78, 2834–2847 (1997).
Morgan, M.L., DeAngelis, G.C. & Angelaki, D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008).
Duffy, C.J. MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816–1827 (1998).
Gu, Y., Watkins, P.V., Angelaki, D.E. & DeAngelis, G.C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73–85 (2006).
Gu, Y., Angelaki, D.E. & DeAngelis, G.C. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11, 1201–1210 (2008).
Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).
Heeger, D.J. Normalization of cell responses in cat striate cortex. Vis. Neurosci. 9, 181–197 (1992).
Carandini, M., Heeger, D.J. & Movshon, J.A. Linearity and normalization in simple cells of the macaque primary visual cortex. J. Neurosci. 17, 8621–8644 (1997).
Busse, L., Wade, A.R. & Carandini, M. Representation of concurrent stimuli by population activity in visual cortex. Neuron 64, 931–942 (2009).
Simoncelli, E.P. & Heeger, D.J. A model of neuronal responses in visual area MT. Vision Res. 38, 743–761 (1998).
Reynolds, J.H. & Heeger, D.J. The normalization model of attention. Neuron 61, 168–185 (2009).
Priebe, N.J. & Ferster, D. Mechanisms underlying cross-orientation suppression in cat visual cortex. Nat. Neurosci. 9, 552–561 (2006).
Abbott, L.F., Varela, J.A., Sen, K. & Nelson, S.B. Synaptic depression and cortical gain control. Science 275, 220–224 (1997).
Priebe, N.J., Mechler, F., Carandini, M. & Ferster, D. The contribution of spike threshold to the dichotomy of cortical simple and complex cells. Nat. Neurosci. 7, 1113–1122 (2004).
Perrault, T.J. Jr., Vaughan, J.W., Stein, B.E. & Wallace, M.T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575–2586 (2005).
Heeger, D.J. Half-squaring in responses of cat striate cells. Vis. Neurosci. 9, 427–443 (1992).
Alvarado, J.C., Vaughan, J.W., Stanford, T.R. & Stein, B.E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).
Meredith, M.A. & Stein, B.E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).
Magosso, E., Cuppini, C., Serino, A., Di Pellegrino, G. & Ursino, M. A theoretical study of multisensory integration in the superior colliculus by a neural network model. Neural Netw. 21, 817–829 (2008).
Ursino, M., Cuppini, C., Magosso, E., Serino, A. & di Pellegrino, G. Multisensory integration in the superior colliculus: a neural network model. J. Comput. Neurosci. 26, 55–73 (2009).
Avillac, M., Ben Hamed, S. & Duhamel, J.R. Multisensory integration in the ventral intraparietal area of the macaque monkey. J. Neurosci. 27, 1922–1932 (2007).
DeAngelis, G.C., Robson, J.G., Ohzawa, I. & Freeman, R.D. Organization of suppression in receptive fields of neurons in cat visual cortex. J. Neurophysiol. 68, 144–163 (1992).
Morrone, M.C., Burr, D.C. & Maffei, L. Functional implications of cross-orientation inhibition of cortical visual cells. I. Neurophysiological evidence. Proc. R. Soc. Lond. B Biol. Sci. 216, 335–354 (1982).
Britten, K.H. & Heuer, H.W. Spatial summation in the receptive fields of MT neurons. J. Neurosci. 19, 5074–5084 (1999).
Recanzone, G.H., Wurtz, R.H. & Schwarz, U. Responses of MT and MST neurons to one and two moving objects in the receptive field. J. Neurophysiol. 78, 2904–2915 (1997).
Gu, Y., Fetsch, C.R., Adeyemo, B., DeAngelis, G.C. & Angelaki, D.E. Decoding of MSTd population activity accounts for variations in the precision of heading perception. Neuron 66, 596–609 (2010).
Kouh, M. & Poggio, T. A canonical neural circuit for cortical nonlinear operations. Neural Comput. 20, 1427–1451 (2008).
Alvarado, J.C., Rowland, B.A., Stanford, T.R. & Stein, B.E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13–23 (2008).
Cuppini, C., Ursino, M., Magosso, E., Rowland, B.A. & Stein, B.E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).
Patton, P.E. & Anastasio, T.J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783–810 (2003).
Rowland, B.A., Stanford, T.R. & Stein, B.E. A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception 36, 1431–1443 (2007).
Wallace, M.T. & Stein, B.E. Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71, 429–432 (1994).
Jiang, W., Wallace, M.T., Jiang, H., Vaughan, J.W. & Stein, B.E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001).
Alvarado, J.C., Stanford, T.R., Rowland, B.A., Vaughan, J.W. & Stein, B.E. Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. J. Neurosci. 29, 6580–6592 (2009).
Tolhurst, D.J. & Heeger, D.J. Comparison of contrast-normalization and threshold models of the responses of simple cells in cat striate cortex. Vis. Neurosci. 14, 293–309 (1997).
Olsen, S.R., Bhandawat, V. & Wilson, R.I. Divisive normalization in olfactory population codes. Neuron 66, 287–299 (2010).
Sugihara, T., Diltz, M.D., Averbeck, B.B. & Romanski, L.M. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J. Neurosci. 26, 11138–11147 (2006).
Albrecht, D.G. & Hamilton, D.B. Striate cortex of monkey and cat: contrast response function. J. Neurophysiol. 48, 217–237 (1982).
Britten, K.H. & Newsome, W.T. Tuning bandwidths for near-threshold stimuli in area MT. J. Neurophysiol. 80, 762–770 (1998).
Heuer, H.W. & Britten, K.H. Linear responses to stochastic motion signals in area MST. J. Neurophysiol. 98, 1115–1124 (2007).
Gu, Y., DeAngelis, G.C. & Angelaki, D.E. A functional link between area MSTd and heading perception based on vestibular signals. Nat. Neurosci. 10, 1038–1047 (2007).
Acknowledgements
We would like to thank R. Jacobs, A. Pouget, J. Drugowitsch, D. Barany, A. Anzai, T. Sanada, R. Sasaki and H. Kim for helpful discussions and comments on the manuscript. This work was supported by US National Institutes of Health R01 grants EY016178 to G.C.D. and EY019087 to D.E.A.
Author information
Authors and Affiliations
Contributions
T.O. and G.C.D. conceived the original model design. T.O. performed all model simulations and data analyses. T.O., D.E.A. and G.C.D. refined the model design and its predictions. T.O., D.E.A. and G.C.D. wrote and edited the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–8 (PDF 770 kb)
Rights and permissions
About this article
Cite this article
Ohshiro, T., Angelaki, D. & DeAngelis, G. A normalization model of multisensory integration. Nat Neurosci 14, 775–782 (2011). https://doi.org/10.1038/nn.2815
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2815
This article is cited by
-
Activation of M1 cholinergic receptors in mouse somatosensory cortex enhances information processing and detection behaviour
Communications Biology (2024)
-
Divisively normalized neuronal processing of uncertain visual feedback for visuomotor learning
Communications Biology (2023)
-
Neural mechanisms for the localization of unexpected external motion
Nature Communications (2023)
-
Mammalian-brain-inspired neuromorphic motion-cognition nerve achieves cross-modal perceptual enhancement
Nature Communications (2023)
-
A calibratable sensory neuron based on epitaxial VO2 for spike-based neuromorphic multisensory system
Nature Communications (2022)