The brain is able to maximize its detection and evaluation of external events by the process of multisensory integration, whereby information from different senses is synthesized and used in concert.
Studies of single neurons in animals have revealed that this process is not present at birth, and studies of multisensory neurons in the superior colliculus have provided a developmental model by which the maturation of these neurons can be understood.
This integrative capability gradually develops during postnatal life as the underlying neural circuit matures and as the brain acquires experience with cross-modal events.
Disruptions of an essential cortical input or the absence of experience with cross-modal events does not preclude the development of multisensory neurons in the superior colliculus. However, it does interfere with the development of a multisensory neuron's capacity to integrate its sensory inputs.
Experience also helps to craft the principles by which multisensory integration is instantiated in order to adapt the system to the environment in which it will be used. Thus, abnormal cross-modal experience can alter the principles that normally govern this process.
Although multisensory integration capability is most rapidly developed in the young brain, under the proper circumstances even a brain deprived of cross-modal experience early in life can later develop this capability, albeit less efficiently.
The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It maximizes the brain's use of the information available to it at any given moment and enhances the physiological salience of external events. Because each sense conveys a unique perspective of the external world, synthesizing information across senses affords computational benefits that cannot otherwise be achieved. Multisensory integration not only has substantial survival value but can also create unique experiences that emerge when signals from different sensory channels are bound together. However, neurons in a newborn's brain are not capable of multisensory integration, and studies in the midbrain have shown that the development of this process is not predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences that alter the underlying neural circuit in such a way that optimizes multisensory integrative capabilities for the environment in which the animal will function.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Reduced activity of parvalbumin-positive interneurons in the posterior parietal cortex causes visually dominant multisensory decisions in freely navigating mice
Molecular Brain Open Access 12 October 2022
Socially meaningful visual context either enhances or inhibits vocalisation processing in the macaque brain
Nature Communications Open Access 19 August 2022
Brain Topography Open Access 12 July 2022
Subscribe to Journal
Get full journal access for 1 year
only $6.58 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).
Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).
Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).
Shams, L., Ma, W. J. & Beierholm, U. Sound-induced flash illusion as an optimal percept. Neuroreport 16, 1923–1927 (2005).
Lewkowicz, D. J. & Lickliter, R. (eds) The Development of Intersensory Perception: Comparative Perspectives (Lawrence Erlbaum Associates, 1994).
Stein, B. (ed.) The New Handbook of Multisensory Processing (MIT Press, 2012).
Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J. Jr & Rowland, B. A. Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp. Brain Res. 198, 113–126 (2009).
Binns, K. E. & Salt, T. E. Importance of NMDA receptors for multimodal integration in the deep layers of the cat superior colliculus. J. Neurophysiol. 75, 920–930 (1996).
Wallace, M. T., Meredith, M. A. & Stein, B. E. Multisensory integration in the superior colliculus of the alert cat. J. Neurophysiol. 80, 1006–1010 (1998).
Bolognini, N., Leo, F., Passamonti, C., Stein, B. E. & Làdavas, E. Multisensory-mediated auditory localization. Perception 36, 1477–1485 (2007).
Corneil, B. D. & Munoz, D. P. The influence of auditory and visual distractors on human orienting gaze shifts. J. Neurosci. 16, 8193–8207 (1996).
Diederich, A. & Colonius, H. Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys. 66, 1388–1404 (2004).
Fetsch, C. R., Pouget, A., DeAngelis, G. C. & Angelaki, D. E. Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neurosci. 15, 146–154 (2012). This is a comprehensive review of the utility of Bayesian frameworks for studying multisensory integration.
Foxe, J. J. & Schroeder, C. E. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16, 419–423 (2005).
Frens, M. A. & Van Opstal, A. J. Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Res. Bull. 46, 211–224 (1998).
Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G. & Fendrich, R. Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J. Exp. Psychol. Hum. Percept. Perform. 20, 131–153 (1994). This early article helped to establish the relationship between multisensory integration and reaction speed.
Naumer, M. J. & Kaiser, J. (eds) Multisensory Object Perception in the Primate Brain (Springer, 2010).
Sánchez-García, C., Alsius, A., Enns, J. T. & Soto-Faraco, S. Cross-modal prediction in speech perception. PLoS ONE 6, e25198 (2011).
Senkowski, D., Talsma, D., Grigutsch, M., Herrmann, C. S. & Woldorff, M. G. Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45, 561–571 (2007).
Stein, B. E., Meredith, M. A., Huneycutt, W. S. & McDade, L. Behavioral indices of multisensory integration: orientation to visual cues is affected by auditory stimuli. J. Cogn. Neurosci. 1, 12–24 (1989). This paper showed that the physiological principles of multisensory integration that are evident in single superior colliculus neurons applies to overt orientation behaviour.
Talsma, D., Doty, T. J. & Woldorff, M. G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690 (2007).
Woods, T. M. & Recanzone, G. H. Visually induced plasticity of auditory spatial perception in macaques. Curr. Biol. 14, 1559–1564 (2004).
Zhang, M., Weisser, V. D., Stilla, R., Prather, S. C. & Sathian, K. Multisensory cortical processing of object shape and its relation to mental imagery. Cogn. Affect. Behav. Neurosci. 4, 251–259 (2004).
Goldring, J. E., Dorris, M. C., Corneil, B. D., Ballantyne, P. A. & Munoz, D. P. Combined eye-head gaze shifts to visual and auditory targets in humans. Exp. Brain Res. 111, 68–78 (1996).
Sperdin, H. F., Cappe, C., Foxe, J. J. & Murray, M. M. Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front. Integr. Neurosci. 3, 2 (2009).
Von Saldern, S. & Noppeney, U. Sensory and striatal areas integrate auditory and visual signals into behavioral benefits during motion discrimination. J. Neurosci. 33, 8841–8849 (2013).
Bremner, A. J., Lewkowicz, D. J. & Spence, C. (eds) Multisensory Development (Oxford Univ. Press, 2012).
Bizley, J. K. & King, A. J. Visual-auditory spatial processing in auditory cortical neurons. Brain Res. 1242, 24–36 (2008).
Brett-Green, B., Fifková, E., Larue, D. T., Winer, J. A. & Barth, D. S. A multisensory zone in rat parietotemporal cortex: intra- and extracellular physiology and thalamocortical connections. J. Comp. Neurol. 460, 223–237 (2003).
Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11–23 (2008).
Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).
Graziano, M., Gross, C., Taylor, C. & Moore, T. in Crossmodal Space and Crossmodal Attention (eds Spence, C. & Driver, J.) 51–67 (Oxford Univ. Press, 2004).
King, A. J. & Palmer, A. R. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60, 492–500 (1985).
Romanski, L. M. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cereb. Cortex 17 (Suppl. 1), i61–i69 (2007).
Schroeder, C. E. & Foxe, J. Multisensory contributions to low-level, 'unisensory' processing. Curr. Opin. Neurobiol. 15, 454–458 (2005). An examination of the impact of multisensory integration in cortical regions believed to be unisensory. The paper examines the evidence for multisensory information coding even within classically defined, modality-specific areas of that cortex.
Wallace, M., Ramachandran, R. & Stein, B. A revised view of sensory cortical parcellation. Proc. Natl Acad. Sci. USA 101, 2167–2172 (2004).
Winkowski, D. E. & Knudsen, E. I. Top-down control of multimodal sensitivity in the barn owl optic tectum. J. Neurosci. 27, 13279–13291 (2007).
Macaluso, E. & Driver, J. in The Handbook of Multisensory Processes (eds Calvert, G. A., Spence, C. & Stein, B. E.) 529–548 (MIT Press, 2004).
Cohen, Y. E. & Andersen, R. A. in The Handbook of Multisensory Processes (eds Calvert, G. A., Spence, C. & Stein, B. E.) 463–479 (MIT Press, 2004).
Burnett, L. R., Stein, B. E., Chaponis, D. & Wallace, M. T. Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124, 535–547 (2004).
Burnett, L. R., Stein, B. E., Perrault, T. J. Jr & Wallace, M. T. Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration. Exp. Brain Res. 179, 325–338 (2007).
Groh, J. M. & Sparks, D. L. Saccades to somatosensory targets. II. Motor convergence in primate superior colliculus. J. Neurophysiol. 75, 428–438 (1996).
Groh, J. M. & Sparks, D. L. Saccades to somatosensory targets. III. Eye-position-dependent somatosensory activity in primate superior colliculus. J. Neurophysiol. 75, 439–453 (1996).
Harrington, L. K. & Peck, C. K. Spatial disparity affects visual-auditory interactions in human sensorimotor processing. Exp. Brain Res. 122, 247–252 (1998).
Jay, M. F. & Sparks, D. L. Sensorimotor integration in the primate superior colliculus. I. Motor convergence. J. Neurophysiol. 57, 22–34 (1987).
Jay, M. F. & Sparks, D. L. Sensorimotor integration in the primate superior colliculus. II. Coordinates of auditory signals. J. Neurophysiol. 57, 35–55 (1987).
Munoz, D. P. & Wurtz, R. H. Saccade-related activity in monkey superior colliculus. I. Characteristics of burst and buildup cells. J. Neurophysiol. 73, 2313–2333 (1995).
Munoz, D. P. & Wurtz, R. H. Saccade-related activity in monkey superior colliculus. II. Spread of activity during saccades. J. Neurophysiol. 73, 2334–2348 (1995).
Lomber, S. G., Payne, B. R. & Cornwell, P. Role of the superior colliculus in analyses of space: superficial and intermediate layer contributions to visual orienting, auditory orienting, and visuospatial discriminations during unilateral and bilateral deactivations. J. Comp. Neurol. 441, 44–57 (2001).
Levine, M. S., Hull, C. D. & Buchwald, N. A. Development of motor activity in kittens. Dev. Psychobiol. 13, 357–371 (1980).
Norton, T. T. Receptive-field properties of superior colliculus cells and development of visual behavior in kittens. J. Neurophysiol. 37, 674–690 (1974).
Meredith, M. A., Wallace, M. T. & Stein, B. E. Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp. Brain Res. 88, 181–186 (1992).
Sparks, D. L. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol. Rev. 66, 118–171 (1986).
Wurtz, R. H. & Goldberg, M. E. The role of the superior colliculus in visually-evoked eye movements. Bibl. Ophthalmol. 82, 149–158 (1972).
Moschovakis, A. K. et al. An anatomical substrate for the spatiotemporal transformation. J. Neurosci. 18, 10219–10229 (1998).
Knill, D. C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004).
Patton, P. E. & Anastasio, T. J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783–810 (2003).
Rowland, B., Stanford, T. & Stein, B. A Bayesian model unifies multisensory spatial localization with the physiological properties of the superior colliculus. Exp. Brain Res. 180, 153–161 (2007).
Stein, B. E., Labos, E. & Kruger, L. Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J. Neurophysiol. 36, 667–679 (1973). This was the first demonstration of the developmental chronology of sensory representations in the multisensory superior colliculus.
Stein, B. E., Labos, E. & Kruger, L. Determinants of response latency in neurons of superior colliculus in kittens. J. Neurophysiol. 36, 680–689 (1973).
Benedetti, F. Differential formation of topographic maps on the cerebral cortex and superior colliculus of the mouse by temporally correlated tactile- tactile and tactile-visual inputs. Eur. J. Neurosci. 7, 1942–1951 (1995).
Meredith, M. A., Clemo, H. R. & Stein, B. E. Somatotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J. Comp. Neurol. 312, 353–370 (1991).
Meredith, M. A. & Stein, B. E. The visuotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J. Neurosci. 10, 3727–3742 (1990).
Middlebrooks, J. C. & Knudsen, E. I. A neural code for auditory space in the cat's superior colliculus. J. Neurosci. 4, 2621–2634 (1984).
Stein, B. E., Magalhaes-Castro, B. & Kruger, L. Superior colliculus: visuotopic-somatotopic overlap. Science 189, 224–226 (1975).
Chalupa, L. M. & Rhoades, R. W. Responses of visual, somatosensory, and auditory neurones in the golden hamster's superior colliculus. J. Physiol. 270, 595–626 (1977).
Dräger, U. C. & Hubel, D. H. Physiology of visual cells in mouse superior colliculus and correlation with somatosensory and auditory input. Nature 253, 203–204 (1975).
Dräger, U. C. & Hubel, D. H. Topography of visual and somatosensory projections to mouse superior colliculus. J. Neurophysiol. 39, 91–101 (1976).
Gordon, B. Receptive fields in deep layers of cat superior colliculus. J. Neurophysiol. 36, 157–178 (1973).
Graham, J., Pearson, H. E., Berman, N. & Murphy, E. H. Laminar organization of superior colliculus in the rabbit: a study of receptive-field properties of single units. J. Neurophysiol. 45, 915–932 (1981).
King, A. J., Schnupp, J. W. & Thompson, I. D. Signals from the superficial layers of the superior colliculus enable the development of the auditory space map in the deeper layers. J. Neurosci. 18, 9394–9408 (1998).
Knudsen, E. I. Auditory and visual maps of space in the optic tectum of the owl. J. Neurosci. 2, 1177–1194 (1982).
Stein, B. E. & Dixon, J. P. Superior colliculus cells respond to noxious stimuli. Brain Res. 158, 65–73 (1978).
Stein, B. E., Magalhães-Castro, B. & Kruger, L. Relationship between visual and tactile representations in cat superior colliculus. J. Neurophysiol. 39, 401–419 (1976).
Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).
Jiang, W., Jiang, H. & Stein, B. E. Neonatal cortical ablation disrupts multisensory development in superior colliculus. J. Neurophysiol. 95, 1380–1396 (2006).
Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444 (1997). This was the first study of the development of multisensory integration in superior colliculus neurons.
Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578–3582 (2000).
Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. Incorporating cross-modal statistics in the development and maintenance of multisensory integration. J. Neurosci. 32, 2287–2298 (2012). This study demonstrates some of the experiential prerequisites for instantiating multisensory integration capabilities in superior colliculus neurons.
Meredith, M. A. & Stein, B. E. Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391 (1983). This was the first paper to demonstrate the multisensory integration capability of single superior colliculus neurons.
Kadunce, D. C., Vaughan, J. W., Wallace, M. T. & Stein, B. E. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp. Brain Res. 139, 303–310 (2001).
Meredith, M. A., Nemitz, J. W. & Stein, B. E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229 (1987).
Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).
Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G. & Stein, B. E. Mechanisms of within- and cross-modality suppression in the superior colliculus. J. Neurophysiol. 78, 2834–2847 (1997).
Pluta, S. R., Rowland, B. A., Stanford, T. R. & Stein, B. E. Alterations to multisensory and unisensory integration by stimulus competition. J. Neurophysiol. 106, 3091–3101 (2011).
Larson, M. A. & Stein, B. E. The use of tactile and olfactory cues in neonatal orientation and localization of the nipple. Dev. Psychobiol. 17, 423–436 (1984).
Kao, C. Q., McHaffie, J. G., Meredith, M. A. & Stein, B. E. Functional development of a central visual map in cat. J. Neurophysiol. 72, 266–272 (1994).
Hubel, D. H. & Wiesel, T. N. Receptive fields of cells in striate cortex of very young, visually inexperienced kittens. J. Neurophysiol. 26, 994–1002 (1963).
Horton, J. C. & Hocking, D. R. An adult-like pattern of ocular dominance columns in striate cortex of newborn monkeys prior to visual experience. J. Neurosci. 16, 1791–1807 (1996).
Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243 (1968).
Rakic, P. Prenatal development of the visual system in rhesus monkey. Philos. Trans. R. Soc. Lond. B 278, 245–260 (1977).
Wallace, M. T., McHaffie, J. G. & Stein, B. E. Visual response properties and visuotopic representation in the newborn monkey superior colliculus. J. Neurophysiol. 78, 2732–2741 (1997).
Wallace, M. T. & Stein, B. E. Sensory organization of the superior colliculus in cat and monkey. Prog. Brain Res. 112, 301–311 (1996).
Edwards, S. B., Ginsburgh, C. L., Henkel, C. K. & Stein, B. E. Sources of subcortical projections to the superior colliculus in the cat. J. Comp. Neurol. 184, 309–329 (1979).
Huerta, M. F. & Harting, J. K. The projection from the nucleus of the posterior commissure to the superior colliculus of the cat: patch-like endings within the intermediate and deep grey layers. Brain Res. 238, 426–432 (1982).
Meredith, M. A. & Clemo, H. R. Auditory cortical projection from the anterior ectosylvian sulcus (Field AES) to the superior colliculus in the cat: an anatomical and electrophysiological study. J. Comp. Neurol. 289, 687–707 (1989).
Stein, B. E., Spencer, R. F. & Edwards, S. B. Corticotectal and corticothalamic efferent projections of SIV somatosensory cortex in cat. J. Neurophysiol. 50, 896–909 (1983).
Scannell, J. W. et al. Visual motion processing in the anterior ectosylvian sulcus of the cat. J. Neurophysiol. 76, 895–907 (1996).
Alvarado, J. C., Stanford, T. R., Rowland, B. A., Vaughan, J. W. & Stein, B. E. Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. J. Neurosci. 29, 6580–6592 (2009).
Alvarado, J. C., Stanford, T. R., Vaughan, J. W. & Stein, B. E. Cortex mediates multisensory but not unisensory integration in superior colliculus. J. Neurosci. 27, 12775–12786 (2007).
Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001). Two areas of association cortical areas were shown to have crucial roles in superior colliculus multisensory integration.
Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240–1255 (2002).
Jiang, W. & Stein, B. E. Cortex controls multisensory depression in superior colliculus. J. Neurophysiol. 90, 2123–2135 (2003).
Stein, B. E., Wallace, M. W., Stanford, T. R. & Jiang, W. Cortex governs multisensory integration in the midbrain. Neuroscientist 8, 306–314 (2002).
Stein, B. E. The development of a dialogue between cortex and midbrain to integrate multisensory information. Exp. Brain Res. 166, 305–315 (2005).
Wallace, M. T. & Stein, B. E. Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71, 429–432 (1994).
Clemo, H. R. & Stein, B. E. Topographic organization of somatosensory corticotectal influences in cat. J. Neurophysiol. 51, 843–858 (1984).
Mucke, L., Norita, M., Benedek, G. & Creutzfeldt, O. Physiologic and anatomic investigation of a visual cortical area situated in the ventral bank of the anterior ectosylvian sulcus of the cat. Exp. Brain Res. 46, 1–11 (1982).
Olson, C. R. & Graybiel, A. M. Ectosylvian visual area of the cat: location, retinotopic organization, and connections. J. Comp. Neurol. 261, 277–294 (1987).
Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797–1809 (1993).
Alvarado, J. C., Rowland, B. A., Stanford, T. R. & Stein, B. E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13–23 (2008).
McHaffie, J. G. et al. in The New Handbook of Multisensory Processing 31–47 (ed. Stein, B.) (MIT Press, 2012).
Fuentes-Santamaria, V., Alvarado, J. C., Stein, B. E. & McHaffie, J. G. Cortex contacts both output neurons and nitrergic interneurons in the superior colliculus: direct and indirect routes for multisensory integration. Cereb. Cortex 18, 1640–1652 (2008).
Fuentes-Santamaria, V., McHaffie, J. G. & Stein, B. E. Maturation of multisensory integration in the superior colliculus: expression of nitric oxide synthase and neurofilament SMI-32. Brain Res. 1242, 45–53 (2008).
McHaffie, J. G., Kruger, L., Clemo, H. R. & Stein, B. E. Corticothalamic and corticotectal somatosensory projections from the anterior ectosylvian sulcus (SIV cortex) in neonatal cats: an anatomical demonstration with HRP and 3H-leucine. J. Comp. Neurol. 274, 115–126 (1988).
Jiang, W., Jiang, H., Rowland, B. A. & Stein, B. E. Multisensory orientation behavior is disrupted by neonatal cortical ablation. J. Neurophysiol. 97, 557–562 (2007).
Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1–10 (1996).
Rauschecker, J. P. Developmental plasticity and memory. Behav. Brain Res. 66, 7–12 (1995).
Wiesel, T. N. & Hubel, D. H. Single-cell responses in striate cortex of kittens deprived of vision in one eye. J. Neurophysiol. 26, 1003–1017 (1963).
Wallace, M. T., Perrault, T. J. Jr, Hairston, W. D. & Stein, B. E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24, 9580–9584 (2004).
Feldman, D. E. The spike-timing dependence of plasticity. Neuron 75, 556–571 (2012).
Crair, M. C. Neuronal activity during development: permissive or instructive? Curr. Opin. Neurobiol. 9, 88–93 (1999).
Hua, J. Y. & Smith, S. J. Neural activity and the dynamics of central nervous system development. Nature Neurosci. 7, 327–332 (2004).
Kirkwood, A., Lee, H. K. & Bear, M. F. Co-regulation of long-term potentiation and experience-dependent synaptic plasticity in visual cortex by age and experience. Nature 375, 328–331 (1995).
Knudsen, E. I. & Brainard, M. S. Visual instruction of the neural map of auditory space in the developing optic tectum. Science 253, 85–87 (1991).
Withington, D. J. The effect of binocular lid suture on auditory responses in the guinea-pig superior colliculus. Neurosci. Lett. 136, 153–156 (1992).
Rauschecker, J. P. & Harris, L. R. Auditory compensation of the effects of visual deprivation in the cat's superior colliculus. Exp. Brain Res. 50, 69–83 (1983). This article provides an early demonstration of the ability of superior colliculus neurons to engage in cross-modal compensation by expanding one sensory representation as a result of restricted experience in another.
Champoux, F., Bacon, B. A., Lepore, F. & Guillemot, J.-P. Effects of early binocular enucleation on auditory and somatosensory coding in the superior colliculus of the rat. Brain Res. 1191, 84–95 (2008).
Mundiñano, I. C. & Martínez-Millán, L. Somatosensory cross-modal plasticity in the superior colliculus of visually deafferented rats. Neuroscience 165, 1457–1470 (2010).
Merabet, L. B. & Pascual-Leone, A. Neural reorganization following sensory loss: the opportunity of change. Nature Rev. Neurosci. 11, 44–52 (2010).
Chang, E. F. & Merzenich, M. M. Environmental noise retards auditory cortical development. Science 300, 498–502 (2003).
Efrati, A. & Gutfreund, Y. Early life exposure to noise alters the representation of auditory localization cues in the auditory space map of the barn owl. J. Neurophysiol. 105, 2522–2535 (2011).
Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. Noise-rearing disrupts the maturation of multisensory integration. Eur. J. Neurosci. 39, 602–613 (2014). This was the first study to demonstrate that superior colliculus neurons require experience with co-varying cross-modal signals to develop the circuitry needed to integrate the information they provide.
Wallace, M. T. & Stein, B. E. Early experience determines how the senses will interact. J. Neurophysiol. 97, 921–926 (2007).
Benedetti, F. & Ferro, I. The effects of early postnatal modification of body shape on the somatosensory-visual organization in mouse superior colliculus. Eur. J. Neurosci. 7, 412–418 (1995).
Benedetti, F. Orienting behaviour and superior colliculus sensory representations in mice with the vibrissae bent into the contralateral hemispace. Eur. J. Neurosci. 7, 1512–1519 (1995).
Brainard, M. S. & Knudsen, E. I. Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum. J. Neurosci. 18, 3929–3942 (1998).
King, A. J. Neural plasticity: how the eye tells the brain about sound location. Curr. Biol. 12, R393–R395 (2002).
Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823 (2004).
De Gelder, B. & Bertelson, P. Multisensory integration, perception and ecological validity. Trends Cogn. Sci. 7, 460–467 (2003).
King, A. J. & Calvert, G. A. Multisensory integration: perceptual grouping by eye and ear. Curr. Biol. 11, R322–R325 (2001).
Baier, B., Kleinschmidt, A. & Müller, N. G. Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information. J. Neurosci. 26, 12260–12265 (2006).
Fiebelkorn, I. C., Foxe, J. J. & Molholm, S. Dual mechanisms for the cross-sensory spread of attention: how much do learned associations matter? Cereb. Cortex 20, 109–120 (2010).
Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W. & Perrett, D. I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 17, 377–391 (2005).
Yu, L., Rowland, B. A., Xu, J. & Stein, B. E. Multisensory plasticity in adulthood: cross-modal experience enhances neuronal excitability and exposes silent inputs. J. Neurophysiol. 109, 464–474 (2013).
Rowland, B., Jiang, W. & Stein, B. Brief cortical deactivation early in life has long-lasting effects on multisensory behavior. J. Neurosci. 34, 7198–7202 (2014). There is a sensitive period during early life when cross-modal experience, through the association cortex, normally alters the multisensory circuit of the superior colliculus so that it can engage in multisensory integration.
Bi, G. & Poo, M. Synaptic modification by correlated activity: Hebb's postulate revisited. Annu. Rev. Neurosci. 24, 139–166 (2001).
Gerstner, W., Ritz, R. & van Hemmen, J. L. Why spikes? Hebbian learning and retrieval of time-resolved excitation patterns. Biol. Cybern. 69, 503–515 (1993).
Cuppini, C., Magosso, E., Rowland, B., Stein, B. & Ursino, M. Hebbian mechanisms help explain development of multisensory integration in the superior colliculus: a neural network model. Biol. Cybern. 106, 691–713 (2012).
Grossberg, S., Roberts, K., Aguilar, M. & Bullock, D. A neural model of multimodal adaptive saccadic eye movement control by superior colliculus. J. Neurosci. 17, 9706–9725 (1997).
Yu, L., Stein, B. E. & Rowland, B. A. Adult plasticity in multisensory neurons: short-term experience-dependent changes in the superior colliculus. J. Neurosci. 29, 15910–15922 (2009).
Mégevand, P., Molholm, S., Nayak, A. & Foxe, J. J. Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS ONE 8, e71608 (2013).
Yu, L., Rowland, B. A. & Stein, B. E. Initiating the development of multisensory integration by manipulating sensory experience. J. Neurosci. 30, 4904–4913 (2010).
Royal, D. W., Krueger, J., Fister, M. C. & Wallace, M. T. Adult plasticity of spatiotemporal receptive fields of multisensory superior colliculus neurons following early visual deprivation. Restor. Neurol. Neurosci. 28, 259–270 (2010).
Smith, A. L. et al. An investigation of the role of auditory cortex in sound localization using muscimol-releasing Elvax. Eur. J. Neurosci. 19, 3059–3072 (2004).
Putzar, L., Gondan, M. & Röder, B. Basic multisensory functions can be acquired after congenital visual pattern deprivation in humans. Dev. Neuropsychol. 37, 697–711 (2012). A demonstration of the inherent plasticity of multisensory integration in the human brain.
Schorr, E. A., Fox, N. A., van Wassenhove, V. & Knudsen, E. I. Auditory-visual fusion in speech perception in children with cochlear implants. Proc. Natl Acad. Sci. USA 102, 18748–18750 (2005).
Putzar, L., Hötting, K. & Röder, B. Early visual deprivation affects the development of face recognition and of audio-visual speech perception. Restor. Neurol. Neurosci. 28, 251–257 (2010).
Rouger, J. et al. Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc. Natl Acad. Sci. USA 104, 7295–7300 (2007).
Carriere, B. N. et al. Visual deprivation alters the development of cortical multisensory integration. J. Neurophysiol. 98, 2858–2867 (2007).
Gori, M., Giuliana, L., Sandini, G. & Burr, D. Visual size perception and haptic calibration during development. Dev. Sci. 15, 854–862 (2012).
Gori, M., Sandini, G. & Burr, D. Development of visuo-auditory integration in space and time. Front. Integr. Neurosci. 6, 77 (2012). The protracted postnatal development of aspects of visual–auditory integration in humans is examined in this study.
Lewkowicz, D. J. Perception of dynamic and static audiovisual sequences in 3- and 4-month-old infants. Child Dev. 79, 1538–1554 (2008).
Neil, P. A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D. J. & Shimojo, S. Development of multisensory spatial integration and perception in humans. Dev. Sci. 9, 454–464 (2006). A description of the postnatal development of multisensory-processing capabilities in human subjects.
Barutchu, A. et al. Audiovisual integration in noise by children and adults. J. Exp. Child Psychol. 105, 38–50 (2010).
Foxe, J. J. et al. Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cereb. Cortex http://dx.doi.org/10.1093/cercor/bht213 (2013). This study identifies the early multisensory-processing disorder associated with autism spectrum disorder and its resolution.
Hairston, W. D., Burdette, J. H., Flowers, D. L., Wood, F. B. & Wallace, M. T. Altered temporal profile of visual-auditory multisensory interactions in dyslexia. Exp. Brain Res. 166, 474–480 (2005).
Stevenson, R. A. et al. Multisensory temporal integration in autism spectrum disorders. J. Neurosci. 34, 691–697 (2014).
Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005).
Cuppini, C., Ursino, M., Magosso, E., Rowland, B. A. & Stein, B. E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).
Fetsch, C. R., Deangelis, G. C. & Angelaki, D. E. Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nature Rev. Neurosci. 14, 429–442 (2013).
Portions of the work described here have been supported by US National Institutes of Health grants EY016716 and NS036916 and a grant from the Wallace Foundation.
The authors declare no competing financial interests.
A process (behaviour) or entity (neuron or circuit) that incorporates information derived from more than one sensory modality.
- Multisensory enhancement
The response to a cross-modal stimulus is significantly greater than its responses to either of the component stimuli.
- Bayesian frameworks
Statistical frameworks used to model perception in which a feature of the world is inferred based on acquired sensory evidence.
- Receptive fields
Regions of external space or location on the body in which stimuli will reliably elicit responses from a given neuron.
- Cross-modal stimulus
A stimulus that activates two or more senses.
- Spatiotemporal concordance
Closely aligned in space and time.
- Spike-timing-dependent plasticity
(STDP). A principle by which synaptic efficacy is strengthened when the presynaptic neuron reliably generates an action potential before the postsynaptic neuron generates an action potential, but is weakened when the reverse relationship occurs or when the activity patterns are decorrelated.
About this article
Cite this article
Stein, B., Stanford, T. & Rowland, B. Development of multisensory integration from the perspective of the individual neuron. Nat Rev Neurosci 15, 520–535 (2014). https://doi.org/10.1038/nrn3742
This article is cited by
Reduced activity of parvalbumin-positive interneurons in the posterior parietal cortex causes visually dominant multisensory decisions in freely navigating mice
Molecular Brain (2022)
Audiovisual integration in the Mauthner cell enhances escape probability and reduces response latency
Scientific Reports (2022)
Socially meaningful visual context either enhances or inhibits vocalisation processing in the macaque brain
Nature Communications (2022)
A calibratable sensory neuron based on epitaxial VO2 for spike-based neuromorphic multisensory system
Nature Communications (2022)
Science China Life Sciences (2022)