Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Development of multisensory integration from the perspective of the individual neuron

Nature Reviews Neuroscience volume 15, pages 520–535 (2014)Cite this article

Subjects

Key Points

  • The brain is able to maximize its detection and evaluation of external events by the process of multisensory integration, whereby information from different senses is synthesized and used in concert.

  • Studies of single neurons in animals have revealed that this process is not present at birth, and studies of multisensory neurons in the superior colliculus have provided a developmental model by which the maturation of these neurons can be understood.

  • This integrative capability gradually develops during postnatal life as the underlying neural circuit matures and as the brain acquires experience with cross-modal events.

  • Disruptions of an essential cortical input or the absence of experience with cross-modal events does not preclude the development of multisensory neurons in the superior colliculus. However, it does interfere with the development of a multisensory neuron's capacity to integrate its sensory inputs.

  • Experience also helps to craft the principles by which multisensory integration is instantiated in order to adapt the system to the environment in which it will be used. Thus, abnormal cross-modal experience can alter the principles that normally govern this process.

  • Although multisensory integration capability is most rapidly developed in the young brain, under the proper circumstances even a brain deprived of cross-modal experience early in life can later develop this capability, albeit less efficiently.

Abstract

The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It maximizes the brain's use of the information available to it at any given moment and enhances the physiological salience of external events. Because each sense conveys a unique perspective of the external world, synthesizing information across senses affords computational benefits that cannot otherwise be achieved. Multisensory integration not only has substantial survival value but can also create unique experiences that emerge when signals from different sensory channels are bound together. However, neurons in a newborn's brain are not capable of multisensory integration, and studies in the midbrain have shown that the development of this process is not predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences that alter the underlying neural circuit in such a way that optimizes multisensory integrative capabilities for the environment in which the animal will function.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: The organization and development of the multisensory superior colliculus.
Figure 2: Developmental profile of multisensory integration in the cat superior colliculus.
Figure 3: A synergy between unisensory subregions of the association cortex drives multisensory integration capabilities in the mature superior colliculus.
Figure 4: The development of multisensory integration depends on concordant experience with cross-modal cues.
Figure 5: Deactivating the association cortex in early life delays the development of multisensory integration.
Figure 6: Adult plasticity in multisensory integration.
Figure 7: Learning to integrate requires a neuron to experience both cues simultaneously.

Similar content being viewed by others

References

  1. Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).

    Google Scholar 

  2. Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).

    Article  CAS  PubMed  Google Scholar 

  3. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    Article  CAS  PubMed  Google Scholar 

  4. Shams, L., Ma, W. J. & Beierholm, U. Sound-induced flash illusion as an optimal percept. Neuroreport 16, 1923–1927 (2005).

    Article  PubMed  Google Scholar 

  5. Lewkowicz, D. J. & Lickliter, R. (eds) The Development of Intersensory Perception: Comparative Perspectives (Lawrence Erlbaum Associates, 1994).

    Google Scholar 

  6. Stein, B. (ed.) The New Handbook of Multisensory Processing (MIT Press, 2012).

    Google Scholar 

  7. Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J. Jr & Rowland, B. A. Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp. Brain Res. 198, 113–126 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Binns, K. E. & Salt, T. E. Importance of NMDA receptors for multimodal integration in the deep layers of the cat superior colliculus. J. Neurophysiol. 75, 920–930 (1996).

    Article  CAS  PubMed  Google Scholar 

  9. Wallace, M. T., Meredith, M. A. & Stein, B. E. Multisensory integration in the superior colliculus of the alert cat. J. Neurophysiol. 80, 1006–1010 (1998).

    Article  CAS  PubMed  Google Scholar 

  10. Bolognini, N., Leo, F., Passamonti, C., Stein, B. E. & Làdavas, E. Multisensory-mediated auditory localization. Perception 36, 1477–1485 (2007).

    Article  PubMed  Google Scholar 

  11. Corneil, B. D. & Munoz, D. P. The influence of auditory and visual distractors on human orienting gaze shifts. J. Neurosci. 16, 8193–8207 (1996).

    Article  CAS  PubMed  Google Scholar 

  12. Diederich, A. & Colonius, H. Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys. 66, 1388–1404 (2004).

    Article  PubMed  Google Scholar 

  13. Fetsch, C. R., Pouget, A., DeAngelis, G. C. & Angelaki, D. E. Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neurosci. 15, 146–154 (2012). This is a comprehensive review of the utility of Bayesian frameworks for studying multisensory integration.

    Article  CAS  Google Scholar 

  14. Foxe, J. J. & Schroeder, C. E. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16, 419–423 (2005).

    Article  PubMed  Google Scholar 

  15. Frens, M. A. & Van Opstal, A. J. Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Res. Bull. 46, 211–224 (1998).

    Article  CAS  PubMed  Google Scholar 

  16. Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G. & Fendrich, R. Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J. Exp. Psychol. Hum. Percept. Perform. 20, 131–153 (1994). This early article helped to establish the relationship between multisensory integration and reaction speed.

    Article  CAS  PubMed  Google Scholar 

  17. Naumer, M. J. & Kaiser, J. (eds) Multisensory Object Perception in the Primate Brain (Springer, 2010).

    Google Scholar 

  18. Sánchez-García, C., Alsius, A., Enns, J. T. & Soto-Faraco, S. Cross-modal prediction in speech perception. PLoS ONE 6, e25198 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Senkowski, D., Talsma, D., Grigutsch, M., Herrmann, C. S. & Woldorff, M. G. Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45, 561–571 (2007).

    Article  PubMed  Google Scholar 

  20. Stein, B. E., Meredith, M. A., Huneycutt, W. S. & McDade, L. Behavioral indices of multisensory integration: orientation to visual cues is affected by auditory stimuli. J. Cogn. Neurosci. 1, 12–24 (1989). This paper showed that the physiological principles of multisensory integration that are evident in single superior colliculus neurons applies to overt orientation behaviour.

    Article  CAS  PubMed  Google Scholar 

  21. Talsma, D., Doty, T. J. & Woldorff, M. G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690 (2007).

    Article  PubMed  Google Scholar 

  22. Woods, T. M. & Recanzone, G. H. Visually induced plasticity of auditory spatial perception in macaques. Curr. Biol. 14, 1559–1564 (2004).

    Article  CAS  PubMed  Google Scholar 

  23. Zhang, M., Weisser, V. D., Stilla, R., Prather, S. C. & Sathian, K. Multisensory cortical processing of object shape and its relation to mental imagery. Cogn. Affect. Behav. Neurosci. 4, 251–259 (2004).

    Article  PubMed  Google Scholar 

  24. Goldring, J. E., Dorris, M. C., Corneil, B. D., Ballantyne, P. A. & Munoz, D. P. Combined eye-head gaze shifts to visual and auditory targets in humans. Exp. Brain Res. 111, 68–78 (1996).

    Article  CAS  PubMed  Google Scholar 

  25. Sperdin, H. F., Cappe, C., Foxe, J. J. & Murray, M. M. Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front. Integr. Neurosci. 3, 2 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Von Saldern, S. & Noppeney, U. Sensory and striatal areas integrate auditory and visual signals into behavioral benefits during motion discrimination. J. Neurosci. 33, 8841–8849 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Bremner, A. J., Lewkowicz, D. J. & Spence, C. (eds) Multisensory Development (Oxford Univ. Press, 2012).

    Book  Google Scholar 

  28. Bizley, J. K. & King, A. J. Visual-auditory spatial processing in auditory cortical neurons. Brain Res. 1242, 24–36 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Brett-Green, B., Fifková, E., Larue, D. T., Winer, J. A. & Barth, D. S. A multisensory zone in rat parietotemporal cortex: intra- and extracellular physiology and thalamocortical connections. J. Comp. Neurol. 460, 223–237 (2003).

    Article  PubMed  Google Scholar 

  30. Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11–23 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).

    Article  PubMed  Google Scholar 

  32. Graziano, M., Gross, C., Taylor, C. & Moore, T. in Crossmodal Space and Crossmodal Attention (eds Spence, C. & Driver, J.) 51–67 (Oxford Univ. Press, 2004).

    Book  Google Scholar 

  33. King, A. J. & Palmer, A. R. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60, 492–500 (1985).

    Article  CAS  PubMed  Google Scholar 

  34. Romanski, L. M. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cereb. Cortex 17 (Suppl. 1), i61–i69 (2007).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Schroeder, C. E. & Foxe, J. Multisensory contributions to low-level, 'unisensory' processing. Curr. Opin. Neurobiol. 15, 454–458 (2005). An examination of the impact of multisensory integration in cortical regions believed to be unisensory. The paper examines the evidence for multisensory information coding even within classically defined, modality-specific areas of that cortex.

    Article  CAS  PubMed  Google Scholar 

  36. Wallace, M., Ramachandran, R. & Stein, B. A revised view of sensory cortical parcellation. Proc. Natl Acad. Sci. USA 101, 2167–2172 (2004).

    Article  CAS  PubMed  Google Scholar 

  37. Winkowski, D. E. & Knudsen, E. I. Top-down control of multimodal sensitivity in the barn owl optic tectum. J. Neurosci. 27, 13279–13291 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Macaluso, E. & Driver, J. in The Handbook of Multisensory Processes (eds Calvert, G. A., Spence, C. & Stein, B. E.) 529–548 (MIT Press, 2004).

    Google Scholar 

  39. Cohen, Y. E. & Andersen, R. A. in The Handbook of Multisensory Processes (eds Calvert, G. A., Spence, C. & Stein, B. E.) 463–479 (MIT Press, 2004).

    Google Scholar 

  40. Burnett, L. R., Stein, B. E., Chaponis, D. & Wallace, M. T. Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124, 535–547 (2004).

    Article  CAS  PubMed  Google Scholar 

  41. Burnett, L. R., Stein, B. E., Perrault, T. J. Jr & Wallace, M. T. Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration. Exp. Brain Res. 179, 325–338 (2007).

    Article  PubMed  Google Scholar 

  42. Groh, J. M. & Sparks, D. L. Saccades to somatosensory targets. II. Motor convergence in primate superior colliculus. J. Neurophysiol. 75, 428–438 (1996).

    Article  CAS  PubMed  Google Scholar 

  43. Groh, J. M. & Sparks, D. L. Saccades to somatosensory targets. III. Eye-position-dependent somatosensory activity in primate superior colliculus. J. Neurophysiol. 75, 439–453 (1996).

    Article  CAS  PubMed  Google Scholar 

  44. Harrington, L. K. & Peck, C. K. Spatial disparity affects visual-auditory interactions in human sensorimotor processing. Exp. Brain Res. 122, 247–252 (1998).

    Article  CAS  PubMed  Google Scholar 

  45. Jay, M. F. & Sparks, D. L. Sensorimotor integration in the primate superior colliculus. I. Motor convergence. J. Neurophysiol. 57, 22–34 (1987).

    Article  CAS  PubMed  Google Scholar 

  46. Jay, M. F. & Sparks, D. L. Sensorimotor integration in the primate superior colliculus. II. Coordinates of auditory signals. J. Neurophysiol. 57, 35–55 (1987).

    Article  CAS  PubMed  Google Scholar 

  47. Munoz, D. P. & Wurtz, R. H. Saccade-related activity in monkey superior colliculus. I. Characteristics of burst and buildup cells. J. Neurophysiol. 73, 2313–2333 (1995).

    Article  CAS  PubMed  Google Scholar 

  48. Munoz, D. P. & Wurtz, R. H. Saccade-related activity in monkey superior colliculus. II. Spread of activity during saccades. J. Neurophysiol. 73, 2334–2348 (1995).

    Article  CAS  PubMed  Google Scholar 

  49. Lomber, S. G., Payne, B. R. & Cornwell, P. Role of the superior colliculus in analyses of space: superficial and intermediate layer contributions to visual orienting, auditory orienting, and visuospatial discriminations during unilateral and bilateral deactivations. J. Comp. Neurol. 441, 44–57 (2001).

    Article  CAS  PubMed  Google Scholar 

  50. Levine, M. S., Hull, C. D. & Buchwald, N. A. Development of motor activity in kittens. Dev. Psychobiol. 13, 357–371 (1980).

    Article  CAS  PubMed  Google Scholar 

  51. Norton, T. T. Receptive-field properties of superior colliculus cells and development of visual behavior in kittens. J. Neurophysiol. 37, 674–690 (1974).

    Article  CAS  PubMed  Google Scholar 

  52. Meredith, M. A., Wallace, M. T. & Stein, B. E. Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp. Brain Res. 88, 181–186 (1992).

    Article  CAS  PubMed  Google Scholar 

  53. Sparks, D. L. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol. Rev. 66, 118–171 (1986).

    Article  CAS  PubMed  Google Scholar 

  54. Wurtz, R. H. & Goldberg, M. E. The role of the superior colliculus in visually-evoked eye movements. Bibl. Ophthalmol. 82, 149–158 (1972).

    CAS  PubMed  Google Scholar 

  55. Moschovakis, A. K. et al. An anatomical substrate for the spatiotemporal transformation. J. Neurosci. 18, 10219–10229 (1998).

    Article  CAS  PubMed  Google Scholar 

  56. Knill, D. C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004).

    Article  CAS  PubMed  Google Scholar 

  57. Patton, P. E. & Anastasio, T. J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783–810 (2003).

    Article  PubMed  Google Scholar 

  58. Rowland, B., Stanford, T. & Stein, B. A Bayesian model unifies multisensory spatial localization with the physiological properties of the superior colliculus. Exp. Brain Res. 180, 153–161 (2007).

    Article  PubMed  Google Scholar 

  59. Stein, B. E., Labos, E. & Kruger, L. Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J. Neurophysiol. 36, 667–679 (1973). This was the first demonstration of the developmental chronology of sensory representations in the multisensory superior colliculus.

    Article  CAS  PubMed  Google Scholar 

  60. Stein, B. E., Labos, E. & Kruger, L. Determinants of response latency in neurons of superior colliculus in kittens. J. Neurophysiol. 36, 680–689 (1973).

    Article  CAS  PubMed  Google Scholar 

  61. Benedetti, F. Differential formation of topographic maps on the cerebral cortex and superior colliculus of the mouse by temporally correlated tactile- tactile and tactile-visual inputs. Eur. J. Neurosci. 7, 1942–1951 (1995).

    Article  CAS  PubMed  Google Scholar 

  62. Meredith, M. A., Clemo, H. R. & Stein, B. E. Somatotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J. Comp. Neurol. 312, 353–370 (1991).

    Article  CAS  PubMed  Google Scholar 

  63. Meredith, M. A. & Stein, B. E. The visuotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J. Neurosci. 10, 3727–3742 (1990).

    Article  CAS  PubMed  Google Scholar 

  64. Middlebrooks, J. C. & Knudsen, E. I. A neural code for auditory space in the cat's superior colliculus. J. Neurosci. 4, 2621–2634 (1984).

    Article  CAS  PubMed  Google Scholar 

  65. Stein, B. E., Magalhaes-Castro, B. & Kruger, L. Superior colliculus: visuotopic-somatotopic overlap. Science 189, 224–226 (1975).

    Article  CAS  PubMed  Google Scholar 

  66. Chalupa, L. M. & Rhoades, R. W. Responses of visual, somatosensory, and auditory neurones in the golden hamster's superior colliculus. J. Physiol. 270, 595–626 (1977).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. Dräger, U. C. & Hubel, D. H. Physiology of visual cells in mouse superior colliculus and correlation with somatosensory and auditory input. Nature 253, 203–204 (1975).

    Article  PubMed  Google Scholar 

  68. Dräger, U. C. & Hubel, D. H. Topography of visual and somatosensory projections to mouse superior colliculus. J. Neurophysiol. 39, 91–101 (1976).

    Article  PubMed  Google Scholar 

  69. Gordon, B. Receptive fields in deep layers of cat superior colliculus. J. Neurophysiol. 36, 157–178 (1973).

    Article  CAS  PubMed  Google Scholar 

  70. Graham, J., Pearson, H. E., Berman, N. & Murphy, E. H. Laminar organization of superior colliculus in the rabbit: a study of receptive-field properties of single units. J. Neurophysiol. 45, 915–932 (1981).

    Article  CAS  PubMed  Google Scholar 

  71. King, A. J., Schnupp, J. W. & Thompson, I. D. Signals from the superficial layers of the superior colliculus enable the development of the auditory space map in the deeper layers. J. Neurosci. 18, 9394–9408 (1998).

    Article  CAS  PubMed  Google Scholar 

  72. Knudsen, E. I. Auditory and visual maps of space in the optic tectum of the owl. J. Neurosci. 2, 1177–1194 (1982).

    Article  CAS  PubMed  Google Scholar 

  73. Stein, B. E. & Dixon, J. P. Superior colliculus cells respond to noxious stimuli. Brain Res. 158, 65–73 (1978).

    Article  CAS  PubMed  Google Scholar 

  74. Stein, B. E., Magalhães-Castro, B. & Kruger, L. Relationship between visual and tactile representations in cat superior colliculus. J. Neurophysiol. 39, 401–419 (1976).

    Article  CAS  PubMed  Google Scholar 

  75. Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).

    Article  PubMed  Google Scholar 

  76. Jiang, W., Jiang, H. & Stein, B. E. Neonatal cortical ablation disrupts multisensory development in superior colliculus. J. Neurophysiol. 95, 1380–1396 (2006).

    Article  PubMed  Google Scholar 

  77. Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444 (1997). This was the first study of the development of multisensory integration in superior colliculus neurons.

    Article  CAS  PubMed  Google Scholar 

  78. Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578–3582 (2000).

    Article  CAS  PubMed  Google Scholar 

  79. Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. Incorporating cross-modal statistics in the development and maintenance of multisensory integration. J. Neurosci. 32, 2287–2298 (2012). This study demonstrates some of the experiential prerequisites for instantiating multisensory integration capabilities in superior colliculus neurons.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Meredith, M. A. & Stein, B. E. Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391 (1983). This was the first paper to demonstrate the multisensory integration capability of single superior colliculus neurons.

    Article  CAS  PubMed  Google Scholar 

  81. Kadunce, D. C., Vaughan, J. W., Wallace, M. T. & Stein, B. E. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp. Brain Res. 139, 303–310 (2001).

    Article  CAS  PubMed  Google Scholar 

  82. Meredith, M. A., Nemitz, J. W. & Stein, B. E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229 (1987).

    Article  CAS  PubMed  Google Scholar 

  83. Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).

    Article  CAS  PubMed  Google Scholar 

  84. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G. & Stein, B. E. Mechanisms of within- and cross-modality suppression in the superior colliculus. J. Neurophysiol. 78, 2834–2847 (1997).

    Article  CAS  PubMed  Google Scholar 

  85. Pluta, S. R., Rowland, B. A., Stanford, T. R. & Stein, B. E. Alterations to multisensory and unisensory integration by stimulus competition. J. Neurophysiol. 106, 3091–3101 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  86. Larson, M. A. & Stein, B. E. The use of tactile and olfactory cues in neonatal orientation and localization of the nipple. Dev. Psychobiol. 17, 423–436 (1984).

    Article  CAS  PubMed  Google Scholar 

  87. Kao, C. Q., McHaffie, J. G., Meredith, M. A. & Stein, B. E. Functional development of a central visual map in cat. J. Neurophysiol. 72, 266–272 (1994).

    Article  CAS  PubMed  Google Scholar 

  88. Hubel, D. H. & Wiesel, T. N. Receptive fields of cells in striate cortex of very young, visually inexperienced kittens. J. Neurophysiol. 26, 994–1002 (1963).

    Article  CAS  PubMed  Google Scholar 

  89. Horton, J. C. & Hocking, D. R. An adult-like pattern of ocular dominance columns in striate cortex of newborn monkeys prior to visual experience. J. Neurosci. 16, 1791–1807 (1996).

    Article  CAS  PubMed  Google Scholar 

  90. Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243 (1968).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  91. Rakic, P. Prenatal development of the visual system in rhesus monkey. Philos. Trans. R. Soc. Lond. B 278, 245–260 (1977).

    Article  CAS  Google Scholar 

  92. Wallace, M. T., McHaffie, J. G. & Stein, B. E. Visual response properties and visuotopic representation in the newborn monkey superior colliculus. J. Neurophysiol. 78, 2732–2741 (1997).

    Article  CAS  PubMed  Google Scholar 

  93. Wallace, M. T. & Stein, B. E. Sensory organization of the superior colliculus in cat and monkey. Prog. Brain Res. 112, 301–311 (1996).

    Article  CAS  PubMed  Google Scholar 

  94. Edwards, S. B., Ginsburgh, C. L., Henkel, C. K. & Stein, B. E. Sources of subcortical projections to the superior colliculus in the cat. J. Comp. Neurol. 184, 309–329 (1979).

    Article  CAS  PubMed  Google Scholar 

  95. Huerta, M. F. & Harting, J. K. The projection from the nucleus of the posterior commissure to the superior colliculus of the cat: patch-like endings within the intermediate and deep grey layers. Brain Res. 238, 426–432 (1982).

    Article  CAS  PubMed  Google Scholar 

  96. Meredith, M. A. & Clemo, H. R. Auditory cortical projection from the anterior ectosylvian sulcus (Field AES) to the superior colliculus in the cat: an anatomical and electrophysiological study. J. Comp. Neurol. 289, 687–707 (1989).

    Article  CAS  PubMed  Google Scholar 

  97. Stein, B. E., Spencer, R. F. & Edwards, S. B. Corticotectal and corticothalamic efferent projections of SIV somatosensory cortex in cat. J. Neurophysiol. 50, 896–909 (1983).

    Article  CAS  PubMed  Google Scholar 

  98. Scannell, J. W. et al. Visual motion processing in the anterior ectosylvian sulcus of the cat. J. Neurophysiol. 76, 895–907 (1996).

    Article  CAS  PubMed  Google Scholar 

  99. Alvarado, J. C., Stanford, T. R., Rowland, B. A., Vaughan, J. W. & Stein, B. E. Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. J. Neurosci. 29, 6580–6592 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  100. Alvarado, J. C., Stanford, T. R., Vaughan, J. W. & Stein, B. E. Cortex mediates multisensory but not unisensory integration in superior colliculus. J. Neurosci. 27, 12775–12786 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  101. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001). Two areas of association cortical areas were shown to have crucial roles in superior colliculus multisensory integration.

    Article  CAS  PubMed  Google Scholar 

  102. Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240–1255 (2002).

    Article  PubMed  Google Scholar 

  103. Jiang, W. & Stein, B. E. Cortex controls multisensory depression in superior colliculus. J. Neurophysiol. 90, 2123–2135 (2003).

    Article  PubMed  Google Scholar 

  104. Stein, B. E., Wallace, M. W., Stanford, T. R. & Jiang, W. Cortex governs multisensory integration in the midbrain. Neuroscientist 8, 306–314 (2002).

    Article  PubMed  Google Scholar 

  105. Stein, B. E. The development of a dialogue between cortex and midbrain to integrate multisensory information. Exp. Brain Res. 166, 305–315 (2005).

    Article  PubMed  Google Scholar 

  106. Wallace, M. T. & Stein, B. E. Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71, 429–432 (1994).

    Article  CAS  PubMed  Google Scholar 

  107. Clemo, H. R. & Stein, B. E. Topographic organization of somatosensory corticotectal influences in cat. J. Neurophysiol. 51, 843–858 (1984).

    Article  CAS  PubMed  Google Scholar 

  108. Mucke, L., Norita, M., Benedek, G. & Creutzfeldt, O. Physiologic and anatomic investigation of a visual cortical area situated in the ventral bank of the anterior ectosylvian sulcus of the cat. Exp. Brain Res. 46, 1–11 (1982).

    Article  CAS  PubMed  Google Scholar 

  109. Olson, C. R. & Graybiel, A. M. Ectosylvian visual area of the cat: location, retinotopic organization, and connections. J. Comp. Neurol. 261, 277–294 (1987).

    Article  CAS  PubMed  Google Scholar 

  110. Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797–1809 (1993).

    Article  CAS  PubMed  Google Scholar 

  111. Alvarado, J. C., Rowland, B. A., Stanford, T. R. & Stein, B. E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13–23 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  112. McHaffie, J. G. et al. in The New Handbook of Multisensory Processing 31–47 (ed. Stein, B.) (MIT Press, 2012).

    Google Scholar 

  113. Fuentes-Santamaria, V., Alvarado, J. C., Stein, B. E. & McHaffie, J. G. Cortex contacts both output neurons and nitrergic interneurons in the superior colliculus: direct and indirect routes for multisensory integration. Cereb. Cortex 18, 1640–1652 (2008).

    Article  PubMed  Google Scholar 

  114. Fuentes-Santamaria, V., McHaffie, J. G. & Stein, B. E. Maturation of multisensory integration in the superior colliculus: expression of nitric oxide synthase and neurofilament SMI-32. Brain Res. 1242, 45–53 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  115. McHaffie, J. G., Kruger, L., Clemo, H. R. & Stein, B. E. Corticothalamic and corticotectal somatosensory projections from the anterior ectosylvian sulcus (SIV cortex) in neonatal cats: an anatomical demonstration with HRP and 3H-leucine. J. Comp. Neurol. 274, 115–126 (1988).

    Article  CAS  PubMed  Google Scholar 

  116. Jiang, W., Jiang, H., Rowland, B. A. & Stein, B. E. Multisensory orientation behavior is disrupted by neonatal cortical ablation. J. Neurophysiol. 97, 557–562 (2007).

    Article  PubMed  Google Scholar 

  117. Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1–10 (1996).

    Article  CAS  PubMed  Google Scholar 

  118. Rauschecker, J. P. Developmental plasticity and memory. Behav. Brain Res. 66, 7–12 (1995).

    Article  CAS  PubMed  Google Scholar 

  119. Wiesel, T. N. & Hubel, D. H. Single-cell responses in striate cortex of kittens deprived of vision in one eye. J. Neurophysiol. 26, 1003–1017 (1963).

    Article  CAS  PubMed  Google Scholar 

  120. Wallace, M. T., Perrault, T. J. Jr, Hairston, W. D. & Stein, B. E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24, 9580–9584 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  121. Feldman, D. E. The spike-timing dependence of plasticity. Neuron 75, 556–571 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  122. Crair, M. C. Neuronal activity during development: permissive or instructive? Curr. Opin. Neurobiol. 9, 88–93 (1999).

    Article  CAS  PubMed  Google Scholar 

  123. Hua, J. Y. & Smith, S. J. Neural activity and the dynamics of central nervous system development. Nature Neurosci. 7, 327–332 (2004).

    Article  CAS  PubMed  Google Scholar 

  124. Kirkwood, A., Lee, H. K. & Bear, M. F. Co-regulation of long-term potentiation and experience-dependent synaptic plasticity in visual cortex by age and experience. Nature 375, 328–331 (1995).

    Article  CAS  PubMed  Google Scholar 

  125. Knudsen, E. I. & Brainard, M. S. Visual instruction of the neural map of auditory space in the developing optic tectum. Science 253, 85–87 (1991).

    Article  CAS  PubMed  Google Scholar 

  126. Withington, D. J. The effect of binocular lid suture on auditory responses in the guinea-pig superior colliculus. Neurosci. Lett. 136, 153–156 (1992).

    Article  CAS  PubMed  Google Scholar 

  127. Rauschecker, J. P. & Harris, L. R. Auditory compensation of the effects of visual deprivation in the cat's superior colliculus. Exp. Brain Res. 50, 69–83 (1983). This article provides an early demonstration of the ability of superior colliculus neurons to engage in cross-modal compensation by expanding one sensory representation as a result of restricted experience in another.

    Article  CAS  PubMed  Google Scholar 

  128. Champoux, F., Bacon, B. A., Lepore, F. & Guillemot, J.-P. Effects of early binocular enucleation on auditory and somatosensory coding in the superior colliculus of the rat. Brain Res. 1191, 84–95 (2008).

    Article  CAS  PubMed  Google Scholar 

  129. Mundiñano, I. C. & Martínez-Millán, L. Somatosensory cross-modal plasticity in the superior colliculus of visually deafferented rats. Neuroscience 165, 1457–1470 (2010).

    Article  CAS  PubMed  Google Scholar 

  130. Merabet, L. B. & Pascual-Leone, A. Neural reorganization following sensory loss: the opportunity of change. Nature Rev. Neurosci. 11, 44–52 (2010).

    Article  CAS  Google Scholar 

  131. Chang, E. F. & Merzenich, M. M. Environmental noise retards auditory cortical development. Science 300, 498–502 (2003).

    Article  CAS  PubMed  Google Scholar 

  132. Efrati, A. & Gutfreund, Y. Early life exposure to noise alters the representation of auditory localization cues in the auditory space map of the barn owl. J. Neurophysiol. 105, 2522–2535 (2011).

    Article  PubMed  Google Scholar 

  133. Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. Noise-rearing disrupts the maturation of multisensory integration. Eur. J. Neurosci. 39, 602–613 (2014). This was the first study to demonstrate that superior colliculus neurons require experience with co-varying cross-modal signals to develop the circuitry needed to integrate the information they provide.

    Article  PubMed  Google Scholar 

  134. Wallace, M. T. & Stein, B. E. Early experience determines how the senses will interact. J. Neurophysiol. 97, 921–926 (2007).

    Article  PubMed  Google Scholar 

  135. Benedetti, F. & Ferro, I. The effects of early postnatal modification of body shape on the somatosensory-visual organization in mouse superior colliculus. Eur. J. Neurosci. 7, 412–418 (1995).

    Article  CAS  PubMed  Google Scholar 

  136. Benedetti, F. Orienting behaviour and superior colliculus sensory representations in mice with the vibrissae bent into the contralateral hemispace. Eur. J. Neurosci. 7, 1512–1519 (1995).

    Article  CAS  PubMed  Google Scholar 

  137. Brainard, M. S. & Knudsen, E. I. Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum. J. Neurosci. 18, 3929–3942 (1998).

    Article  CAS  PubMed  Google Scholar 

  138. King, A. J. Neural plasticity: how the eye tells the brain about sound location. Curr. Biol. 12, R393–R395 (2002).

    Article  CAS  PubMed  Google Scholar 

  139. Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823 (2004).

    Article  CAS  PubMed  Google Scholar 

  140. De Gelder, B. & Bertelson, P. Multisensory integration, perception and ecological validity. Trends Cogn. Sci. 7, 460–467 (2003).

    Article  PubMed  Google Scholar 

  141. King, A. J. & Calvert, G. A. Multisensory integration: perceptual grouping by eye and ear. Curr. Biol. 11, R322–R325 (2001).

    Article  CAS  PubMed  Google Scholar 

  142. Baier, B., Kleinschmidt, A. & Müller, N. G. Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information. J. Neurosci. 26, 12260–12265 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  143. Fiebelkorn, I. C., Foxe, J. J. & Molholm, S. Dual mechanisms for the cross-sensory spread of attention: how much do learned associations matter? Cereb. Cortex 20, 109–120 (2010).

    Article  PubMed  Google Scholar 

  144. Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W. & Perrett, D. I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 17, 377–391 (2005).

    Article  PubMed  Google Scholar 

  145. Yu, L., Rowland, B. A., Xu, J. & Stein, B. E. Multisensory plasticity in adulthood: cross-modal experience enhances neuronal excitability and exposes silent inputs. J. Neurophysiol. 109, 464–474 (2013).

    Article  PubMed  Google Scholar 

  146. Rowland, B., Jiang, W. & Stein, B. Brief cortical deactivation early in life has long-lasting effects on multisensory behavior. J. Neurosci. 34, 7198–7202 (2014). There is a sensitive period during early life when cross-modal experience, through the association cortex, normally alters the multisensory circuit of the superior colliculus so that it can engage in multisensory integration.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  147. Bi, G. & Poo, M. Synaptic modification by correlated activity: Hebb's postulate revisited. Annu. Rev. Neurosci. 24, 139–166 (2001).

    Article  CAS  PubMed  Google Scholar 

  148. Gerstner, W., Ritz, R. & van Hemmen, J. L. Why spikes? Hebbian learning and retrieval of time-resolved excitation patterns. Biol. Cybern. 69, 503–515 (1993).

    Article  CAS  PubMed  Google Scholar 

  149. Cuppini, C., Magosso, E., Rowland, B., Stein, B. & Ursino, M. Hebbian mechanisms help explain development of multisensory integration in the superior colliculus: a neural network model. Biol. Cybern. 106, 691–713 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  150. Grossberg, S., Roberts, K., Aguilar, M. & Bullock, D. A neural model of multimodal adaptive saccadic eye movement control by superior colliculus. J. Neurosci. 17, 9706–9725 (1997).

    Article  CAS  PubMed  Google Scholar 

  151. Yu, L., Stein, B. E. & Rowland, B. A. Adult plasticity in multisensory neurons: short-term experience-dependent changes in the superior colliculus. J. Neurosci. 29, 15910–15922 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  152. Mégevand, P., Molholm, S., Nayak, A. & Foxe, J. J. Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS ONE 8, e71608 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  153. Yu, L., Rowland, B. A. & Stein, B. E. Initiating the development of multisensory integration by manipulating sensory experience. J. Neurosci. 30, 4904–4913 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  154. Royal, D. W., Krueger, J., Fister, M. C. & Wallace, M. T. Adult plasticity of spatiotemporal receptive fields of multisensory superior colliculus neurons following early visual deprivation. Restor. Neurol. Neurosci. 28, 259–270 (2010).

    PubMed  Google Scholar 

  155. Smith, A. L. et al. An investigation of the role of auditory cortex in sound localization using muscimol-releasing Elvax. Eur. J. Neurosci. 19, 3059–3072 (2004).

    Article  PubMed  Google Scholar 

  156. Putzar, L., Gondan, M. & Röder, B. Basic multisensory functions can be acquired after congenital visual pattern deprivation in humans. Dev. Neuropsychol. 37, 697–711 (2012). A demonstration of the inherent plasticity of multisensory integration in the human brain.

    Article  PubMed  Google Scholar 

  157. Schorr, E. A., Fox, N. A., van Wassenhove, V. & Knudsen, E. I. Auditory-visual fusion in speech perception in children with cochlear implants. Proc. Natl Acad. Sci. USA 102, 18748–18750 (2005).

    Article  CAS  PubMed  Google Scholar 

  158. Putzar, L., Hötting, K. & Röder, B. Early visual deprivation affects the development of face recognition and of audio-visual speech perception. Restor. Neurol. Neurosci. 28, 251–257 (2010).

    PubMed  Google Scholar 

  159. Rouger, J. et al. Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc. Natl Acad. Sci. USA 104, 7295–7300 (2007).

    Article  CAS  PubMed  Google Scholar 

  160. Carriere, B. N. et al. Visual deprivation alters the development of cortical multisensory integration. J. Neurophysiol. 98, 2858–2867 (2007).

    Article  PubMed  Google Scholar 

  161. Gori, M., Giuliana, L., Sandini, G. & Burr, D. Visual size perception and haptic calibration during development. Dev. Sci. 15, 854–862 (2012).

    Article  PubMed  Google Scholar 

  162. Gori, M., Sandini, G. & Burr, D. Development of visuo-auditory integration in space and time. Front. Integr. Neurosci. 6, 77 (2012). The protracted postnatal development of aspects of visual–auditory integration in humans is examined in this study.

    Article  PubMed  PubMed Central  Google Scholar 

  163. Lewkowicz, D. J. Perception of dynamic and static audiovisual sequences in 3- and 4-month-old infants. Child Dev. 79, 1538–1554 (2008).

    Article  PubMed  Google Scholar 

  164. Neil, P. A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D. J. & Shimojo, S. Development of multisensory spatial integration and perception in humans. Dev. Sci. 9, 454–464 (2006). A description of the postnatal development of multisensory-processing capabilities in human subjects.

    Article  PubMed  Google Scholar 

  165. Barutchu, A. et al. Audiovisual integration in noise by children and adults. J. Exp. Child Psychol. 105, 38–50 (2010).

    Article  PubMed  Google Scholar 

  166. Foxe, J. J. et al. Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cereb. Cortex http://dx.doi.org/10.1093/cercor/bht213 (2013). This study identifies the early multisensory-processing disorder associated with autism spectrum disorder and its resolution.

  167. Hairston, W. D., Burdette, J. H., Flowers, D. L., Wood, F. B. & Wallace, M. T. Altered temporal profile of visual-auditory multisensory interactions in dyslexia. Exp. Brain Res. 166, 474–480 (2005).

    Article  PubMed  Google Scholar 

  168. Stevenson, R. A. et al. Multisensory temporal integration in autism spectrum disorders. J. Neurosci. 34, 691–697 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  169. Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  170. Cuppini, C., Ursino, M., Magosso, E., Rowland, B. A. & Stein, B. E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).

    PubMed  PubMed Central  Google Scholar 

  171. Fetsch, C. R., Deangelis, G. C. & Angelaki, D. E. Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nature Rev. Neurosci. 14, 429–442 (2013).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

Portions of the work described here have been supported by US National Institutes of Health grants EY016716 and NS036916 and a grant from the Wallace Foundation.

Author information

Authors and Affiliations

  1. Department of Neurobiology and Anatomy, Wake Forest School of Medicine, Winston-Salem, 27157, North Carolina, USA

    Barry E. Stein, Terrence R. Stanford & Benjamin A. Rowland

Authors
  1. Barry E. Stein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Terrence R. Stanford
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin A. Rowland
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Barry E. Stein.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

PowerPoint slides

Glossary

Multisensory

A process (behaviour) or entity (neuron or circuit) that incorporates information derived from more than one sensory modality.

Multisensory enhancement

The response to a cross-modal stimulus is significantly greater than its responses to either of the component stimuli.

Bayesian frameworks

Statistical frameworks used to model perception in which a feature of the world is inferred based on acquired sensory evidence.

Receptive fields

Regions of external space or location on the body in which stimuli will reliably elicit responses from a given neuron.

Cross-modal stimulus

A stimulus that activates two or more senses.

Spatiotemporal concordance

Closely aligned in space and time.

Spike-timing-dependent plasticity

(STDP). A principle by which synaptic efficacy is strengthened when the presynaptic neuron reliably generates an action potential before the postsynaptic neuron generates an action potential, but is weakened when the reverse relationship occurs or when the activity patterns are decorrelated.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stein, B., Stanford, T. & Rowland, B. Development of multisensory integration from the perspective of the individual neuron. Nat Rev Neurosci 15, 520–535 (2014). https://doi.org/10.1038/nrn3742

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrn3742

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing