Multisensory integration: current issues from the perspective of the single neuron

  • A Corrigendum to this article was published on 01 May 2008

Key Points

  • Having information from multiple senses converge onto the same neurons allows the neurons to work in concert so that their combined product can enhance the physiological salience of an event, increase the ability to render a judgment about its identity, and initiate responses faster than would otherwise be possible.

  • This interactive synergy among the senses, or 'multisensory integration', is manifested in individual neurons, by enhancing or degrading their responses, and in behaviour, by producing corresponding alterations in performance.

  • Multisensory integration is guided by principles that relate to the spatial and temporal relationship among cross-modal stimuli, as well as to the vigor of the neuron's responses to their individual component stimuli.

  • The spatial principle of multisensory integration relies on faithful register among a neuron's different receptive fields and this register must be maintained in spite of independent movement of the sense organs (such as the eyes). Recent studies suggest that compensation for such movement is less than perfect, and occurs to varying degrees in different neurons and brain regions. Degradation in receptive-field register has strong implications for multisensory integration, but these remain to be examined empirically.

  • Multisensory integration is crucial for high-level cognitive functions in which considerations such as semantic congruence might determine its neural products and the perceptions and behaviours that depend on them.

  • Multiple approaches have demonstrated the impact of multisensory integration in different brain structures in different species, including single-neuron and event-related-potential recordings and brain-imaging techniques.

  • Primary, sensory-specific areas of the brain have now been shown to receive inputs from other senses. The functional role of these other inputs is not yet known, but they might facilitate the processing of information in the native sense.

Abstract

For thousands of years science philosophers have been impressed by how effectively the senses work together to enhance the salience of biologically meaningful events. However, they really had no idea how this was accomplished. Recent insights into the underlying physiological mechanisms reveal that, in at least one circuit, this ability depends on an intimate dialogue among neurons at multiple levels of the neuraxis; this dialogue cannot take place until long after birth and might require a specific kind of experience. Understanding the acquisition and usage of multisensory integration in the midbrain and cerebral cortex of mammals has been aided by a multiplicity of approaches. Here we examine some of the fundamental advances that have been made and some of the challenging questions that remain.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Multisensory integration aids detection and speeds responses.
Figure 2: Multisensory enhancement in a single superior colliculus neuron.
Figure 3: Superior colliculus multisensory integration depends on the cortex.
Figure 4: Multisensory regions in the monkey and human cortex.
Figure 5: Shifting receptive fields are relevant to multisensory integration.

References

  1. 1

    Stein, B. E. & Meredith, M. A. The Merging of the Senses (ed. Gazzaniga, M. S.) (The MIT Press, Cambridge, Massachusetts, 1993).

    Google Scholar 

  2. 2

    Meredith, M. A. & Stein, B. E. Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391 (1983). This seminal paper demonstrated multisensory integration in single superior colliculus neurons.

    CAS  Google Scholar 

  3. 3

    Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18, 787–792 (2007). This study considers the relatively disproportionate effect of superadditivity in multisensory neurons. Superadditivity in multisensory integration is obtained only when the neuron's response to the component stimuli is weak.

    Google Scholar 

  4. 4

    Spence, C. & Driver, J. Crossmodal Space and Crossmodal Attention (Oxford Univ. Press, Oxford, 2004).

    Google Scholar 

  5. 5

    Gillmeister, H. & Eimer, M. Tactile enhancement of auditory detection and perceived loudness. Brain Res. 1160, 58–68 (2007).

    CAS  Google Scholar 

  6. 6

    Calvert, G. A., Spence, C. & Stein, B. E. The Handbook of Multisensory Processes (The MIT Press, Cambridge, Massachusetts, 2004).

    Google Scholar 

  7. 7

    Bell, A. H., Meredith, M. A., Van Opstal, A. J. & Munoz, D. P. Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements. J. Neurophysiol. 93, 3659–3673 (2005).

    Google Scholar 

  8. 8

    Diederich, A. & Colonius, H. Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys. 66, 1388–1404 (2004).

    Google Scholar 

  9. 9

    Frens, M. A., Van Opstal, A. J. & Van der Willigen, R. F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802–816 (1995).

    CAS  Google Scholar 

  10. 10

    Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G. & Fendrich, R. Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J. Exp. Psychol. Hum. Percept. Perform. 20, 131–153 (1994).

    CAS  Google Scholar 

  11. 11

    Nozawa, G., Reuter-Lorenz, P. A. & Hughes, H. C. Parallel and serial processes in the human oculomotor system: bimodal integration and express saccades. Biol. Cybern. 72, 19–34 (1994). This is an early article establishing the relationship between reaction speed and multisensory integration.

    CAS  Google Scholar 

  12. 12

    Rowland, B., Quessy, S., Stanford, T. R. & Stein, B. E. Multisensory integration shortens physiological response latencies. J. Neurosci. 27, 5879–5884 (2007).

    CAS  Google Scholar 

  13. 13

    Burr, D. & Alais, D. Combining visual and auditory information. Prog. Brain Res. 155, 243–258 (2006).

    Google Scholar 

  14. 14

    Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    CAS  PubMed  PubMed Central  Google Scholar 

  15. 15

    Shams, L., Ma, W. J. & Beierholm, U. Sound-induced flash illusion as an optimal percept. Neuroreport 16, 1923–1927 (2005).

    Google Scholar 

  16. 16

    Kadunce, D. C., Vaughan, J. W., Wallace, M. T. & Stein, B. E. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp. Brain Res. 139, 303–310 (2001).

    CAS  Google Scholar 

  17. 17

    Meredith, M. A. & Stein, B. E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J. Neurophysiol. 75, 1843–1857 (1996).

    CAS  Google Scholar 

  18. 18

    Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).

    CAS  Google Scholar 

  19. 19

    Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G. & Stein, B. E. Mechanisms of within- and cross-modality suppression in the superior colliculus. J. Neurophysiol. 78, 2834–2847 (1997).

    CAS  Google Scholar 

  20. 20

    Hartline, P. H., Vimal, R. L., King, A. J., Kurylo, D. D. & Northmore, D. P. Effects of eye position on auditory localization and neural representation of space in superior colliculus of cats. Exp. Brain Res. 104, 402–408 (1995).

    CAS  Google Scholar 

  21. 21

    Jay, M. F. & Sparks, D. L. Auditory receptive fields in primate superior colliculus shift with changes in eye position. Nature 309, 345–347 (1984). This article is a seminal study of sensory coordinate frames; it is the first to systematically examine the eye-position-dependence of auditory receptive fields in the superior colliculus.

    CAS  Google Scholar 

  22. 22

    Peck, C. K., Baro, J. A. & Warder, S. M. Effects of eye position on saccadic eye movements and on the neuronal responses to auditory and visual stimuli in cat superior colliculus. Exp. Brain Res. 103, 227–242 (1995).

    CAS  Google Scholar 

  23. 23

    Groh, J. M. & Sparks, D. L. Saccades to somatosensory targets. III. Eye-position-dependent somatosensory activity in primate superior colliculus. J. Neurophysiol. 75, 439–453 (1996).

    CAS  Google Scholar 

  24. 24

    Meredith, M. A., Nemitz, J. W. & Stein, B. E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229 (1987).

    CAS  Google Scholar 

  25. 25

    Recanzone, G. H. Auditory influences on visual temporal rate perception. J. Neurophysiol. 89, 1078–1093 (2003).

    Google Scholar 

  26. 26

    Meredith, M. A. & Stein, B. E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640–662 (1986).

    CAS  Google Scholar 

  27. 27

    Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005).

    CAS  PubMed  PubMed Central  Google Scholar 

  28. 28

    Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575–2586 (2005).

    Google Scholar 

  29. 29

    Wallace, M. T., Wilkinson, L. K. & Stein, B. E. Representation and integration of multiple sensory inputs in primate superior colliculus. J. Neurophysiol. 76, 1246–1266 (1996).

    CAS  Google Scholar 

  30. 30

    Stein, B. E., Meredith, M. A., Huneycutt, W. S. & McDade, L. Behavioral indices of multisensory integration: orientation to visual cues is affected by visual stimuli. J. Cogn. Neurosci. 1, 12–24 (1989).

    CAS  Google Scholar 

  31. 31

    Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1–10 (1996).

    CAS  Google Scholar 

  32. 32

    Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240–1255 (2002).

    Google Scholar 

  33. 33

    Jiang, W., Jiang, H. & Stein, B. E. Neonatal cortical ablation disrupts multisensory development in superior colliculus. J. Neurophysiol. 95, 1380–1396 (2006).

    Google Scholar 

  34. 34

    Wallace, M. T. & Stein, B. E. Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71, 429–432 (1994).

    CAS  Google Scholar 

  35. 35

    Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).

    Google Scholar 

  36. 36

    Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001). The critical nature of influences from two cortical areas on multisensory integration in superior colliculus neurons is demonstrated here.

    CAS  Google Scholar 

  37. 37

    Stein, B. E., Spencer, R. F. & Edwards, S. B. Corticotectal and corticothalamic efferent projections of SIV somatosensory cortex in cat. J. Neurophysiol. 50, 896–909 (1983).

    CAS  Google Scholar 

  38. 38

    Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797–1809 (1993).

    CAS  Google Scholar 

  39. 39

    Rowland, B., Stanford, T. R. & Stein, B. E. A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception 36, 1431–1443 (2007).

    Google Scholar 

  40. 40

    Anastasio, T. J. & Patton, P. E. A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system. J. Neurosci. 23, 6713–6727 (2003).

    CAS  Google Scholar 

  41. 41

    Wallace, M. T., Meredith, M. A. & Stein, B. E. Integration of multiple sensory modalities in cat cortex. Exp. Brain Res. 91, 484–488 (1992).

    CAS  Google Scholar 

  42. 42

    Stein, B. E. & Wallace, M. T. Comparisons of cross-modality integration in midbrain and cortex. Prog. Brain Res. 112, 289–299 (1996).

    CAS  Google Scholar 

  43. 43

    Stricanne, B., Andersen, R. A. & Mazzoni, P. Eye-centered, head-centered, and intermediate coding of remembered sound locations in area LIP. J. Neurophysiol. 76, 2071–2076 (1996). This was one of the first studies to demonstrate that the auditory receptive fields of LIP neurons can be coded in an eye-centred reference frame.

    CAS  Google Scholar 

  44. 44

    Batista, A. P., Buneo, C. A., Snyder, L. H. & Andersen, R. A. Reach plans in eye-centered coordinates. Science 285, 257–260 (1999).

    CAS  Google Scholar 

  45. 45

    Cohen, Y. E. & Andersen, R. A. Reaches to sounds encoded in an eye-centered reference frame. Neuron 27, 647–652 (2000).

    CAS  Google Scholar 

  46. 46

    Cohen, Y. E. & Andersen, R. A. A common reference frame for movement plans in the posterior parietal cortex. Nature Rev. Neurosci. 3, 553–562 (2002).

    CAS  Google Scholar 

  47. 47

    Pouget, A., Ducom, J. C., Torri, J. & Bavelier, D. Multisensory spatial representations in eye-centered coordinates for reaching. Cognition 83, 1–11 (2002).

    Google Scholar 

  48. 48

    Pouget, A., Deneve, S. & Duhamel, J. R. A computational perspective on the neural basis of multisensory spatial representations. Nature Rev. Neurosci. 3, 741–747 (2002). In this article, the issue of partial reference-frame shifts is considered in detail from a computational perspective.

    CAS  Google Scholar 

  49. 49

    Avillac, M., Deneve, S., Olivier, E., Pouget, A. & Duhamel, J. R. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neurosci. 8, 941–949 (2005).

    CAS  Google Scholar 

  50. 50

    Mullette-Gillman, O. A., Cohen, Y. E. & Groh, J. M. Eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus. J. Neurophysiol. 94, 2331–2352 (2005).

    Google Scholar 

  51. 51

    Schlack, A., Sterbing-D'Angelo, S. J., Hartung, K., Hoffmann, K.-P. & Bremmer, F. Multisensory space representations in the macaque ventral intraparietal area. J. Neurosci. 25, 4616–4625 (2005).

    CAS  Google Scholar 

  52. 52

    Snyder, L. H. Frame-up. Focus on “eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus”. J. Neurophysiol. 94, 2259–2260 (2005).

    Google Scholar 

  53. 53

    Avillac, M., Ben Hamed, S. & Duhamel, J. R. Multisensory integration in the ventral intraparietal area of the macaque monkey. J. Neurosci. 27, 1922–1932 (2007).

    CAS  Google Scholar 

  54. 54

    Laurienti, P. J. et al. Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum. Brain Mapp. 19, 213–223 (2003).

    Google Scholar 

  55. 55

    Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W. & Perrett, D. I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 17, 377–391 (2005).

    Google Scholar 

  56. 56

    Sugihara, T., Diltz, M. D., Averbeck, B. B. & Romanski, L. M. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J. Neurosci. 26, 11138–11147 (2006). This paper is one of the first to examine multisensory integration in single neurons of the prefrontal cortex. It suggests a role for this region in the integration of the visual and auditory stimuli that constitute coherent communication signals.

    CAS  PubMed  PubMed Central  Google Scholar 

  57. 57

    Romanski, L. M. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cereb. Cortex 17, i61–i69 (2007).

    PubMed  PubMed Central  Google Scholar 

  58. 58

    Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).

    Google Scholar 

  59. 59

    Ghazanfar, A. A., Maier, J. X., Hoffman, K. L. & Logothetis, N. K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25, 5004–5012 (2005).

    CAS  Google Scholar 

  60. 60

    Calvert, G. A., Campbell, R. & Brammer, M. J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10, 649–657 (2000).

    CAS  Google Scholar 

  61. 61

    Calvert, G. A., Hansen, P. C., Iversen, S. D. & Brammer, M. J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14, 427–438 (2001). A rationale is provided here for using superadditivity of the BOLD signal as a definitive criterion for detecting multisensory integration in functional-imaging studies.

    CAS  Google Scholar 

  62. 62

    Logothetis, N. K. & Pfeuffer, J. On the nature of the BOLD fMRI contrast mechanism. Magn. Reson. Imaging 22, 1517–1531 (2004).

    Google Scholar 

  63. 63

    Laurienti, P. J., Perrault, T. J., Stanford, T. R., Wallace, M. T. & Stein, B. E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166, 289–297 (2005).

    Google Scholar 

  64. 64

    Beauchamp, M. S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3, 93–113 (2005).

    PubMed  PubMed Central  Google Scholar 

  65. 65

    Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823 (2004).

    CAS  Google Scholar 

  66. 66

    Stevenson, R. A., Geoghegan, M. L. & James, T. W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp. Brain Res. 179, 85–95 (2007). Using the stringent superadditivity criterion for BOLD activation, this study is the first to demonstrate that the superior temporal sulcus integrates information about multisensory objects.

    Google Scholar 

  67. 67

    Foxe, J. J. & Schroeder, C. E. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16, 419–423 (2005). The issue dealt with here is how multisensory convergence at the level of the sensory cortex can be accomplished as a consequence of feedforward projections in the CNS.

    Google Scholar 

  68. 68

    Busse, L., Roberts, K. C., Crist, R. E., Weissman, D. H. & Woldorff, M. G. The spread of attention across modalities and space in a multisensory object. Proc. Natl Acad. Sci. USA 102, 18751–18756 (2005).

    CAS  Google Scholar 

  69. 69

    Talsma, D., Doty, T. J. & Woldorff, M. G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690 (2007).

    Google Scholar 

  70. 70

    Giard, M. H. & Peronnet, F. Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 11, 473–490 (1999). This early event-related-potential study in human subjects demonstrated that multisensory integration can take place very early in the processing of auditory and visual information.

    CAS  Google Scholar 

  71. 71

    Macaluso, E. & Driver, J. Multisensory spatial interactions: a window onto functional integration in the human brain. Trends Neurosci. 28, 264–271 (2005).

    CAS  Google Scholar 

  72. 72

    Bizley, J. K., Nodal, F. R., Bajo, V. M., Nelken, I. & King, A. J. Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb. Cortex 17, 2172–2189 (2007).

    Google Scholar 

  73. 73

    Cappe, C. & Barone, P. Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. Eur. J. Neurosci. 22, 2886–2902 (2005).

    Google Scholar 

  74. 74

    Rockland, K. S. & Ojima, H. Multisensory convergence in calcarine visual areas in macaque monkey. Int. J. Psychophysiol. 50, 19–26 (2003).

    Google Scholar 

  75. 75

    Falchier, A., Clavagnier, S., Barone, P. & Kennedy, H. Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22, 5749–5759 (2002). This paper provided one of the first anatomical demonstrations of non-visual inputs to the primary visual cortex.

    CAS  Google Scholar 

  76. 76

    Meredith, M. A., Keniston, L. R., Dehner, L. R. & Clemo, H. R. Crossmodal projections from somatosensory area SIV to the auditory field of the anterior ectosylvian sulcus (FAES) in cat: further evidence for subthreshold forms of multisensory processing. Exp. Brain Res. 172, 472–484 (2006).

    Google Scholar 

  77. 77

    Allman, B. L. & Meredith, M. A. Multisensory processing in “unimodal” neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. J. Neurophysiol. 98, 545–549 (2007).

    Google Scholar 

  78. 78

    Wallace, M. T., Ramachandran, R. & Stein, B. E. A revised view of sensory cortical parcellation. Proc. Natl Acad. Sci. USA 101, 2167–2172 (2004).

    CAS  Google Scholar 

  79. 79

    Brosch, M., Selezneva, E. & Scheich, H. Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J. Neurosci. 25, 6797–6806 (2005).

    CAS  Google Scholar 

  80. 80

    Fishman, M. C. & Michael, P. Integration of auditory information in the cat's visual cortex. Vision Res. 13, 1415–1419 (1973).

    CAS  Google Scholar 

  81. 81

    Morrell, F. Visual system's view of acoustic space. Nature 238, 44–46 (1972).

    CAS  Google Scholar 

  82. 82

    Schroeder, C. E. et al. Somatosensory input to auditory association cortex in the macaque monkey. J. Neurophysiol. 85, 1322–1327 (2001).

    CAS  Google Scholar 

  83. 83

    Penfield, W. & Rasmussen, T. The Cerebral Cortex of Man: A Clinical Study of Localization of Function (Macmillan, New York, 1950).

    Google Scholar 

  84. 84

    Brindley, G. S. & Lewin, W. S. The sensations produced by electrical stimulation of the visual cortex. J. Physiol. 196, 479–493 (1968).

    CAS  PubMed  PubMed Central  Google Scholar 

  85. 85

    Dobelle, W. H., Stensaas, S. S., Mladejovsky, M. G. & Smith, J. B. A prosthesis for the deaf based on cortical stimulation. Ann. Otol. Rhinol. Laryngol. 82, 445–463 (1973).

    CAS  Google Scholar 

  86. 86

    Penfield, W. Some mechanisms of consciousness discovered during electrical stimulation of the brain. Proc. Natl Acad. Sci. USA 44, 51–66 (1958).

    CAS  Google Scholar 

  87. 87

    Lakatos, P., Chen, C. M., O'Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  88. 88

    McGurk, H. & MacDonald, J. Hearing lips and seeing voices. Nature 264, 746–748 (1976).

    CAS  Google Scholar 

  89. 89

    Sumby, W. H. & Pollack, I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26, 212–215 (1954).

    Google Scholar 

  90. 90

    Sams, M. et al. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci. Lett. 127, 141–145 (1991).

    CAS  Google Scholar 

  91. 91

    Howard, I. P. & Templeton, W. B. Human Spatial Orientation (Wiley, London, 1966).

    Google Scholar 

  92. 92

    Jousmaki, V. & Hari, R. Parchment-skin illusion: sound-biased touch. Curr. Biol. 8, R190 (1998).

    CAS  Google Scholar 

  93. 93

    Shams, L., Kamitani, Y. & Shimojo, S. Visual illusion induced by sound. Brain Res. Cogn. Brain Res. 14, 147–152 (2002).

    Google Scholar 

  94. 94

    Graybiel, A. Oculogravic illusion. AMA Arch. Ophthalmol. 48, 605–615 (1952).

    CAS  Google Scholar 

  95. 95

    Wallace, M. T., Carriere, B. N., Perrault, T. J. Jr, Vaughan, J. W. & Stein, B. E. The development of cortical multisensory integration. J. Neurosci. 26, 11844–11849 (2006).

    CAS  Google Scholar 

  96. 96

    Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444 (1997).

    CAS  Google Scholar 

  97. 97

    Stein, B. E., Labos, E. & Kruger, L. Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J. Neurophysiol. 36, 667–679 (1973).

    CAS  Google Scholar 

  98. 98

    Wallace, M. T. & Stein, B. E. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21, 8886–8894 (2001).

    CAS  Google Scholar 

  99. 99

    Stein, B. E., Wallace, M. W., Stanford, T. R. & Jiang, W. Cortex governs multisensory integration in the midbrain. Neuroscientist 8, 306–314 (2002).

    Google Scholar 

  100. 100

    Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578–3582 (2000).

    CAS  Google Scholar 

  101. 101

    Wallace, M. T., Perrault, T. J. Jr, Hairston, W. D. & Stein, B. E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24, 9580–9584 (2004).

    CAS  Google Scholar 

  102. 102

    Wallace, M. T. & Stein, B. E. Early experience determines how the senses will interact. J. Neurophysiol. 97, 921–926 (2007). This was the first paper to show that the gradual postnatal development of multisensory integration in superior colliculus neurons allows early experience to form its operational principles.

    Google Scholar 

  103. 103

    Rowland, B., Jiang, W. & Stein, B. E. Long-term plasticity in multisensory integration. Soc. Neurosci. Abstr. 31, 505.8 (2005).

    Google Scholar 

  104. 104

    Neil, P. A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D. J. & Shimojo, S. Development of multisensory spatial integration and perception in humans. Dev. Sci. 9, 454–464 (2006).

    Google Scholar 

  105. 105

    Putzar, L., Goerendt, I., Lange, K., Rosler, F. & Roder, B. Early visual deprivation impairs multisensory interactions in humans. Nature Neurosci. 10, 1243–1245 (2007).

    CAS  Google Scholar 

  106. 106

    Stein, B. E. The development of a dialogue between cortex and midbrain to integrate multisensory information. Exp. Brain Res. 166, 305–315 (2005).

    Google Scholar 

  107. 107

    Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11–23 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The author's research is supported in part by NIH grant N536916, EY016716 and EY12389.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Barry E. Stein.

Related links

Related links

FURTHER INFORMATION

Barry Stein's homepage

Terrence Stanford's homepage

Glossary

Multisensory integration

The neural processes that are involved in synthesizing information from cross-modal stimuli. It should not be confused with the particular underlying neural computation that determines multisensory integration's relative magnitude (superadditive, additive or subadditive).

Cross-modal stimuli

Stimuli from two or more sensory modalities or an event providing such stimuli. This term should not be confused with the term 'multisensory'.

Multisensory enhancement

A situation in which the response to the cross-modal stimulus is greater than the response to the most effective of its component stimuli.

Multisensory depression

A situation in which the response to the cross-modal stimulus is less than the response to the most effective of its component stimuli.

Qualia

The qualities of sensation such as the subjective impression that a sensation gives.

Multisensory neuron

A neuron that responds to, or is influenced by, stimuli from more than one sensory modality.

Receptive field

The area of sensory space in which presentation of a stimulus leads to the response of a particular neuron.

Inverse effectiveness

The phenomenon whereby the degree to which a multisensory response exceeds the response to the most effective modality-specific stimulus component declines as the effectiveness of the modality-specific stimulus components increases.

Additivity

A neural computation in which the multisensory response is not different from the arithmetic sum of the responses to the component stimuli.

Subadditivity

A neural computation in which the multisensory response is smaller than the arithmetic sum of the responses to the component stimuli.

Evoked-potential studies

Electrophysiological studies in which the electrical activity (that is, the electrical potential) of the brain in response to a stimulus is measured using scalp-surface electrodes.

Blood-oxygen-level-dependent (BOLD) signal

An index of brain activation based on detecting changes in blood oxygenation with functional MRI.

Superadditivity

A neural computation in which the multisensory response is larger than the arithmetic sum of the responses to the component stimuli.

Summation

A neural computation in which the response to a multisensory stimulus (for example, a number of action potentials) equals the sum of the responses to each of the modality-specific component stimuli presented individually.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Stein, B., Stanford, T. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9, 255–266 (2008). https://doi.org/10.1038/nrn2331

Download citation

Further reading