Key Points
-
Integration of multiple sensory cues is crucial for adaptive behaviour, but understanding its neural basis has been hampered by a divergence in the literature into two largely non-overlapping conceptual approaches. One approach seeks to predict and quantify cue integration behaviour in a rigorous psychophysical framework, whereas the other has focused on the empirical principles that govern multisensory integration by neurons, including its anatomical and developmental origins.
-
Tasked with reconciling these camps are the computational theorists who strive to create models of multisensory processing that are both biologically realistic and capable of explaining psychophysical performance. Progress has been made on this front, an example of which is the theory of probabilistic population coding. However, several obstacles remain, such as accounting for key differences between model predictions and neurophysiological data and a lack of neuronal recordings from behaving animals performing psychophysical cue integration tasks.
-
A recent set of studies on the integration of visual and vestibular cues for self-motion perception draws from both sets of approaches and has attempted to narrow the conceptual gap between them. This work uses macaque monkeys trained to report their heading direction in a virtual-reality apparatus, while researchers monitor single-unit activity in a multisensory region of the extrastriate visual cortex (the dorsal medial superior temporal area (MSTd)).
-
The emerging picture is one in which computations performed by multisensory neurons in area MSTd can be linked to psychophysical performance and may also help to explain the empirical principles that have driven the field for two decades. Divisive normalization, a common property of neuronal circuits, may be one of the key computations that give rise to the desired behaviour.
-
Further work will be needed to develop more complete models that connect the neurobiological details of multisensory integration with the probabilistic computations underlying cue integration performance and perception in general.
Abstract
The richness of perceptual experience, as well as its usefulness for guiding behaviour, depends on the synthesis of information across multiple senses. Recent decades have witnessed a surge in our understanding of how the brain combines sensory cues. Much of this research has been guided by one of two distinct approaches: one is driven primarily by neurophysiological observations, and the other is guided by principles of mathematical psychology and psychophysics. Conflicting results and interpretations have contributed to a conceptual gap between psychophysical and physiological accounts of cue integration, but recent studies of visual–vestibular cue integration have narrowed this gap considerably.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$189.00 per year
only $15.75 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Faisal, A. A., Selen, L. P. & Wolpert, D. M. Noise in the nervous system. Nature Rev. Neurosci. 9, 292–303 (2008).
Gepshtein, S. Two psychologies of perception and the prospect of their synthesis. Philos. Psychol. 23, 217–281 (2010).
Knill, D. C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004). A concise review that provides a good introduction to the idea of sensory uncertainty and the Bayesian perspective on behaviour and neural coding, including the incorporation of priors and studies of motor control.
Einstein, A. Über das Relativitätsprinzip und die aus demselben gezogenen Folgerungen. Jahrbuch Radioaktivität Elektronik 4, 411–462 (1907).
Angelaki, D. E., Shaikh, A. G., Green, A. M. & Dickman, J. D. Neurons compute internal models of the physical laws of motion. Nature 430, 560–564 (2004).
Merfeld, D., Zupan, L. & Peterka, R. Humans use internal models to estimate gravity and linear acceleration. Nature 398, 615–618 (1999).
Landy, M. S., Banks, M. S. & Knill, D. C. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 5–29 (Oxford Univ. Press, 2011).
Angelaki, D. E., Gu, Y. & DeAngelis, G. C. Multisensory integration: psychophysics, neurophysiology, and computation. Curr. Opin. Neurobiol. 19, 452–458 (2009).
Kajikawa, Y., Falchier, A., Musacchia, G., Lakatos, P. & Schroeder, C. E. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).
Lakatos, P., Chen, C. M., O'Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292 (2007).
Kayser, C., Petkov, C. I., Remedios, R. & Logothetis, N. K. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).
Kayser, C. & Logothetis, N. K. Do early sensory cortices integrate cross-modal information? Brain Struct. Funct. 212, 121–132 (2007).
Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).
Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11–23 (2008).
Geisler, W. S. Contributions of ideal observer theory to vision research. Vision Res. 51, 771–781 (2011).
Jacobs, R. A. Optimal integration of texture and motion cues to depth. Vision Res. 39, 3621–3629 (1999).
Landy, M. S., Maloney, L. T., Johnston, E. B. & Young, M. Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35, 389–412 (1995). Focusing on the array of visual cues available for the perception of depth, this paper develops several key ideas underlying contemporary ideal observer models of cue integration and also introduces a psychophysical procedure that has become a standard method for testing such models.
Maloney, L. T. & Landy, M. S. in Proceedings of the SPIE (ed. Pearlman, W. A.) 1154–1163 (SPIE, 1989).
Clark, J. J. & Yuille, A. L. Data Fusion For Sensory Information Processing Systems (Kluwer, 1990).
Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).
Battaglia, P. W., Jacobs, R. A. & Aslin, R. N. Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391–1397 (2003).
Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002). This study provides one of the earliest and clearest psychophysical demonstrations of optimal cue integration across separate sensory modalities. The authors showed that human subjects integrate vision and touch to estimate the width of a grasped object by taking into account the relative reliability of the cues and combining them to improve their performance. Importantly, cue reliability was varied randomly from trial to trial, suggesting that the brain may not need to explicitly learn or represent the uncertainty of the cues to accomplish the task.
Sober, S. J. & Sabes, P. N. Flexible strategies for sensory integration during motor planning. Nature Neurosci. 8, 490–497 (2005).
van Beers, R. J., Sittig, A. C. & Gon, J. J. Integration of proprioceptive and visual position-information: an experimentally supported model. J. Neurophysiol. 81, 1355–1364 (1999).
Körding, K. P. & Wolpert, D. M. Bayesian decision theory in sensorimotor control. Trends Cogn. Sci. 10, 319–326 (2006).
Gu, Y., Angelaki, D. E. & DeAngelis, G. C. Neural correlates of multisensory cue integration in macaque MSTd. Nature Neurosci. 11, 1201–1210 (2008). Using a visual–vestibular heading discrimination task, this study showed that monkeys, like humans, are capable of combining sensory cues to improve perceptual performance. The authors also characterized a population of neurons in extrastriate visual cortex (specifically, in area MSTd) that could underlie the behaviour.
de Winkel, K. N., Weesie, J., Werkhoven, P. J. & Groen, E. L. Integration of visual and inertial cues in perceived heading of self-motion. J. Vis. 10, 1 (2010).
Edwards, M., O'Mahony, S., Ibbotson, M. R. & Kohlhagen, S. Vestibular stimulation affects optic-flow sensitivity. Perception 39, 1303–1310 (2010).
Butler, J. S., Smith, S. T., Campos, J. L. & Bulthoff, H. H. Bayesian integration of visual and vestibular signals for heading. J. Vis. 10, 23 (2010).
Fetsch, C. R., Turner, A. H., DeAngelis, G. C. & Angelaki, D. E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601–15612 (2009).
Cochran, W. G. Problems arising in the analysis of a series of similar experiments. Appl. Stat. 4, 102–118 (1937).
Yuille, A. L. & Bülthoff, H. H. in Perception as Bayesian Inference (eds Knill, D. C. & Richards, W.) 123–162 (Cambridge Univ. Press, 1996).
Young, M. J., Landy, M. S. & Maloney, L. T. A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Res. 33, 2685–2696 (1993).
Knill, D. C. & Saunders, J. A. Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558 (2003).
Hillis, J. M., Watt, S. J., Landy, M. S. & Banks, M. S. Slant from texture and disparity cues: optimal cue combination. J. Vis. 4, 967–992 (2004).
Clemens, I. A., De Vrijer, M., Selen, L. P., Van Gisbergen, J. A. & Medendorp, W. P. Multisensory processing in spatial orientation: an inverse probabilistic approach. J. Neurosci. 31, 5365–5377 (2011).
Ma, W., Beck, J. M. & Pouget, A. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 393–405 (Oxford Univ. Press, 2011).
Oruc, I., Maloney, L. T. & Landy, M. S. Weighted linear cue combination with possibly correlated error. Vision Res. 43, 2451–2468 (2003).
Rosas, P., Wagemans, J., Ernst, M. O. & Wichmann, F. A. Texture and haptic cues in slant discrimination: reliability-based cue weighting without statistically optimal cue combination. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 22, 801–809 (2005).
Rosas, P., Wichmann, F. A. & Wagemans, J. Texture and object motion in slant discrimination: failure of reliability-based weighting of cues may be evidence for strong fusion. J. Vis. 7, 3 (2007).
Körding, K. P. et al. Causal inference in multisensory perception. PLoS ONE 2, e943 (2007).
Knill, D. C. Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J. Vis. 7, 5 (2007).
Ernst, M. O. & Di Luca, M. in Sensory Cue Integration (eds Trommershäuser, J., Landy, M. S. & Kording, K. P.) 224–250 (Oxford Univ. Press, 2011).
Zaidel, A., Turner, A. H. & Angelaki, D. E. Multisensory calibration is independent of cue reliability. J. Neurosci. 31, 13949–13962 (2011).
Raposo, D., Sheppard, J. P., Schrater, P. R. & Churchland, A. K. Multisensory decision-making in rats and humans. J. Neurosci. 32, 3726–3735 (2012).
Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).
Meredith, M. & Stein, B. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640–662 (1986). This paper is among the first to demonstrate the impressive capacity of SC neurons to combine visual, tactile and auditory cues, yielding multisensory responses that were often considerably enhanced (and sometimes suppressed) relative to unisensory responses. These early observations laid the foundation for the well-known empirical 'principles' of multisensory integration, including the spatial and temporal principle and inverse effectiveness.
Sparks, D. L. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol. Rev. 66, 118–171 (1986).
Stein, B. E. Development of the superior colliculus. Annu. Rev. Neurosci. 7, 95–125 (1984).
Wurtz, R. H. & Albano, J. E. Visual-motor function of the primate superior colliculus. Annu. Rev. Neurosci. 3, 189–226 (1980).
Stein, B. E. & Stanford, T. R. Multisensory integration: current issues from the perspective of the single neuron. Nature Rev. Neurosci. 9, 255–266 (2008).
James, T. W. & Stevenson, R. A. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).
Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823 (2004).
Sarko, D. K. et al. in The Neural Bases of Multisensory Processes (eds Murray, M. M. & Wallace, M. T.) (CRC Press, 2012).
Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005). This study explores multisensory interactions in the SC in a more systematic manner than previous work by varying the intensity and timing of visual and auditory stimuli. The results suggest that most SC neurons combine their inputs additively and that the oft-cited phenomenon of 'superadditivity' may only occur for very weak stimuli.
Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240–1255 (2002).
Stein, B. E., Huneycutt, W. S. & Meredith, M. A. Neurons and behavior: the same rules of multisensory integration apply. Brain Res. 448, 355–358 (1988).
Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1–10 (1996).
Frens, M. A., Van Opstal, A. J. & Van der Willigen, R. F. Spatial and temporal factors determine auditory–visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802–816 (1995).
Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797–1809 (1993).
Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001).
Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444 (1997).
Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578–3582 (2000).
Wallace, M. T. & Stein, B. E. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21, 8886–8894 (2001).
Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J. Neurophysiol. 90, 4022–4026 (2003).
Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).
Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575–2586 (2005).
Cuppini, C., Ursino, M., Magosso, E., Rowland, B. A. & Stein, B. E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).
Patton, P. E. & Anastasio, T. J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783–810 (2003).
Alvarado, J. C., Rowland, B. A., Stanford, T. R. & Stein, B. E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13–23 (2008).
Rowland, B. A., Stein, B. E. & Stanford, T. R. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 333–344 (Oxford Univ. Press, 2011).
Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966).
Kim, B. & Basso, M. A. Saccade target selection in the superior colliculus: a signal detection theory approach. J. Neurosci. 28, 2991–3007 (2008).
Kim, B. & Basso, M. A. A probabilistic strategy for understanding action selection. J. Neurosci. 30, 2340–2355 (2010).
Ma, W. J., Beck, J. M., Latham, P. E. & Pouget, A. Bayesian inference with probabilistic population codes. Nature Neurosci. 9, 1432–1438 (2006). This theoretical study outlines a simple and flexible strategy for performing optimal Bayesian inference with populations of neurons, using multisensory cue integration as a primary example. By exploiting a mathematical property of neuronal noise, the authors show that simple summation of population activity in sensory areas can be sufficient to implement Bayes-optimal cue integration.
Beck, J. M. et al. Probabilistic population codes for Bayesian decision making. Neuron 60, 1142–1152 (2008).
Jazayeri, M. & Movshon, J. A. Optimal representation of sensory information by neural populations. Nature Neurosci. 9, 690–696 (2006).
Ma, W. J. & Pouget, A. Linking neurons to behavior in multisensory perception: a computational review. Brain Res. 1242, 4–12 (2008).
Populin, L. C. & Yin, T. C. Bimodal interactions in the superior colliculus of the behaving cat. J. Neurosci. 22, 2826–2834 (2002).
Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18, 787–792 (2007).
Perrault, T. J. Jr, Stein, B. E. & Rowland, B. A. Non-stationarity in multisensory neurons in the superior colliculus. Front. Psychol. 2, 144 (2011).
Gibson, J. J. The Perception of the Visual World (Houghton-Mifflin, 1950).
Warren, W. H. Jr, Morris, M. W. & Kalish, M. Perception of translational heading from optical flow. J. Exp. Psychol. Hum. Percept. Perform. 14, 646–660 (1988).
Benson, A. J., Spencer, M. B. & Stott, J. R. Thresholds for the detection of the direction of whole-body, linear movement in the horizontal plane. Aviat. Space Environ. Med. 57, 1088–1096 (1986).
Fernandez, C. & Goldberg, J. M. Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. II. Directional selectivity and force-response relations. J. Neurophysiol. 39, 985–995 (1976).
Fernandez, C. & Goldberg, J. M. Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. I. Response to static tilts and to long-duration centrifugal force. J. Neurophysiol. 39, 970–984 (1976).
Guedry, F. E. in Handbook of Sensory Physiology, The Vestibular System (ed. Kornhuber, H. H. ) (Springer, 1974).
Chen, A., DeAngelis, G. C. & Angelaki, D. E. Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex. J. Neurosci. 31, 12036–12052 (2011).
Gu, Y., Watkins, P. V., Angelaki, D. E. & DeAngelis, G. C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73–85 (2006).
Chen, A., DeAngelis, G. C. & Angelaki, D. E. Convergence of vestibular and visual self-motion signals in an area of the posterior sylvian fissure. J. Neurosci. 31, 11617–11627 (2011).
Duffy, C. J. MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816–1827 (1998).
Britten, K. H. & van Wezel, R. J. Electrical microstimulation of cortical area MST biases heading perception in monkeys. Nature Neurosci. 1, 59–63 (1998).
Gu, Y., DeAngelis, G. C. & Angelaki, D. E. Causal links between dorsal medial superior temporal area neurons and multisensory heading perception. J. Neurosci. 32, 2299–2313 (2012).
Morgan, M. L., DeAngelis, G. C. & Angelaki, D. E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008). The authors recorded neuronal responses in area MSTd to a wide array of stimulus combinations, and found a simple mathematical rule by which multisensory neurons combine their inputs. Neurons appeared to take a weighted sum of their inputs, with weights that depend on cue reliability — a surprising finding that conflicted with theoretical predictions and was not well understood until later.
Holmes, N. P. & Spence, C. Multisensory integration: space, time and superadditivity. Curr. Biol. 15, R762–R764 (2005).
Parker, A. J. & Newsome, W. T. Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227–277 (1998).
Bradley, A., Skottun, B. C., Ohzawa, I., Sclar, G. & Freeman, R. D. Visual orientation and spatial frequency discrimination: a comparison of single neurons and behavior. J. Neurophysiol. 57, 755–772 (1987).
Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966).
Britten, K. H., Shadlen, M. N., Newsome, W. T. & Movshon, J. A. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745–4765 (1992).
Fetsch, C. R., Pouget, A., DeAngelis, G. C. & Angelaki, D. E. Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neurosci. 15, 146–154 (2011). In this study, the authors recorded from area MSTd while monkeys performed a cue-conflict version of the visual–vestibular heading task. Behaviourally, the animals were able to re-weight the cues on a trial-by-trial basis as reliability varied, and neuronal activity accounted for the behavioural results. The authors also derived a mathematically optimal combination rule for this task and used it to help explain deviations from optimality at the level of behaviour and neural responses.
Paradiso, M. A. A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biol. Cybern. 58, 35–49 (1988).
Seung, H. S. & Sompolinsky, H. Simple models for reading neuronal population codes. Proc. Natl Acad. Sci. USA 90, 10749–10753 (1993).
Carandini, M. & Heeger, D. J. Normalization as a canonical neural computation. Nature Rev. Neurosci. 13, 51–62 (2011). This review summarizes the large and diverse literature on the divisive normalization, including candidate biophysical mechanisms as well as its benefits for the efficiency of neural coding.
Louie, K., Grattan, L. E. & Glimcher, P. W. Reward value-based gain control: divisive normalization in parietal cortex. J. Neurosci. 31, 10627–10639 (2011).
Ohshiro, T., Angelaki, D. E. & DeAngelis, G. C. A normalization model of multisensory integration. Nature Neurosci. 14, 775–782 (2011). This paper presents a computational model that relies on a widespread neural computation called divisive normalization. Normalization of responses at the level of multisensory integration helps to explain several key empirical findings: the reliability-dependent combination rule in area MSTd as well as the ubiquitous empirical principles that were initially described in classic studies of the SC.
Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).
Carandini, M., Heeger, D. & Movshon, J. Linearity and normalization in simple cells of the macaque primary visual cortex. J. Neurosci. 17, 8621–8644 (1997).
Carandini, M. From circuits to behavior: a bridge too far? Nature Neurosci. 15, 507–509 (2012).
Beauchamp, M. S., Pasalar, S. & Ro, T. Neural substrates of reliability-weighted visual-tactile multisensory integration. Front. Syst. Neurosci. 4, 25 (2010).
Helbig, H. B. et al. The neural mechanisms of reliability weighted integration of shape information from vision and touch. Neuroimage 60, 1063–1072 (2012).
Takahashi, K. et al. Multimodal coding of three-dimensional rotation and translation in area MSTd: comparison of visual and vestibular selectivity. J. Neurosci. 27, 9742–9756 (2007).
Gu, Y., DeAngelis, G. C. & Angelaki, D. E. A functional link between area MSTd and heading perception based on vestibular signals. Nature Neurosci. 10, 1038–1047 (2007).
Chen, A., DeAngelis, G. C. & Angelaki, D. E. A comparison of vestibular spatiotemporal tuning in macaque parietoinsular vestibular cortex, ventral intraparietal area, and medial superior temporal area. J. Neurosci. 31, 3082–3094 (2011).
Shams, L., Kamitani, Y. & Shimojo, S. Illusions. What you see is what you hear. Nature 408, 788 (2000).
Shimojo, S. & Shams, L. Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11, 505–509 (2001).
Sekuler, R., Sekuler, A. B. & Lau, R. Sound alters visual motion perception. Nature 385, 308 (1997).
Avillac, M., Denève, S., Olivier, E., Pouget, A. & Duhamel, J.-R. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neurosci. 8, 941–949 (2005).
Wallace, M. T. et al. Unifying multisensory signals across time and space. Exp. Brain Res. 158, 252–258 (2004).
Fetsch, C. R., DeAngelis, G. C. & Angelaki, D. E. Visual–vestibular cue integration for heading perception: applications of optimal cue integration theory. Eur. J. Neurosci. 31, 1721–1729 (2010).
Acknowledgements
We acknowledge the prodigious contributions of Y. Gu (Institute of Neuroscience, Shanghai, China) to the research discussed in this Review. We also thank M. Morgan, T. Ohshiro and many other current and former laboratory members, collaborators and technicians who made the work possible. Research was supported by US National Institutes of Health grants R01-EY016178 (to G.C.D.) and R01-EY019087 (to D.E.A.).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Related links
Glossary
- Normative
-
A general term referring to an idea, statement or model that describes how something ought to be: that is, relating to an ideal or standard of correctness.
- Cue
-
Any signal or piece of information bearing on the state of some property of the environment. Examples include binocular disparity in the visual system, interaural time or level differences in audition and proprioceptive signals (for example, from muscle spindles) conveying the position of the arm in space.
- Ideal observer
-
A theoretical construct used to quantify optimal performance in a given task, where optimality is defined according to a pre-defined mathematical function (for example, minimizing a cost function or maximizing a utility function). The term 'ideal' does not imply perfect (error-free) performance, which is generally impossible given the uncertainty associated with all sensory data.
- Reliability
-
Although the term reliability can mean different things in different fields, here we use it as a synonym for the precision of a measurement, which is defined mathematically as its inverse variance.
- Bayesian probability theory
-
The branch of statistics and probability theory in which probability is interpreted as the 'degree of belief' that an event will occur (or that a hypothesis is true) rather than the relative frequency with which it has occurred. It is chiefly associated with the process of updating a prior belief about a hypothesis in light of new data, but the essence of Bayesian theory is this way of thinking about probability itself, which permits the estimation of a statistical parameter (or property of the environment) from experimental observations (or sensory information).
- Accuracy
-
Accuracy refers to how close the measurement is to the true value: that is, how unbiased it is.
- Poisson-like variability
-
Neurons respond differently to repeated presentations of the same stimulus, and this variability often resembles a family of probability distributions that includes the Poisson distribution (hence the term 'Poisson-like'). A prominent feature of this family is that the variance of neuronal responses (that is, the variance of the number of action potentials across repeated stimulus presentations) is proportional to the mean response.
- Heading
-
An organism's instantaneous direction of translational movement.
- Motion coherence
-
A property of random-dot motion stimuli — used in visual psychophysics and neurophysiology — that is often varied to control stimulus strength and therefore task difficulty. Motion coherence is the percentage of dots moving in the prescribed direction (the 'signal') while the remaining dots are re-plotted randomly on every video frame (the 'noise').
- Divisive normalization
-
A neural computation in which the would-be response of an individual neuron (that is, its excitatory drive) is divided by the summed activity of a pool of neurons before generating an output.
Rights and permissions
About this article
Cite this article
Fetsch, C., DeAngelis, G. & Angelaki, D. Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci 14, 429–442 (2013). https://doi.org/10.1038/nrn3503
Published:
Issue Date:
DOI: https://doi.org/10.1038/nrn3503
This article is cited by
-
Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex
Nature Neuroscience (2024)
-
Multisensory learning binds neurons into a cross-modal memory engram
Nature (2023)
-
Sensory fusion in the hoverfly righting reflex
Scientific Reports (2023)
-
How deep is the brain? The shallow brain hypothesis
Nature Reviews Neuroscience (2023)
-
Bayesian encoding and decoding as distinct perspectives on neural coding
Nature Neuroscience (2023)