Integration of multiple sensory cues is essential for precise and accurate perception and behavioral performance, yet the reliability of sensory signals can vary across modalities and viewing conditions. Human observers typically employ the optimal strategy of weighting each cue in proportion to its reliability, but the neural basis of this computation remains poorly understood. We trained monkeys to perform a heading discrimination task from visual and vestibular cues, varying cue reliability randomly. The monkeys appropriately placed greater weight on the more reliable cue, and population decoding of neural responses in the dorsal medial superior temporal area closely predicted behavioral cue weighting, including modest deviations from optimality. We found that the mathematical combination of visual and vestibular inputs by single neurons is generally consistent with recent theories of optimal probabilistic computation in neural circuits. These results provide direct evidence for a neural mechanism mediating a simple and widespread form of statistical inference.
At a glance
- The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004). &
- Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002). &
- Slant from texture and disparity cues: optimal cue combination. J. Vis. 4, 967–992 (2004). , , &
- Optimal integration of texture and motion cues to depth. Vision Res. 39, 3621–3629 (1999).
- Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558 (2003). &
- Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35, 389–412 (1995). , , &
- When feeling is more important than seeing in sensorimotor adaptation. Curr. Biol. 12, 834–837 (2002). , &
- A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Res. 33, 2685–2696 (1993). , &
- Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Boston, 1990). &
- Problems arising in the analysis of a series of similar experiments. J. R. Stat. Soc. 4 (suppl.) 102–118 (1937).
- Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11, 1201–1210 (2008). , &
- A functional link between area MSTd and heading perception based on vestibular signals. Nat. Neurosci. 10, 1038–1047 (2007). , &
- Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601–15612 (2009). , , &
- Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006). , , &
- Ideal-observer models of cue integration. in Sensory Cue Integration (eds. Trommershäuser, J., Kording, K.P. & Landy, M.S.) 5–29 (Oxford University Press, New York, 2011). , &
- Perception as Bayesian Inference (Cambridge University Press, New York, 1996). &
- Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73–85 (2006). , , &
- Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391–1397 (2003). , &
- Texture and haptic cues in slant discrimination: reliability-based cue weighting without statistically optimal cue combination. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 22, 801–809 (2005). , , &
- MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816–1827 (1998).
- Multimodal coding of three-dimensional rotation and translation in area MSTd: comparison of visual and vestibular selectivity. J. Neurosci. 27, 9742–9756 (2007). et al.
- Theoretical Neuroscience (MIT press, Cambridge, Massachusetts, 2001). &
- The 'ideal homunculus': statistical inference from neural population responses. in Computation and Neural Systems (eds. F.H. Eeckman & J.M. Bower) 55–60 (Kluwer Academic Publishers, Norwell, Massachusetts, 1993).
- Probability density estimation for the interpretation of neural population codes. J. Neurophysiol. 76, 2790–2793 (1996).
- Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008). , &
- Responses of neurons in macaque MT to stochastic motion signals. Vis. Neurosci. 10, 1157–1169 (1993). , , &
- Linear responses to stochastic motion signals in area MST. J. Neurophysiol. 98, 1115–1124 (2007). &
- A normalization model of multisensory integration. Nat. Neurosci. 14, 775–782 (2011). , &
- The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004). &
- The Merging of the Senses (MIT Press, Cambridge, Massachusetts, 1993). &
- Visual-vestibular interactive responses in the macaque ventral intraparietal area (VIP). Eur. J. Neurosci. 16, 1569–1586 (2002). , , , &
- Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex. J. Neurosci. 31, 12036–12052 (2011). , &
- Corticovestibular interactions: anatomy, electrophysiology and functional considerations. Exp. Brain Res. 117, 1–16 (1997).
- Radial motion bias in macaque frontal eye field. Vis. Neurosci. 23, 49–60 (2006). , &
- Corticocortical connections of visual, sensorimotor and multimodal processing areas in the parietal lobe of the macaque monkey. J. Comp. Neurol. 428, 112–137 (2000). &
- Topography of projections to posterior cortical areas from the macaque frontal eye fields. J. Comp. Neurol. 353, 291–305 (1995). , &
- Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991). &
- The connections of the middle temporal visual area (MT) and their relationship to a cortical hierarchy in the macaque monkey. J. Neurosci. 3, 2563–2586 (1983). &
- Optic flow processing in monkey STS: a theoretical and experimental approach. J. Neurosci. 16, 6265–6285 (1996). , , , &
- Extraction of visual motion and optic flow. Neural Netw. 21, 774–785 (2008).
- Emulating the visual receptive-field properties of MST neurons with a template model of heading estimation. J. Neurosci. 18, 5958–5975 (1998). &
- Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J. Vis. 7, 1–24 (2007).
- Bayesian integration of spatial information. Psychol. Bull. 133, 625–637 (2007). , , &
- Causal inference in multisensory perception. PLoS ONE 2, e943 (2007). et al.
- Relation of cortical areas MT and MST to pursuit eye movements. I. Localization and visual properties of neurons. J. Neurophysiol. 60, 580–603 (1988). &
- Analysis of motion of the visual field by direction, expansion/contraction, and rotation cells clustered in the dorsal part of the medial superior temporal area of the macaque monkey. J. Neurophysiol. 62, 626–641 (1989). &
- Spatiotemporal properties of vestibular responses in area MSTd. J. Neurophysiol. 104, 1506–1522 (2010). et al.
- The psychometric function. I. Fitting, sampling, and goodness of fit. Percept. Psychophys. 63, 1293–1313 (2001). &
- Optimal representation of sensory information by neural populations. Nat. Neurosci. 9, 690–696 (2006). &
- Decoding of MSTd population activity accounts for variations in the precision of heading perception. Neuron 66, 596–609 (2010). , , , &
- Supplementary Text and Figures (2 MB)
Supplementary Figures 1–6 and Supplementary Analysis