Auditory spatial perception is strongly affected by visual cues. For example, if auditory and visual stimuli are presented synchronously but from different positions, the auditory event is mislocated towards the locus of the visual stimulus—the ventriloquism effect1,2. This ‘visual capture’ also occurs in motion perception in which a static auditory stimulus appears to move with the visual moving object3,4. We investigated how the human perceptual system coordinates complementary inputs from auditory and visual senses. Here we show that an auditory aftereffect occurs from adaptation to visual motion in depth. After a few minutes of viewing a square moving in depth, a steady sound was perceived as changing loudness in the opposite direction. Adaptation to a combination of auditory and visual stimuli changing in a compatible direction increased the aftereffect and the effect of visual adaptation almost disappeared when the directions were opposite. On the other hand, listening to a sound changing in intensity did not affect the visual changing-size aftereffect. The results provide psychophysical evidence that, for processing of motion in depth, the auditory system responds to both auditory changing intensity and visual motion in depth.
Subscribe to Journal
Get full journal access for 1 year
only $3.90 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Jack, C. E. & Thurlow, W. R. Effects of degree of visual association and angle of displacement on the “ventriloquism” effect. Percept. Motor Skill. 37, 967–979 (1973).
Bertelson, P. & Radeau, M. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29, 578–584 (1981).
Mateeff, S., Hohnsbein, J. & Noack, T. Dynamic visual capture: apparent auditory motion induced by a moving visual target. Perception 14, 721–727 (1985).
Kitajima, N. & Yamashita, Y. Dynamic capture of sound motion by light stimuli moving in three-dimensional space. Percept. Motor Skill. 89, 1139–1158 (1999).
Wade, N. J. A selective history of the study of visual motion aftereffects. Perception 23, 1111–1134 (1994).
Mather, G., Verstraten, F. & Anstis, S. The Motion Aftereffect: A Modern Perspective (MIT Press, Cambridge, Massachusetts, 1998).
Grantham, D. W. Motion aftereffects with horizontally moving sound sources in the free field. Percept. Psychophys. 45, 129–136 (1989).
Dong, C. J., Swindale, N. V., Zakarauskas, P., Hayward, V. & Cynader, M. S. The auditory motion aftereffect: its tuning and specificity in the spatial and frequency domains. Percept. Psychophys. 62, 1099–1111 (2000).
Shu, Z. J., Swindale, N. V. & Cynader, M. S. Spectral motion produces an auditory after-effect. Nature 364, 721–723 (1993).
Reinhardt-Rutland, A. H. & Anstis, S. Auditory adaptation to gradual rise or fall in intensity of a tone. Percept. Psychophys. 31, 63–67 (1982).
Reinhardt-Rutland, A. H. Increasing-loudness aftereffect following decreasing-intensity adaptation: spectral dependence in interotic and monotic testing. Perception 27, 473–482 (1998).
Kayahara, T. Aftereffect of adaptation to uni-direction frequency change: evidence for selective processing mechanism. Acoust. Sci. Technol. 22, 49–51 (2001).
Regan, D. & Beverley, K. I. Illusory motion in depth: aftereffect of adaptation to changing size. Vision Res. 18, 209–212 (1978).
Cumming, B. in Visual Detection of Motion (eds Smith, A. T. & Snowden, R. J.) 333–366 (Academic, London, 1994).
Ehrenstein, W. H. & Reinhardt-Rutland, A. H. A cross-modal aftereffect: auditory displacement following adaptation to visual motion. Percept. Motor Skill. 82, 23–26 (1996).
Welch, R. B. & Warren, D. H. in Handbook of Perception and Human Performance. Vol. 1 Sensory Processes and Perception (eds Boff, K. R., Kaufman, L. & Thomas, J. P.) Ch. 25 (Wiley, New York, 1986).
Blauert, J. Spatial Hearing: the Psychophysics of Human Sound Localization Revised edn (MIT Press, Cambridge, Massachusetts, 1997).
Calvert, G. A. et al. Activation of auditory cortex during silent lipreading. Science 276, 593–596 (1997).
Macaluso, E., Frith, C. D. & Driver, J. Modulation of human visual cortex by crossmodal spatial attention. Science 289, 1206–1208 (2000).
Lewis, J. W., Beauchamp, M. S. & DeYoe, E. A. A comparison of visual and auditory motion processing in human cerebral cortex. Cereb. Cortex 10, 873–888 (2000).
Driver, J. & Spence, C. Multisensory perception: beyond modularity and convergence. Curr. Biol. 10, R731–R735 (2000).
Calvert, G. A., Campbell, R. & Brammer, M. J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10, 649–657 (2000).
Brainard, D. H. The psychophysics toolbox. Spatial Vision 10, 433–436 (1997).
Pelli, D. G. The Video Toolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision 10, 437–442 (1997).
Cornsweet, T. N. The staircase-method in psychophysics. Am. J. Psychol. 75, 485–491 (1962).
We thank D. Levi, H. Akutsu and D. Erickson for helpful comments and suggestions.
The authors declare no competing financial interests.
About this article
Cite this article
Kitagawa, N., Ichihara, S. Hearing visual motion in depth. Nature 416, 172–174 (2002). https://doi.org/10.1038/416172a
Attention, Perception, & Psychophysics (2020)
Current Biology (2020)
Asymmetrical characteristics of emotional responses to pictures and sounds: Evidence from pupillometry
PLOS ONE (2020)
Quality assessment of acoustic environment reproduction methods for cinematic virtual reality in soundscape applications
Building and Environment (2019)
Congruent audio-visual stimulation during adaptation modulates the subsequently experienced visual motion aftereffect
Scientific Reports (2019)