Hearing visual motion in depth

Abstract

Auditory spatial perception is strongly affected by visual cues. For example, if auditory and visual stimuli are presented synchronously but from different positions, the auditory event is mislocated towards the locus of the visual stimulus—the ventriloquism effect1,2. This ‘visual capture’ also occurs in motion perception in which a static auditory stimulus appears to move with the visual moving object3,4. We investigated how the human perceptual system coordinates complementary inputs from auditory and visual senses. Here we show that an auditory aftereffect occurs from adaptation to visual motion in depth. After a few minutes of viewing a square moving in depth, a steady sound was perceived as changing loudness in the opposite direction. Adaptation to a combination of auditory and visual stimuli changing in a compatible direction increased the aftereffect and the effect of visual adaptation almost disappeared when the directions were opposite. On the other hand, listening to a sound changing in intensity did not affect the visual changing-size aftereffect. The results provide psychophysical evidence that, for processing of motion in depth, the auditory system responds to both auditory changing intensity and visual motion in depth.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Magnitude of the auditory changing-loudness aftereffect after adaptation to combinations of auditory changing-intensity and visual changing-size stimuli.
Figure 2: Schematic illustration of approaching visual stimulus.
Figure 3: Effect of visual adaptation to changing-disparity on the auditory changing-loudness aftereffect.
Figure 4: The visual changing-size aftereffect for four combinations in expanding and shrinking directions.

References

  1. 1

    Jack, C. E. & Thurlow, W. R. Effects of degree of visual association and angle of displacement on the “ventriloquism” effect. Percept. Motor Skill. 37, 967–979 (1973).

    CAS  Google Scholar 

  2. 2

    Bertelson, P. & Radeau, M. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29, 578–584 (1981).

    CAS  Article  Google Scholar 

  3. 3

    Mateeff, S., Hohnsbein, J. & Noack, T. Dynamic visual capture: apparent auditory motion induced by a moving visual target. Perception 14, 721–727 (1985).

    CAS  Article  Google Scholar 

  4. 4

    Kitajima, N. & Yamashita, Y. Dynamic capture of sound motion by light stimuli moving in three-dimensional space. Percept. Motor Skill. 89, 1139–1158 (1999).

    CAS  Article  Google Scholar 

  5. 5

    Wade, N. J. A selective history of the study of visual motion aftereffects. Perception 23, 1111–1134 (1994).

    CAS  Article  Google Scholar 

  6. 6

    Mather, G., Verstraten, F. & Anstis, S. The Motion Aftereffect: A Modern Perspective (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  7. 7

    Grantham, D. W. Motion aftereffects with horizontally moving sound sources in the free field. Percept. Psychophys. 45, 129–136 (1989).

    CAS  Article  Google Scholar 

  8. 8

    Dong, C. J., Swindale, N. V., Zakarauskas, P., Hayward, V. & Cynader, M. S. The auditory motion aftereffect: its tuning and specificity in the spatial and frequency domains. Percept. Psychophys. 62, 1099–1111 (2000).

    CAS  Article  Google Scholar 

  9. 9

    Shu, Z. J., Swindale, N. V. & Cynader, M. S. Spectral motion produces an auditory after-effect. Nature 364, 721–723 (1993).

    ADS  CAS  Article  Google Scholar 

  10. 10

    Reinhardt-Rutland, A. H. & Anstis, S. Auditory adaptation to gradual rise or fall in intensity of a tone. Percept. Psychophys. 31, 63–67 (1982).

    CAS  Article  Google Scholar 

  11. 11

    Reinhardt-Rutland, A. H. Increasing-loudness aftereffect following decreasing-intensity adaptation: spectral dependence in interotic and monotic testing. Perception 27, 473–482 (1998).

    CAS  Article  Google Scholar 

  12. 12

    Kayahara, T. Aftereffect of adaptation to uni-direction frequency change: evidence for selective processing mechanism. Acoust. Sci. Technol. 22, 49–51 (2001).

    Article  Google Scholar 

  13. 13

    Regan, D. & Beverley, K. I. Illusory motion in depth: aftereffect of adaptation to changing size. Vision Res. 18, 209–212 (1978).

    CAS  Article  Google Scholar 

  14. 14

    Cumming, B. in Visual Detection of Motion (eds Smith, A. T. & Snowden, R. J.) 333–366 (Academic, London, 1994).

    Google Scholar 

  15. 15

    Ehrenstein, W. H. & Reinhardt-Rutland, A. H. A cross-modal aftereffect: auditory displacement following adaptation to visual motion. Percept. Motor Skill. 82, 23–26 (1996).

    CAS  Article  Google Scholar 

  16. 16

    Welch, R. B. & Warren, D. H. in Handbook of Perception and Human Performance. Vol. 1 Sensory Processes and Perception (eds Boff, K. R., Kaufman, L. & Thomas, J. P.) Ch. 25 (Wiley, New York, 1986).

    Google Scholar 

  17. 17

    Blauert, J. Spatial Hearing: the Psychophysics of Human Sound Localization Revised edn (MIT Press, Cambridge, Massachusetts, 1997).

    Google Scholar 

  18. 18

    Calvert, G. A. et al. Activation of auditory cortex during silent lipreading. Science 276, 593–596 (1997).

    CAS  Article  Google Scholar 

  19. 19

    Macaluso, E., Frith, C. D. & Driver, J. Modulation of human visual cortex by crossmodal spatial attention. Science 289, 1206–1208 (2000).

    ADS  CAS  Article  Google Scholar 

  20. 20

    Lewis, J. W., Beauchamp, M. S. & DeYoe, E. A. A comparison of visual and auditory motion processing in human cerebral cortex. Cereb. Cortex 10, 873–888 (2000).

    CAS  Article  Google Scholar 

  21. 21

    Driver, J. & Spence, C. Multisensory perception: beyond modularity and convergence. Curr. Biol. 10, R731–R735 (2000).

    CAS  Article  Google Scholar 

  22. 22

    Calvert, G. A., Campbell, R. & Brammer, M. J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10, 649–657 (2000).

    CAS  Article  Google Scholar 

  23. 23

    Brainard, D. H. The psychophysics toolbox. Spatial Vision 10, 433–436 (1997).

    CAS  Article  Google Scholar 

  24. 24

    Pelli, D. G. The Video Toolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision 10, 437–442 (1997).

    CAS  Article  Google Scholar 

  25. 25

    Cornsweet, T. N. The staircase-method in psychophysics. Am. J. Psychol. 75, 485–491 (1962).

    CAS  Article  Google Scholar 

Download references

Acknowledgements

We thank D. Levi, H. Akutsu and D. Erickson for helpful comments and suggestions.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Norimichi Kitagawa.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kitagawa, N., Ichihara, S. Hearing visual motion in depth. Nature 416, 172–174 (2002). https://doi.org/10.1038/416172a

Download citation

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.