Abstract
To perceive the auditory and visual aspects of a physical event as occurring simultaneously, the brain must adjust for differences between the two modalities in both physical transmission time and sensory processing time. One possible strategy to overcome this difficulty is to adaptively recalibrate the simultaneity point from daily experience of audiovisual events. Here we report that after exposure to a fixed audiovisual time lag for several minutes, human participants showed shifts in their subjective simultaneity responses toward that particular lag. This 'lag adaptation' also altered the temporal tuning of an auditory-induced visual illusion, suggesting that adaptation occurred via changes in sensory processing, rather than as a result of a cognitive shift while making task responses. Our findings suggest that the brain attempts to adjust subjective simultaneity across different modalities by detecting and reducing time lags between inputs that likely arise from the same physical events.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Pöppel, E. Grenzen des Bewuβtseins: Über Wirklichkeit und Welterfahrung (Deutsche Verlags-Anstalt GmbH, Stuttgart, 1985).
Spence, C. & Squire, S. Multisensory integration: maintaining the perception of synchrony. Curr. Biol. 13, R519–R521 (2003).
King, A.J. & Palmer, A.R. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain. Res. 60, 492–500 (1985).
Regan, D. Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine (Elsevier, New York, 1989).
Tappe, T., Niepel, M. & Neumann, O. A dissociation between reaction time to sinusoidal gratings and temporal-order judgment. Perception 23, 335–347 (1994).
Stone, J.V. et al. When is now? Perception of simultaneity. Proc. R. Soc. Lond. B Biol. Sci. 268, 31–38 (2001).
Sugita, Y. & Suzuki, Y. Audiovisual perception: implicit estimation of sound-arrival time. Nature 421, 911 (2003).
James, W. Principles of Psychology (Holt, New York, 1890).
Spence, C., Shore, D.I. & Klein, R.M. Multisensory prior entry. J. Exp. Psychol. Gen. 130, 799–832 (2001).
Sekuler, R., Sekuler, A.B. & Lau, R. Sound alters visual motion perception. Nature 385, 308 (1997).
Watanabe, K. & Shimojo, S. When sound affects vision: effects of auditory grouping on visual motion perception. Psychol. Sci. 12, 109–116 (2001).
Shimojo, S. & Shams, L. Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11, 505–509 (2001).
Dragoi, V., Rivadulla, C. & Sur, M. Foci of orientation plasticity in visual cortex. Nature 411, 80–86 (2001).
Mather, G., Verstraten, F.A.J. & Anstis, S.M. The Motion Aftereffect: A Modern Perspective (MIT Press, Cambridge, Massachusetts, 1998).
Kashino, M. & Nishida, S. Adaptation in the processing of interaural time differences revealed by the auditory localization aftereffect. J. Acoust. Soc. Am. 103, 3597–3604 (1998).
Dolezal, H. Living in a World Transformed: Perceptual and Performatory Adaptation to Visual Distortion (Academic, New York, 1982).
Barlow, H.B. & Földiák, P. Adaptation and decorrelation in the cortex. in The Computing Neuron (eds. Durbin, R., Miall, C. & Mitchison, G.) 54–72 (Addison-Wesley, Boston, 1989).
Lewald, J. & Guski, R. Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neurosci. Lett. 357, 119–122 (2004).
Recanzone, G.H. Auditory influences on visual temporal rate perception. J. Neurophysiol. 89, 1078–1093 (2003).
Bennett, R.G. & Westheimer, G. A shift in the perceived simultaneity of adjacent visual stimuli following adaptation to stroboscopic motion along the same axis. Vision Res. 25, 565–569 (1985).
Okada, M. & Kashino, M. The role of spectral change detectors in temporal order judgment of tones. Neuroreport 14, 261–264 (2003).
Cunningham, D.W., Billock, V.A. & Tsou, B.H. Sensorimotor adaptation to violations of temporal contiguity. Psychol. Sci. 12, 532–535 (2001).
Cunningham, D.W., Chatziastros, A., von der Heyde, M. & Bulthoff, H.H. Driving in the future: temporal visuomotor adaptation and generalization. J. Vis. 1, 88–98 (2001).
Canon, L.K. Intermodality inconsistency of input and directed attention as determinants of the nature of adaptation. J. Exp. Psychol. 84, 141–147 (1970).
Recanzone, G.H. Rapidly induced auditory plasticity: the ventriloquism aftereffect. Proc. Natl. Acad. Sci. USA 95, 869–875 (1998).
Knudsen, E.I. & Knudsen, P.F. Vision guides the adjustment of auditory localization in young barn owls. Science 230, 545–548 (1985).
Zwiers, M.P., Van Opstal, A.J. & Paige, G.D. Plasticity in human sound localization induced by compressed spatial vision. Nat. Neurosci. 6, 175–181 (2003).
Shams, L., Kamitani, Y. & Shimojo, S. Illusions. What you see is what you hear. Nature 408, 788 (2000).
Kitagawa, N. & Ichihara, S. Hearing visual motion in depth. Nature 416, 172–174 (2002).
Foder, J.A. The Modularity of Mind (MIT Press, Cambridge, Massachusetts, 1983).
Dennett, D.C. & Kinsbourne, M. Time and the observer: the where and when of consciousness in the brain. Behav. Brain Sci. 15, 183–247 (1992).
Nishida, S. & Johnston, A. Marker correspondence, not processing latency, determines temporal binding of visual attributes. Curr. Biol. 12, 359–368 (2002).
Jeffress, L.A. A place theory of sound localization. J. Comp. Physiol. Psychol. 41, 35–39 (1948).
McAlpine, D. & Grothe, B. Sound localization and delay lines – do mammals fit the model? Trends Neurosci. 26, 347–350 (2003).
Reichardt, W. Autocorrelation, a principle for the evaluation of sensory information by the central nervous system. in Sensory Communication (ed. Rosenblith, W.A.) 303–317 (MIT Press, Cambridge, Massachusetts, 1961).
Lu, Z.L. & Sperling, G. Three-systems theory of human visual motion perception: review and update. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 18, 2331–2370 (2001).
Barlow, H.B. & Hill, R.M. Evidence for a physiological explanation of the waterfall phenomenon and figural after-effects. Nature 200, 1345–1347 (1963).
Bushara, K.O., Grafman, J. & Hallett, M. Neural correlates of auditory-visual stimulus onset asynchrony detection. J. Neurosci. 21, 300–304 (2001).
Bushara, K.O. et al. Neural correlates of cross-modal binding. Nat. Neurosci. 6, 190–195 (2003).
Meredith, M.A., Nemitz, J.W. & Stein, B.E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7, 3215–3229 (1987).
Corneil, B.D., Van Wanrooij, M., Munoz, D.P. & Van Opstal, A.J. Auditory-visual interactions subserving goal-directed saccades in a complex scene. J. Neurophysiol. 88, 438–454 (2002).
Brainard, D.H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).
Pelli, D.G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10, 437–442 (1997).
Watson, A.B. Probability summation over time. Vision Res. 19, 515–522 (1979).
Acknowledgements
We thank D. Arnold, M. Changizi, T. Hirahara, A. Johnston and D. Wu. This work was partially supported by the Human Frontier Science Program (RGP0070/2003-C).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Rights and permissions
About this article
Cite this article
Fujisaki, W., Shimojo, S., Kashino, M. et al. Recalibration of audiovisual simultaneity. Nat Neurosci 7, 773–778 (2004). https://doi.org/10.1038/nn1268
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn1268
This article is cited by
-
Facilitation of sensorimotor temporal recalibration mechanisms by cerebellar tDCS in patients with schizophrenia spectrum disorders and healthy individuals
Scientific Reports (2024)
-
Increased temporal binding during voluntary motor task under local anesthesia
Scientific Reports (2023)
-
Weakened Bayesian Calibration for Tactile Temporal Order Judgment in Individuals with Higher Autistic Traits
Journal of Autism and Developmental Disorders (2023)
-
Audiovisual speech perception and its relation with temporal processing in children with and without autism
Reading and Writing (2023)
-
Perceptual simultaneity between nociceptive and visual stimuli depends on their spatial congruence
Experimental Brain Research (2023)