Vision is believed to dominate our multisensory perception of the world. Here we overturn this established view by showing that auditory information can qualitatively alter the perception of an unambiguous visual stimulus to create a striking visual illusion. Our findings indicate that visual perception can be manipulated by other sensory modalities.
We have discovered a visual illusion that is induced by sound: when a single visual flash is accompanied by multiple auditory beeps, the single flash is incorrectly perceived as multiple flashes. These results were obtained by flashing a uniform white disk (subtending 2 degrees at 5 degrees eccentricity) for a variable number of times (50 milliseconds apart) on a black background. Flashes were accompanied by a variable number of beeps, each spaced 57 milliseconds apart. Observers were asked to judge how many visual flashes were presented on each trial. The trials were randomized and each stimulus combination was run five times on eight naive observers.
Surprisingly, observers consistently and incorrectly reported seeing multiple flashes whenever a single flash was accompanied by more than one beep (Fig. 1a). Control conditions and catch trials ( Fig. 1 legend) indicate that the illusory flashing phenomenon is a perceptual illusion and is not due to the difficulty of the task, cognitive bias or other factors.
Figure 1b shows that observers' performance was the same, irrespective of whether a single flash was accompanied by two beeps, or two flashes by one or no beeps, suggesting that the illusory double flash is perceptually equivalent to the physical double flash. This was confirmed by the testimonies of the observers after the experiment. The performance of non-naive subjects (results not shown) indicated that the illusion persisted even when subjects were aware of the fact that the disk was actually flashed only once.
We next investigated the temporal properties of this illusion by varying the relative timing of visual and auditory stimuli. The illusory flashing effect declined from 70 milliseconds separation onwards. However, illusory flashing occurred as long as the beep and flash were within approximately 100 milliseconds of each other, which is consistent with the integration time of polysensory neurons in the brain1,2.
Our results indicate that the illusory flashing is caused by an alteration of visual perception by auditory stimuli. The modification of the visual percept by sound, however, was not categorical. It was rather selective, as sound did not have a fusing effect when multiple flashes were accompanied by a single beep. We suggest therefore that the direction of cross-modal interactions is partly dependent on the type of stimulus. Consistent with previous observations in other modalities3, we propose that the percept of a continuous stimulus in one modality is rendered more malleable by a discontinuous stimulus in another modality than vice versa.
The influence of auditory cues on visual perception has been demonstrated in other settings, in which perceived visual intensity is affected by the presence of an auditory stimulus4. This influence, however, is quantitative and does not alter the phenomenological quality of the percept. Others have shown that the perceived direction of ambiguous visual motion is influenced by auditory stimulation5.
Our results extend these previous findings by showing that visual perception can be qualitatively altered by sound even when the visual stimulus is not ambiguous. The conditions under which this alteration occurs — the stimulus configuration and the task — are very simple. The illusion is also surprisingly robust to variation in the many parameters we manipulated (including disk eccentricity and contrast, spatial disparity between sound and flash, shape and texture of the flashing pattern, flash and beep durations). The simplicity and robustness of the illusory flashing phenomenon indicate that it reflects a fundamental and widespread property of polysensory mechanisms in the brain.
Meredith, M. A., Nemitz, J. W. & Stein, B. E. J. Neurosci. 10, 3215– 3229 (1987).
Wallace, M. T. & Stein, B. E. J. Neurophysiol. 71 , 429–432 (1994).
Saldaña, H. M. & Rosenblum, L. D. Percept. Psychophys. 54, 406–416 (1993).
Stein, B. E., London, N., Wilkinson, L. K. & Price, D. D. J. Cogn. Neurosci. 8, 497–506 (1996).
Sekuler, R., Sekuler, A. B. & Lau, R. Nature 385, 308 ( 1997).
About this article
Cite this article
Shams, L., Kamitani, Y. & Shimojo, S. What you see is what you hear. Nature 408, 788 (2000). https://doi.org/10.1038/35048669
This article is cited by
Experimental Brain Research (2023)
Cognitive Research: Principles and Implications (2022)
Nature Human Behaviour (2022)
Psychological Research (2022)
Review of Philosophy and Psychology (2022)