Serial dependence in visual perception

Abstract

Visual input often arrives in a noisy and discontinuous stream, owing to head and eye movements, occlusion, lighting changes, and many other factors. Yet the physical world is generally stable; objects and physical characteristics rarely change spontaneously. How then does the human visual system capitalize on continuity in the physical environment over time? We found that visual perception in humans is serially dependent, using both prior and present input to inform perception at the present moment. Using an orientation judgment task, we found that, even when visual input changed randomly over time, perceived orientation was strongly and systematically biased toward recently seen stimuli. Furthermore, the strength of this bias was modulated by attention and tuned to the spatial and temporal proximity of successive stimuli. These results reveal a serial dependence in perception characterized by a spatiotemporally tuned, orientation-selective operator—which we call a continuity field—that may promote visual stability over time.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Experiment 1 event sequence.
Figure 2: Orientation perception is serially dependent.
Figure 3: Attentional modulation of serial dependence.
Figure 4: The spatial tuning of serial dependence.

References

  1. 1

    Webster, M.A. Pattern selective adaptation in color and form perception. Vis. Neurosci. 2, 936–947 (2003).

  2. 2

    Fang, F. & He, S. Viewer-centered object representation in the human visual system revealed by viewpoint aftereffects. Neuron 45, 793–800 (2005).

  3. 3

    Clifford, C.W., Wyatt, A.M., Arnold, D.H., Smith, S.T. & Wenderoth, P. Orthogonal adaptation improves orientation discrimination. Vision Res. 41, 151–159 (2001).

  4. 4

    Dragoi, V., Sharma, J. & Sur, M. Adaptation-induced plasticity of orientation tuning in adult visual cortex. Neuron 28, 287–298 (2000).

  5. 5

    Kohn, A. Visual adaptation: physiology, mechanisms, and functional benefits. J. Neurophysiol. 97, 3155–3164 (2007).

  6. 6

    Dong, D.W. & Atick, J.J. Statistics of natural time-varying images. Network 6, 345–358 (1995).

  7. 7

    Magnussen, S. & Greenlee, M.W. The psychophysics of perceptual memory. Psychol. Res. 62, 81–92 (1999).

  8. 8

    Magnussen, S., Greenlee, M.W., Asplund, R. & Dyrnes, S. Stimulus-specific mechanisms of visual short-term memory. Vision Res. 31, 1213–1219 (1991).

  9. 9

    Shaffer, L. Timing in the motor programming of typing. Q. J. Exp. Psychol. 30, 333–345 (1978).

  10. 10

    Wing, A.M. & Kristofferson, A.B. Response delays and the timing of discrete motor responses. Percept. Psychophys. 14, 5–12 (1973).

  11. 11

    Fender, D. & Julesz, B. Extension of Panum's fusional area in binocularly stabilized vision. J. Opt. Soc. Am. 57, 819–830 (1967).

  12. 12

    Williams, D., Phillips, G. & Sekuler, R. Hysteresis in the perception of motion direction as evidence for neural cooperativity. Nature 324, 253–255 (1986).

  13. 13

    Hock, H.S., Scott, J. & Schöner, G. Bistability and hysteresis in the organization of apparent motion patterns. J. Exp. Psychol. Hum. Percept. Perform. 19, 63–80 (1993).

  14. 14

    Kanai, R. & Verstraten, F.A. Attentional modulation of perceptual stabilization. Proc. Biol. Sci. 273, 1217–1222 (2006).

  15. 15

    Holmes, D.J., Hancock, S. & Andrews, T.J. Independent binocular integration for form and colour. Vision Res. 46, 665–677 (2006).

  16. 16

    Hancock, S., Whitney, D. & Andrews, T.J. The initial interactions underlying binocular rivalry require visual awareness. J. Vis. 8, 1–9 (2008).

  17. 17

    Wolfe, J.M. Reversing ocular dominance and suppression in a single flash. Vision Res. 24, 471–478 (1984).

  18. 18

    Brascamp, J.W. et al. Multi-timescale perceptual history resolves visual ambiguity. PLoS ONE 3, e1497 (2008).

  19. 19

    Sterzer, P., Frith, C. & Petrovic, P. Believing is seeing: expectations alter visual awareness. Curr. Biol. 18, R697–R698 (2008).

  20. 20

    Maus, G.W., Chaney, W., Liberman, A. & Whitney, D. The challenge of measuring long-term positive aftereffects. Curr. Biol. 23, R438–R439 (2013).

  21. 21

    Kang, M.-S., Hong, S., Blake, R. & Woodman, G. Visual working memory contaminates perception. Psychon. Bull. Rev. 18, 860–869 (2011).

  22. 22

    Swets, J.A. & Green, D.M. Sequential observations by human observers of signals in noise. in Information Theory (ed. Cherry, C.) 221–242 (Butterworths, London, 1961).

  23. 23

    Maljkovic, V. & Nakayama, K. Priming of pop-out. I. Role of features. Mem. Cognit. 22, 657–672 (1994).

  24. 24

    Kristjánsson, A. & Campana, G. Where perception meets memory: a review of repetition priming in visual search tasks. Atten. Percept. Psychophys. 72, 5–18 (2010).

  25. 25

    Treisman, A. & Kahneman, D. The accumulation of information within object files. Bull. Psychon. Soc. 21, 354 (1983).

  26. 26

    Gibson, J.J. & Radner, M. Adaptation, after-effect and contrast in the perception of tilted lines. I. Quantitative studies. J. Exp. Psychol. 20, 453–467 (1937).

  27. 27

    Gibson, J.J. Adaptation, after-effect and contrast in the perception of tilted lines. II. Simultaneous contrast and the areal restriction of the after-effect. J. Exp. Psychol. 20, 553–569 (1937).

  28. 28

    Spivey, M.J. & Spirn, M. Selective visual attention modulates the direct tilt aftereffect. Percept. Psychophys. 62, 1525–1533 (2000).

  29. 29

    Knapen, T., Rolfs, M., Wexler, M. & Cavanagh, P. The reference frame of the tilt aftereffect. J. Vis. 10, 8 (2010).

  30. 30

    Maunsell, J.H. & Treue, S. Feature-based attention in visual cortex. Trends Neurosci. 29, 317–322 (2006).

  31. 31

    Stocker, A. & Simoncelli, E. Sensory adaptation within a Bayesian framework for perception. in Advances in Neural Information Processing Systems vol. 18 (eds. Weiss, Y., Schölkopf, B. & Platt, J.) 1291–1298 (MIT Press, 2006).

  32. 32

    Sasaki, Y., Nanez, J.E. & Watanabe, T. Advances in visual perceptual learning and plasticity. Nat. Rev. Neurosci. 11, 53–60 (2010).

  33. 33

    Fiser, J.z. & Aslin, R.N. Statistical learning of higher-order temporal structure from visual shape sequences. J. Exp. Psychol. Learn. Mem. Cogn. 28, 458–467 (2002).

  34. 34

    Chopin, A. & Mamassian, P. Predictive properties of visual adaptation. Curr. Biol. 22, 622–626 (2012).

  35. 35

    Kahneman, D. & Treisman, A. Changing views of attention and automaticity. in Varieties of Attention (eds. Parusuraman, R. & Davies, D.R.) 29–61 (Academic Press, 1984).

  36. 36

    Kahneman, D., Treisman, A. & Gibbs, B.J. The reviewing of object files - object-specific integration of information. Cognit. Psychol. 24, 175–219 (1992).

  37. 37

    Noles, N.S., Scholl, B.J. & Mitroff, S.R. The persistence of object file representations. Percept. Psychophys. 67, 324–334 (2005).

  38. 38

    Whitney, D. The influence of visual motion on perceived position. Trends Cogn. Sci. 6, 211–216 (2002).

  39. 39

    Ross, H.E. & Plug, C. The history of size constancy and size illusions. in Perceptual Constancy: Why Things Look as They Do (eds. Walsh, V. & Kulikowski, J.) 499–528 (Cambridge University Press, 1998).

  40. 40

    Palmer, S.E. Vision Science: Photons to Phenomenology (The MIT press, 1999).

  41. 41

    Kristjánsson, A., Wang, D. & Nakayama, K. The role of priming in conjunctive visual search. Cognition 85, 37–52 (2002).

  42. 42

    Kristjánsson, A. Simultaneous priming along multiple feature dimensions in a visual search task. Vision Res. 46, 2554–2570 (2006).

  43. 43

    Maljkovic, V. & Nakayama, K. Priming of pop-out. II. The role of position. Percept. Psychophys. 58, 977–991 (1996).

  44. 44

    Geyer, T., Müller, H.J. & Krummenacher, J. Cross-trial priming in visual search for singleton conjunction targets: role of repeated target and distractor features. Percept. Psychophys. 68, 736–749 (2006).

  45. 45

    Kim, N., Ivry, R.B. & Robertson, L.C. Sequential priming in hierarchically organized figures: effects of target level and target resolution. J. Exp. Psychol. Hum. Percept. Perform. 25, 715–729 (1999).

  46. 46

    Simons, D.J. & Rensink, R.A. Change blindness: Past, present and future. Trends Cogn. Sci. 9, 16–20 (2005).

  47. 47

    Rensink, R.A., O'Regan, J.K. & Clark, J.J. To see or not to see: the need for attention to perceive changes in scenes. Psychol. Sci. 8, 368–373 (1997).

  48. 48

    Rensink, R.A., O'Regan, J.K. & Clark, J.J. On the failure to detect changes in scenes across brief interruptions. Vis. Cogn. 7, 127–145 (2000).

  49. 49

    Neisser, U. & Becklen, R. Selective looking: attending to visually specified events. Cognit. Psychol. 7, 480–494 (1975).

  50. 50

    Simons, D.J. & Chabris, C.F. Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 28, 1059–1074 (1999).

  51. 51

    Mitchell, D.E. & Muir, D.W. Does the tilt after-effect occur in the oblique meridian? Vision Res. 16, 609–613 (1976).

  52. 52

    Ringach, D.L., Shapley, R.M. & Hawken, M.J. Orientation selectivity in macaque V1: diversity and laminar dependence. J. Neurosci. 22, 5639–5651 (2002).

Download references

Acknowledgements

We thank J. Shankey for assistance with data collection, and G. Maus, S. Teng and E. Whitney for comments on the manuscript. This work was supported in part by grants from the US National Institutes of Health (EY018216) and the National Science Foundation (1245461) to D.W.

Author information

Affiliations

Authors

Contributions

J.F. and D.W. designed the experiments. J.F. collected the data and carried out the analyses. J.F. and D.W. wrote the manuscript.

Corresponding author

Correspondence to Jason Fischer.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Measuring perceptual serial dependence.

To measure serial dependence in orientation perception, we presented subjects with an oriented grating (Gabor) on each trial and asked subjects to report the orientation of the grating by adjusting a bar using the arrow keys (panel a). Of interest was whether the reported orientations were systematically dependent on the stimulus orientations seen in previous trials. We constructed a plot capturing the relationship between the perceptual error on each trial and the orientation presented on the previous trial (panel b). For each trial we computed the relative orientation of the previous stimulus to the present one (e.g., by how much was the previously seen orientation clockwise or counter-clockwise of the present orientation; indicated by the blue box) with positive values indicating that the orientation seen on the previous trial was clockwise of the orientation seen on the present trial. We also computed the error on the present trial with positive values indicating perceptual errors in the clockwise direction (indicated by the red box). Plotting error as a function of the relative orientation of the previous trial allowed us to examine whether subjects' errors were drawn in the direction of the previous stimulus (shaded region on the plot), or repelled away from the orientation of the previous stimulus as would be found for traditional negative aftereffects at short inter-stimulus intervals (unshaded region on the plot). A preponderance of points within the shaded region of the plot, as was the case in the actual data (see Fig. 2a), indicates that perceived orientation was attracted toward the orientation seen on the previous trial.

Supplementary Figure 2 Computing the just noticeable difference (JND) from two alternative forced choice data.

To compute the just noticeable difference (minimum orientation difference that can be reliably discriminated) for the Gabors presented in our experiments, we used the 2AFC discrimination data from Experiment 3, this time using only the information from the 2AFC judgments. For each individual subject, we plotted the percent of the time that the subject responded “right” as a function of the relative orientations of the Gabors presented in the left and right visual fields (positive values on the abscissa indicate that the right-hand Gabor was more clockwise). Data shown here are from one example subject. We fit a logistic function to the resulting plot as in the analysis of Experiment 3, and found the x values for which the logistic curve had a y value of 25% and 75%. One half of the difference between these two x values was taken as the JND. The mean JND for the three subjects who participated in Experiment 3 was 5.39. The same three subjects participated in Experiment 1; the mean amplitude of serial dependence for these subjects was 8.21; that is, two orientations that differed by 16.42°. could appear identical based on the stimulus that preceded them, an effect that is 3.05 times the JND as measured here in the same subjects.

Supplementary Figure 3 Serial dependence measured using randomized trial sequences.

We repeated Experiment 1, this time presenting a random sequence of orientations on each run. All aspects of the experimental design and analysis were identical to the counterbalanced version of Experiment 1 except that a random orientation was presented on each trial, and we analyzed all trials in the sequence. Four participants each completed eight 104-trial runs. The gray line in each plot shows a running average of the subject's orientation judgment errors, and data points show mean error within 20 deg. bins along the abscissa. Black lines show derivative of Gaussian (DoG) fits to subjects' error distributions. Each subject individually showed significant serial dependence in orientation judgments (permutation tests; 824 data points per subject). The amplitude of serial dependence found with this randomized design did not differ from the amplitude measured with the counterbalanced design (group mean amplitudes were ±8.19° for the counterbalanced design (4 subjects, 260 data points per subject) and ±4.85° for the randomized design (4 subjects, 824 data points per subject); permutation test comparing amplitudes from the randomized and counterbalanced versions; P = 0.15).

Supplementary Figure 4 Statistics of Experiment 1 trial sequences.

Experiment 1 used two approaches for generating trial sequences: a fully random sequence of orientations and a counterbalanced sequence designed to ensure that every orientation used in the analysis was preceded equally often by all orientations in the range of -60 to 60 degrees relative to it (see Methods). Data points here show the autocorrelation (correlation between the orientation presented on a given trial and the relative orientation of the previous trial) for each run that subjects completed in Experiment 1; one point represents one run. Within the counterbalanced design, the autocorrelation was exactly zero for all runs when considering the trials used in the analysis (“baseline” trials; red data). Considering all trials within the counterbalanced runs including those not intended for analysis and discarded prior to computing the strength of serial dependence, the correlation tended to be positive (blue data) and the mean was significantly greater than zero (mean correlation of z = 0.11 across 40 runs; P = 1x10-5, evaluated by a permutation test in which we shuffled the trial ordering within runs and recomputed the mean correlation 100,000 times to generate a null distribution). However, there was no significant difference in the amplitude of serial dependence measured within only baseline trials vs. all trials in the counterbalanced sequences (group mean amplitudes were ±5.51° for all trials (1030 trials per subject) and ±8.19° for baseline trials (260 trials per subject); P = 0.10; permutation test). Within the randomized stimulus sequences, the autocorrelation was never exactly zero (which is why we also employed a counterbalanced design), but across all runs the autocorrelation centered on zero and did not significantly differ from zero (green data; mean correlation of z = -0.025 across 32 runs; P = .15 evaluated with a permutation test as above). Within both the counterbalanced and randomized paradigms, we found significant serial dependence in orientation perception for all subjects. These results, together with the negative aftereffect modeling (Supplementary Fig. 5), confirm that perceptual serial dependence is not the result of statistical dependencies in the presented trial sequences.

Supplementary Figure 5 Serial dependence does not arise from negative aftereffects alone.

We conducted a simulation to test whether apparent serial dependence in perceived orientation could arise from negative aftereffects alone, for example due to an interaction between negative aftereffects and random fluctuations in the stimulus sequence. Using the same trial sequences presented to subjects in Experiment 1, we simulated subjects' responses based on negative aftereffects of varying strength and persistence over time. We found that regardless of the aftereffect strength or duration, negative aftereffects alone predict a negative amplitude in the measured serial dependence (repulsion rather than attraction; panel a shows the simulation results for the stimulus sequence presented to one subject; all subjects showed comparable results), contrary to what we observed in subjects' responses. Panel b shows a sample position discrimination plot from the simulation. Predicted responses based solely on negative aftereffects are qualitatively dissimilar to the pattern of responses that subjects made in Experiments 1-6. This simulation shows that it is not possible for negative aftereffects alone to produce an apparent attraction of perceived orientation toward previously seen orientations within our experimental design and analysis.

Supplementary Figure 6 Event sequence for one trial in Experiment 3.

Subjects made two judgments per trial; a cue reminded subjects which judgment to perform on the upcoming stimuli. At the outset of a trial, a cue (white dot) appeared to the left or right of fixation. This cue instructed the subject to judge the orientation of the Gabor that appeared at that location; the Gabor in the opposite visual field was task-irrelevant. After a delay, two Gabors appeared simultaneously to the left and right of fixation for 500 ms, followed by noise patches presented for 1 s. The subject then adjusted a response bar to match the perceived orientation of the Gabor at the cued location. After the subject's response, there was a delay period of 1.5 s during which only the fixation point was present. Following the delay period, the fixation point dimmed slightly for 350 ms to indicate that the subject was to judge the relative orientations of both upcoming Gabors, deciding which was oriented more clockwise (or, on separate runs, which was more counter-clockwise). Two Gabors then appeared to the left and right of fixation for 500 ms, followed by noise patches presented for 1 s. The subject then indicated which of the Gabors was oriented more clockwise (or counter-clockwise) in a two alternative forced choice (2AFC) response. Following the subject's response there was a delay period of 1.5 s prior to the onset of the next trial.

Supplementary Figure 7 Experiment 6: The coordinate frame of serial dependence.

We compared the strength of serial dependence in perceived orientation in retinotopic vs. spatiotopic (head-centered) coordinates by varying the stimulus and fixation locations across trials (panel a). On each trial the fixation point appeared in one of two locations, at 6.5° to the left or right of the center of the screen, and the Gabor patch appeared in one of three locations, at 6.5° or 19.4° to the left or right of fixation depending on the location of the fixation point. Trials were paired, and every second (baseline) trial was analyzed. We manipulated the change in fixation-Gabor positioning between the trials in each pair: on ¼ of trial pairs both the spatial and retinal positions stayed the same (“both” trials), on ¼ of trial pairs the spatial position of the Gabor changed while its retinal position remained the same (“retinal” trials), on ¼ of trial pairs the retinal position of the Gabor changed while its spatial position remained the same (“spatial” trials), and on the final ¼ of trial pairs both the retinal and spatial position of the Gabor changed (“neither” trials; hence the trials are named for the aspect of the stimulus position that remained constant between the two successive trials in a pair). We counterbalanced the orientations presented so that subjects saw exactly the same set of orientations on all four of the spatial conditions – the only thing that differed among the spatial conditions was the change in the relative positioning of the fixation point and Gabor between trial pairs. We found significant serial dependence within each of the four conditions (panel b; permutation tests based on 180 trials from each of four subjects in each condition; error bars are ±1 s.d. of the bootstrapped group mean). Serial dependence was significantly stronger in those trials in which the retinal stimulus position was the same as in the previous trial (retinal & both trials) than in trials when the retinal position changed (spatial & neither trials), replicating the tuning of serial dependence to retinal position found in Experiments 4 and 5 (P = 0.019; permutation test based on 360 trials per condition from each of four subjects). At the same time, serial dependence was also stronger when the spatiotopic stimulus position stayed constant between trials (spatial & both trials) than when it changed (retinal & neither trials; P = 0.037; permutation test based on 360 trials per condition from each of four subjects), indicating that serial dependence is tuned to an object's spatiotopic position as well. This tuning to both spatiotopic and retinotopic stimulus position suggests that serial dependence likely operates at multiple levels in the visual processing hierarchy.

Supplementary Figure 8 Experiment 7: Negative aftereffects in orientation judgments.

We tested whether negative aftereffects would arise in a similar experimental paradigm to the one used in Experiments 1 and 2 if the stimulus presentation was longer, allowing for orientation adaptation. a) In each trial, subjects saw two Gabors: a long duration Gabor (5 s) and a short duration Gabor (500 ms). Each Gabor was followed by a 1s noise patch, and subjects were instructed to report the orientation of the second (short duration) Gabor using an adjustment response. This design was intended to be as similar as possible to the one used in Experiments 1 and 2 while allowing for orientation adaptation to occur and minimizing the time between the adapting (long duration) and test (short duration) stimuli. The absence of a response between the first and second Gabors is akin to Experiment 2, where we found serial dependence in orientation judgments even in the absence of a response on the previous trial. Both Gabor orientations were randomized on each trial. Two participants each completed six 104-trial runs. b) We analyzed the data as in Experiment 1 (see Supplementary Fig. 1) by plotting orientation judgment errors as a function of the difference in orientation between the first and second Gabors (Gabor 1 minus Gabor 2). Gray lines show a running average of subjects' errors, and data points show mean error within 20 deg. bins along the abscissa. Black lines show derivative of Gaussian (DoG) fits to subjects' error distributions. Crucially, the sign of the curve fit for each subject was flipped as compared with the data from Experiments 1 and 2 (see Fig. 2a and Supplementary Fig. 2) – both subjects showed significant negative orientation aftereffects (permutation tests; 618 data points per subject). These data show that both negative and positive aftereffects (serial dependence) can arise within the same paradigm depending on stimulus duration and the time between successive stimuli.

Supplementary Figure 9 Labeled-line models of how serial dependence may arise from changes in single-unit orientation tuning.

a) Gain model: channels tuned to recently-seen stimuli temporarily become more responsive. This increased sensitivity shifts the population response to a current stimulus toward the orientations of previous stimuli. b) Shift model: single-unit orientation tuning is temporarily shifted away from the orientations of recently-seen stimuli. Shifts in orientation tuning skew the population response to a current stimulus toward the orientations of previous stimuli. c) We fit the models to subjects' responses from the fully randomized version of Experiment 1 using one half of the data (odd numbered runs; 412 trials per subject) and tested model performance on the remaining (independent) half of the data (even numbered runs; 412 trials per subject). Both the gain model (blue data) and shift model (red data) performed significantly above chance in predicting subjects' errors on the left-out data (permutation tests; shaded region shows the permuted chance distribution). While the gain model produced somewhat better fits in three of four subjects, both models provide plausible accounts of how serial dependence in orientation perception might arise from simple changes in single-unit tuning.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–9 (PDF 3077 kb)

Supplementary Modeling

Supplementary Modeling (PDF 224 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fischer, J., Whitney, D. Serial dependence in visual perception. Nat Neurosci 17, 738–743 (2014). https://doi.org/10.1038/nn.3689

Download citation

Further reading