Predictive coding theories propose that the brain creates internal models of the environment to predict upcoming sensory input. Hierarchical predictive coding models of vision postulate that higher visual areas generate predictions of sensory inputs and feed them back to early visual cortex. In V1, sensory inputs that do not match the predictions lead to amplified brain activation, but does this amplification process dynamically update to new retinotopic locations with eye-movements? We investigated the effect of eye-movements in predictive feedback using functional brain imaging and eye-tracking whilst presenting an apparent motion illusion. Apparent motion induces an internal model of motion, during which sensory predictions of the illusory motion feed back to V1. We observed attenuated BOLD responses to predicted stimuli at the new post-saccadic location in V1. Therefore, pre-saccadic predictions update their retinotopic location in time for post-saccadic input, validating dynamic predictive coding theories in V1.
Predictive coding accounts of vision propose that higher cortical areas use internal models of the world to predict sensory inputs1,2,3,4,5. Cortical feedback carries these predictions back to V1 where a neural mechanism compares them to the actual sensory inputs6. However, a central tenet of predictive coding remains to be tested. Humans saccade approximately three times per second, changing the retinotopic pattern of sensory inputs to V17. Therefore, for cortical predictive feedback to be functional, it must update to new retinotopic locations in V1 in time to meet post-saccadic input8,9.
Central to our study is the creation of an internal model in the brain, during which feedback carries sensory predictions from higher areas down to V1. The apparent motion illusion offers a paradigm for investigating such a model. Apparent motion is an illusion of a moving token between two alternating flashing stimuli10,11. Apparent motion is integrated in V512,13,14,15,16, which feeds back a retinotopically-specific spatiotemporal prediction of a moving token to V1, inducing activation along the non-stimulated illusory motion trace6,14,17,18,19. The spatio-temporal prediction on the apparent motion trace can be probed with targets presented either in-time with the motion illusion (predicted), or out-of-time (unpredicted). In-time targets are detected more accurately than out-of-time targets15,20,21,22 but lead to less (i.e. dis-amplified) BOLD activation in V1 compared with amplified activation for out-of-time targets6. According to predictive coding theories, the amplified BOLD activation for out-of-time targets is indicative of an error signal because they are less predictable in the context of the illusory moving token. Likewise, dis-amplified BOLD activity for in-time targets is consistent with the successful prediction (or ‘explaining away’) of sensory input. Here, we tested if pre-saccadic predictions feed back to new retinotopic locations in V1 in time for post-saccadic target processing. To this end, we presented the apparent motion illusion to one visual hemifield and then prompted a saccade, transferring motion-related feedback to the opposite hemisphere. Our data confirms that cortical feedback dynamically updates to new predicted retinotopic coordinates in V1.
We first induced an apparent motion illusion, which creates a spatio-temporally precise prediction of apparent motion along the illusory motion trace in V1. We then instructed subjects to perform a right-to-left saccade across the apparent motion stimulus. When the saccade landed left of the apparent motion illusion, a target was presented either in-time or out-of-time during the last cycle of apparent motion (Fig. 1a–d). After the saccade, the stimulus was in the right visual field, and processed by post-saccadic left V1. The post-saccadic left V1 processed only one apparent motion inducer after the target appeared, thus illusion-related activity is present even though the post-saccadic left V1 was not stimulated with a full apparent motion cycle. Without prior expectation from the contralateral hemisphere, both target stimuli are equally predictable by the surrounding stimulation. We tested for the presence of post-saccadic predictions by comparing the BOLD activity to in-time and out-of-time targets, in a test region on the illusory motion trace in post-saccadic left V1 (Fig. 2). This test region corresponds to the new retinotopic location at which the cortex processes the target stimulus after the saccade. We examined this test region in three apparent motion conditions: no target, in-time target, and out-of-time target. Subjects were instructed to report if they perceived a target presented immediately after saccade. We ran four functional magnetic resonance imaging (fMRI) experiments. Our pilot fMRI experiment led us to use high-resolution eye-tracking data for trial exclusion in the main apparent motion experiment. In the pilot experiment, we found consistent results with the main experiment in the subjects with valid eye movement data (see supplemental material and Figure S1). To confirm that our results are due to predictions created by the internal model of apparent motion, we also ran a flicker control fMRI experiment. Here, we presented the two flashing inducer stimuli simultaneously, instead of in alternating rhythm as in the apparent motion stimulation (Fig. 1e). In the context of the flicker stimulation, neither target had a temporal predictability but for simplicity we kept the conditions labeled as “in-time” and “out-of-time” as the targets were presented at the same stimulus onset time in relation to the lower apparent motion inducer stimulus as in the main fMRI experiment. Finally, we ran a replication of our main apparent motion and flicker experiments. The design of the replication was identical to the main and flicker experiment except for two variations: (1) we ran the replication as a within- group design and (2) we used the same hardware throughout the replication (32-channel head coil) while in the original experiment we switched between experiments using the 12-channel head coil for the main and the 32-channel for the flicker control. The replication supported and strengthened our original findings.
Post-saccadic apparent motion prediction at the new retinotopic location in left V1
We tested for prediction-related activity on the illusory motion trace by first mapping the cortical representation of the target stimulus located between the two blinking apparent motion inducing stimuli. We presented a static target stimulus in between the apparent motion inducers, and a static stimulus at the same location as the lower apparent motion inducer. To map the target region of interest (ROI), we used a GLM contrast of ‘target stimulus’ > ‘lower AM inducer stimulus’ (Fig. 2, online methods). After identifying the target ROI in post-saccadic left V1, we compared activation for the three apparent motion conditions. The illusory trace was activated during apparent motion even when no target was presented (no-target condition) in 5/9 subjects for the Main experiment, and in 7/8 subjects for the Replication experiment (single subject: *p < 0.05; Main experiment group mean beta-weight (SE) = 0.2(0.02), p < 0.0001, Replication group mean beta-weight (SE) = 1.9(0.3), p = 0.0007 Figs 3a,b and 4a and b). Importantly, we observed apparent-motion trace activity in post-saccadic (left) V1 even though we only presented one apparent motion inducing stimuli to this hemisphere. Illusion-related activity on the non-stimulated apparent motion trace has been shown before13,14,17,18,19, but it is novel to find that the illusory activity transfers across hemispheres to the post-saccadic (left) V1 in anticipation of post-saccadic continuation of the illusion. It is unlikely that activity found for the ‘no target apparent motion’ condition is due to the lower apparent motion stimulus. We conservatively selected voxels specific to the target-processing region and confirmed their retinotopic position on individual maps. At the target ROI, activity to apparent motion was higher than to the mapping stimulus confirmed by a GLM contrast between ‘no target apparent motion’ and the ‘lower inducer’ (Main experiment: t(8) = 3.81, p = 0.005, Replication: t(7) = 2.98, p = 0.02). Therefore, at the target position it is the illusion which drives activity, not the inducer stimulus. In contrast, at the inducer position, activation for the mapping stimulus was stronger than the apparent motion stimulus, which we expect because the mapping stimulus was shown 48 times longer than the apparent motion inducing stimulus (4000 ms vs 83.3 ms). Lastly, individual subject activity maps indicated that the peak apparent motion activation overlaps more voxels within the target region than within the inducer stimulus (Fig. 4a (t(8)4.67, p = 0.0016) & 4b (t(7)5.56, p < 0.0001)).
To investigate apparent motion prediction in post-saccadic (left) V1, we tested whether in-time and out-of-time targets were dis-amplified or amplified (Fig. 3b and c). If predictive codes update to new retinotopic positions in post-saccadic (left) V1, in-time target activation may be dis-amplified (because targets are more predictable) and out-of-time targets amplified6 (because targets are less predictable). We found less increased BOLD in the left V1 target ROI for in-time versus out-of-time targets in 7/9 subjects for the Main experiment (group mean (SE) beta-weight for in-time target trials = 0.23(0.13); out-of-time target trials = 0.29(0.17); t(8) = 2.39, p = 0.04, Fig. 3c) and in 5/8 for the Replication (group mean (SE) beta-weight for in-time target = 1.99(0.35); out-of-time target trials = 2.3(0.33), t(7)2.62,p = 0.035, Fig. 3d). The increase of BOLD activity for out-of-time targets may indicate a prediction error signal or dis-amplification for predicted in-time targets. This error signal occurs at the new retinotopic location due to pre-saccadic predictions updating their retinotopic location in time for post-saccadic input.
With the flicker control experiment we show that the increased BOLD activity for out-of-time targets is specific to the spatiotemporal expectation triggered by the apparent motion context. For the flicker control experiment, we again mapped the retinotopic region of left V1 processing the target stimulus but we found, as expected, no significant difference between the in-time and out-of-time targets in the group analysis (non-parametric signed rank test: Main experiment: p = 0.625, Fig. 5a; Replication: t(7)0.35, p = 0.73, Fig. 5b). Only one subject showed a significant difference between in-time and out-of-time target activity in the Replication flicker experiment, with an increased activity for in-time targets (p < 0.05). We tested if the difference in activation to in-time and out-of-time targets was greater in the main apparent motion experiment (where we expect an enhanced response to the out-of-time target) than in the flicker control (where we expect to find no consistent pattern of difference in activation to the target conditions). We conducted a t-test which does not assume equal variance or equal group size for the main fMRI experiment versus the flicker experiment and found no significant activity difference between ‘in-time’ and ‘out-of-time’ targets (p = 0.3699). When we replicated our apparent motion experiment and flicker control using the within subjects design, we found that activation differences between in-time and out-of-time targets were significantly different between the apparent motion stimulation and the flicker stimulation (t(7)2.65, p = 0.03). Our use of a within -subject design for the replication study reduced the variability and increased power of analysis. Furthermore, we equated the time difference between saccade offset and target onset for both target conditions, and found that the out-of-time > in-time activation difference was maintained in the apparent motion experiment only (t(7)3.02, p = 0.0193). When pooling across all experiments, a rank sums test on unequal groups between AM (n = 22) and flicker (n = 13) reveals that activation differences between in-time and out-of-time targets are significantly different (p = 0.04). Flicker perception might trigger different low precision predictions for additional target appearance in different individuals explaining the higher variance we see in subjects’ BOLD responses. In summary, the pattern of results show, that only the internal model of apparent motion imposes a spatiotemporal expectation on the processing of the target. Without the internal model of apparent motion illusion, one temporal sequence (i.e. target stimulus) cannot be more expected than the other, and we observe no activity difference in target processing. This also confirms the timing of the targets relative to the inducer does not create a differential BOLD signal by itself, but only if it is accompanied by the internal model of an apparent motion illusion.
Location and temporal specificity of apparent motion prediction in V1
We find the internal model of apparent motion (but not flicker) creates a predictive signal at the post-saccadic target processing location in left V1. To ensure this signal is spatially and temporally specific, we ran a 3-way ANOVA on location in V1 (target ROI and lower inducer ROI) vs. time-point of trial (pre-saccade and post-saccade) vs. condition (in-time or out-of-time target activity). We found a significant interaction between location, target condition and time-point for our Main experiment (F(1, 8)6.46, p = 0.035; Fig. 5c) and for our Replication (F(1, 7)7.9, p = 0.007; Fig. 5d). Post-hoc tests revealed the interaction is driven by time-point (p < 0.04) and ROI (p < 0.05), the out-of-time versus in-time difference only reported at the post-saccadic time-point accounts for these findings in both the Main (p = 0.04) and Replication (p = 0.035) experiments. Note, there was no significant difference between in-time and out-of-time targets at the post-saccadic time-point in the lower inducer stimulus processing region for the Main (p > 0.48) or Replication experiments (p > 0.14). Prior to the saccade we found no significant difference between conditions in neither the target nor the apparent motion region for the main (p = 0.85; p = 0.19) and replication experiments (p = 0.92; p = 0.72). This supports our interpretation that trans-saccadic cortical feedback to V1 is spatially and temporally specific to the region of cortex processing a predicted sensory input. Furthermore, in Fig. 4c and d we present individual subject cortical inflations demonstrating that voxels activated for the predictive effect (out-of-time > in-time target activity) overlap and surround the cortical representation of the target stimulus but rarely spill over into region of cortex processing the lower inducer stimulus. On average, 78 more voxels were activated in the target ROI in comparison to the lower inducer ROI for an out-of-time > in-time contrast in both the main (t(8)8.54, p < 0.0001) and replication experiments (t(7)3.92, p = 0.0058).
Region specific apparent motion prediction
In the region of left dorsal V2 activated by the target stimulus, we found weak responses to all apparent motion conditions after the saccade for both Main and Replication experiments (in-time targets p < 0.05; out-of-time targets p < 0.05; no target p < 0.003; Figure S2a and S2b). We also confirmed activation of motion-sensitive right and left V5 during apparent motion perception (Figure S2c and S2d). Right and left V5 were mapped using a contrast of apparent motion with no target > baseline. We find a significant effect of region when performing a repeated measures ANOVA for the out-of-time > in-time activation difference across all regions (left V1, left V2, right V5 and left V5) in the main experiment (F(3, 8) 2.97, p = 0.03) and replication (F(3, 7) 3.77, p < 0.01). We do not find the out-of-time > in-time target effect in V2 (Main: p = 0.57; Replication: p = 0.87), right V5 (Main: p = 0.3; Replication: p = 0.45), or left V5 (Main: p = 0.9; Replication: p = 0.85) in either experiment, meaning that our ANOVA is driven by the out-of-time > in-time target effect in V1 reported previously (Main: p = 0.04; Replication: p = 0.035). Our lack of prediction effect in left and right V5 is consistent with previous findings6.
Since V2 has larger receptive fields, the target ROI also responded slightly to the lower AM inducing stimulus (Main experiment: beta-weight = 0.16(0.05), p < 0.009; Replication: beta-weight = 0.41(0.046), p < 0.001; Figure S2a and S2b). The lack of retinotopic specificity might have affected the sensitivity to pick up differential effects amongst the AM conditions. Alternatively, it might indicate a stronger prediction effect in V1 than in V2.
Extra-session psychophysical control experiment
During the fMRI experiments, we found no behavioural detection advantage for either in-time or out-of-time targets as shown previously (Figure S4; refs20,21). Participants’ may have expected the target to always appear directly after the saccade, diluting our hypothesised behavioural result. In a separate behavioural experiment, subjects were again required to respond if they detected a target presented within the apparent motion trace before, during and after saccade (Online Methods). Subjects were more accurate at detecting predictable in-time targets after saccades. The in-time advantage effect was significant at group level (t(7)3.07, p = 0.015) and for 4/8 subjects (Figure S4c; p < 0.02), replicating previous work with fixation15,20 and after eye-movement21.
We provide evidence for dynamic updating of cortical predictions to new retinotopic locations at the lowest level of the cortical hierarchy: primary visual area V1. Our findings support the proposals of Helmholtz (see refs23,24) that the brain creates predictive models explaining away expected changes, including those made by planned eye-movements. Moreover, our results support the theories of Mumford4 and Friston25 who suggest visual predictions descend the cortical hierarchy in feedback. However, until now, it has not been established that cortical predictions feed back to retinotopic visual cortex in time to interact with post-saccadic sensory input. We measured fMRI of subjects presented with the apparent motion illusion, during which V5 feeds back spatio-temporal predictions to V115. We then prompted subjects to make a saccade, necessitating that predictive feedback updates to a new retinotopic location in V1. In our data, we observed a top-down signal in V1 which is retinotopically-precise, predictive of spatiotemporal sensory inputs and effective after a saccade. We confirm an anticipated, but as yet unobserved, mechanism of human vision: cortical predictions update to new locations in retinotopic cortex in time for post-saccadic processing.
Prediction in vision
Our perception of the world is not deconstructed each time we move our eyes, and reconstructed after each new fixation. Instead, the brain maintains perceptual continuity26, because visual experience is the result of the brain’s internal model of the physical world merged with sensory signals27. The brain performs this modelling process in order to operate most energy-efficiently, and eye movements function to test these internal models28. Under predictive coding hypotheses, internal models generate predictions of forthcoming sensory inputs29. Cortical feedback carries these predictions down the visual hierarchy where a neural mechanism compares them with actual sensory inputs. However, the early visual system is flooded with different patterns of sensory inputs each time we move our eyes7. This means that if the cortex is to compare predictions to sensory inputs, predictions must update to the new retinotopic location in sensory cortex. This study is the first to provide evidence for the retinotopic updating of predictive feedback to V1.
Predictive cortical feedback updates to new retinotopic locations with long-range saccades
In V1, cortical predictions must be spatially and temporally specific in order to be functional. Our spatial location of interest was a region of V1 that responds to a portion of the visual field between two apparent motion inducing stimuli. It was in this region of cortex that we observed a lower BOLD response to predictable versus unpredictable targets. This attenuation in BOLD signal is due to higher cortical areas (likely V5) feeding back predictions15 which dis-amplify expected inputs in sensory cortex6. Over the trial time there are cognitive and feedforward effects that may contribute to the observed activation in the post-saccadic apparent motion trace. Our subjects observed apparent motion initially in the left visual hemifield. During this time we assume attention is divided between the red fixation cross in the fovea, the illusion percept (presented peripherally), and the future fixation position (green fixation cross, even further peripherally). During this pre-saccadic phase, we hypothesize three ongoing processes: (1) anticipatory remapping of the apparent motion stimulus due to the upcoming saccade (2) the establishment of visual predictions in higher areas induced by apparent motion, and (3) task related, top-down attention processes monitoring colour changes in the peripheral location. In the post-saccadic phase, we observed in left V1: apparent motion and saccade related activity, on top of which we found target-related activity that is amplified if out-of-time with the apparent motion illusion (or dis-amplified in case of a spatiotemporal match between the prediction of illusory motion and the in-time target).
Let us turn first to how saccade-related remapping signals could modulate our target region in V1. Receptive field remapping is the anticipatory activation of receptive fields which are about to receive sensory input after saccade30. In our ‘no-target’ condition, some activity that we observed on the apparent motion trace in the left V1, i.e. at the new retinotopic location could be due to remapping effects in this region of cortex31, related to the remapping of the inducer stimuli. Apparent motion in the left hemifield induces an activity increase on the apparent motion trace in the right hemisphere19 and we suggest this activity likely remaps to the new, left hemisphere. However, for both the in-time and out-of-time target activity on the apparent motion trace, receptive field remapping of sensory input is not a credible explanation because we did not present a target in the left hemifield to be remapped. Further, although remapping activity might be present around the AM inducers and the fixation cross, these stimuli all map to regions of cortex separate from that processing the Target region on the AM trace. Even if anticipatory activity related to upcoming saccade accounted for the activation we observed on the apparent motion trace during the No Target condition, it would not explain the activation difference between in-time and out-of-time targets. It remains to be tested if or how the remapping of the illusory trace (i.e. the remapping of top-down feedback signals) could initiate predictive perception in V1 directly after saccade.
The next top-down influence we consider is spatial attention. How does visual spatial attention contribute to our observed activation pattern on the post-saccadic apparent motion trace? A recent study by Cravo et al., demonstrated increased target detection during low-frequency phase entrainment using rhythmic stimuli. The authors suggest oscillations carry temporal expectations during attention32. However, our main effect is the difference in BOLD activation for in-time and out-of-time targets, which attention models struggle to account for in two ways. Attention is often conceptualized as a gain model where increased activity correlates with better performance33,34. According to our fMRI and extra-session psychophysical experiments, out-of-time targets induce increased BOLD activity, but are detected less accurately than in-time targets. Moreover, we have shown previously that when subjects direct attention away from the location of apparent motion, feedback still fills in a “moving” token on the apparent motion trace19.
We believe the most plausible interpretation of our data is that the predictive codes fed back to V1 from higher cortical area V5 update to new retinotopic post-saccadic locations and amplify visual responses when they do not fit the temporal prediction triggered by the surrounding apparent motion illusion6,15. As V1 is acallosal35,36,37 predictive feedback must relocate via higher cortical areas. hMT/V5 + , as a motion sensitive region38,39 known to integrate long-range apparent motion13,15,16,19,40, is a likely candidate. Vetter et al., (2015), demonstrated a causal effect of feedback from V5 on predictive signals in V1 by showing a significant reduction in the detection of predictable in-time targets within the apparent motion trace after single-pulse transcranial magnetic stimulation to V5. From our fMRI data, we do not know the temporal dynamics of the prediction relocation21, but importantly, we do know it is effective in time for the post-saccadic processing.
Predictive coding theories of vision state that cortical predictions are compared with sensory signals to compute an error in the case of mismatch. These error signals update the brain’s internal models of the visual world to stabilize perception, cognition and behaviour. However, with each saccade, the visual system is flooded with a new pattern of sensory signals. Therefore, to be functional, predictions must feed back to the relevant retinotopic location at which their sensory counterpart will arrive to V1 after each saccade. We show that predictions do update retinotopic position in time for post-saccadic input to V1. Hence, predictive coding is an ecologically viable mechanism of the human visual system.
We recruited 27 healthy subjects with normal or corrected-to-normal vision (15 male; 19–34 years) using the University of Glasgow, School of Psychology subject pool. Subjects gave written informed consent. The ethics committee of the College of Science and Engineering, University of Glasgow provided ethical approval. All methods were performed in accordance with the relevant guidelines and regulations provided by the ethics committee of the College of Science and Engineering. We performed a power analysis based on data from Alink et al., (2010) where subjects performed a very similar apparent motion task in the fMRI, but without eye-movements. The power analysis estimated a minimum sample size of n = 5 (effect size = 0.06, standard deviation = 0.04, null hypothesis = 0, type I error rate = 2.5%, power = 0.8). We then performed a pilot experiment in which we combined the apparent motion effect with eye-movements. The subsequent power-analysis of five pilot-subjects confirmed the effect size from Alink et al., (2010) (see supplemental material, effect size = 0.08, standard deviation = 0.05, null hypothesis = 0, type I error rate = 2.5%, power = 0.8). For the main experiment we settled for a group size of 10, well above the estimated minimum group size for these effects. Due to the saccade criterion (see further in methods), one subject was removed in the Main apparent motion experiment (leaving n = 9), two subjects were removed from the flicker control experimental analysis (leaving n = 5), and two subjects were removed from the analysis of the Replication experiments (leaving n = 8). Two subjects who took part in the flicker control also performed the replication experiments (CV & GV).
Apparent Motion Stimulus
We presented the apparent motion stimulation in the center of a grey screen (RBG: 153, 153, 153; Fig. 1). We induced the illusion of vertical motion by flashing two white rectangles in an alternating rhythm at a frequency of 3.75 Hz (Fig. 1). These two white rectangles were either 14.84° (pilot experiment) or 8.84° of visual angle apart from each other (main experiment). Each rectangle was presented for 5 frames (83.3ms), followed by an inter-stimulus interval of 5 frames. In the pilot experiment, we presented a green fixation cross (2.1° visual angle in size) 10.27° to the left of the apparent motion illusion centre, and a red cross (sized 2.1°) 10.27° to the right of the illusion. In the main fMRI experiment, the fixation crosses were 0.7° size and positioned 5.81° horizontally from the illusion. Subjects fixated on the red cross. The red cross alternated horizontally with the green cross prior to the 7th cycle of apparent motion, approximately 2 s after the apparent motion onset, cuing subjects to saccade across the illusion (Fig. 1a–d). Shortly after the subjects’ saccade landed, a target appeared (for one frame of 16.67 ms) during the 8th cycle and the illusion ceased. This design ensured that the right hemisphere processed the illusion (and a prediction was generated in higher visual areas) while the left hemisphere processed the target. Targets presented on the apparent motion trace were either in-time or out-of-time with the illusion. In-time targets were presented approximately 83 ms after saccade offset and out-of-time targets approximately 50 ms after saccade offset (which was 117 ms and 150 ms into the 8th cycle respectively). There were three apparent motion conditions: in-time target, out-of-time target and no target. Our previous data revealed that subjects take approximately 300 ms between saccade cue and saccade completion21. In order to avoid saccadic suppression effects on the BOLD signal in V1, we designed the experiment so the earliest target was presented 450 ms after the red fixation cross shifted location, in the 8th cycle of apparent motion. Therefore, we presented targets approximately 150 ms after projected saccade onset, which allows a 100 ms buffer outside the saccadic suppression window found in previous research41,42,43,44. Specifically, we presented in-time targets 483 ms after projected saccade onset and out-of-time targets approximately 450 ms after projected saccade onset (which was 117 ms and 150 ms into the 8th cycle respectively). After the 8th cycle, subjects were presented with the question ‘target?’ indicating that they should respond ‘yes’ or’no’ via button-press. Subjects had to respond within 1500 ms, there was then an inter-trial-interval of 1333 ms prior to the beginning of the next trial. At the end of the interval, there was a cross colour change to indicate that subjects must fixate back to the right of the screen before a new trial began. In total 6 conditions were presented: 60% apparent motion conditions (3 conditions of 20% of each), 20% of the mapping conditions (2 conditions, 10% each), and 20% of the baseline condition. In the baseline condition participants were only presented with fixation and alternating cross, without any saccade required. Participants were instructed and trained to only saccade when there was a cross-colour change. In the baseline condition, no cross-colour change occurred, and no saccades were performed. Our replication apparent motion experimental design had no alterations from the original design.
We presented mapping conditions in the same runs as the apparent motion illusion. Mapping conditions comprised static images of the lower apparent motion inducer and the target, and were both presented for 4000 ms (Fig. 1b). During the mapping conditions, participants fixated on a red fixation cross presented to the left of the mapping stimuli. These conditions enabled us to map the separate locations in V1 that respond to the inducing stimuli and the target (GLM contrast of Target > Lower inducing stimulus: beta-weight = 0.88, t = 18.53, p < 0.0001 in the main fMRI experiment; beta-weight (SE) = 0.92(0.06), t = 18.81, p < 0.0001 in the flicker control experiment; beta-weight (SE) = 1.139(0.043), t = 26.221, p < 0.0001 in apparent motion replication experiment; beta-weight (SE) = 4.313(0.185), t = 23.368, p < 0.0001 in flicker control replication experiment).
Flicker control stimulus
We tested five new subjects in an fMRI experiment. The flicker control stimulation was identical to the apparent motion stimulation apart from the two blinking inducer stimuli were presented simultaneously instead of in alternating rhythm. The upper and lower inducers were presented in unison for 5 frames (83.3ms), followed by an inter-stimulus interval of 5 frames, resulting in a flicker at the same rate as the apparent motion stimulus (3.75 Hz; Fig. 1e). All other parameters remain the same as the main fMRI experiment. We cued subjects to saccade across the flicker stimulus and we presented targets after the saccade in the final flicker cycle, with the same timings as in the main experiment. The flicker stimulus was processed by the right V1 and the targets were presented to the left V1. No design alterations were made during our replication of the flicker control experiment.
Subjects completed four functional runs of 15 minutes each. In the main fMRI, flicker control, and replication experiments, subjects viewed stimuli on an fMRI compatible screen positioned in the bore of the magnet at a distance of 110 cm (screen resolution: 1024 × 768). We used Neurobehavioral systems Presentation® software (Version 14.9) to present the paradigm, with a screen refresh rate of 60 Hz.We presented the three apparent motion conditions and a baseline condition of 25 times per run and 100 times across the whole experiment. We presented the two mapping conditions 12 times each per run and 48 times across the experiment. We used a randomisation scheme to order the trials, ensuring that no triplets of conditions were repeated.
MRI Data Acquisition
We acquired functional and anatomical MRI data using a 3 Tesla MRI system (Siemens Tim Trio) with a 12-channel head coil (and a 32-channel head coil for the control and replication experiments). For the functional scans, an echo-planar imaging sequence had the following parameters: 17 slices, TR-1, TE-30, 860 volumes per run, an FOV of 205 mm, and a resolution of 2.5 × 2.5 × 2.5 mm. The 17 slices were orientated perpendicular to the calcarine sulcus to capture the early visual cortex. The anatomical MRI sequence used had a TR of 1.9, 192 volumes, and a resolution of 1 × 1 × 1 mm. fMRI parameters and analyses were identical in main, pilot, flicker control, and replication experiments.
In the main experiment and flicker control, subjects’ eye-movements were recorded using an Eyelink 1000 (SR Research) mounted on the fMRI compatible projector screen with a sampling rate of 500 Hz (calibrated at the start of each run). The data was recorded by Eyelink software and downloaded for analysis using Eyelink Data Viewer.
The saccade criterion denoted that subjects had to complete a saccade 400 ms after cue, and the saccade must cover at least 200 pixels horizontally across the apparent motion between onset and offset (Figure S5a). The criteria ensured that subjects processed the apparent motion in the right hemisphere and the target in the left. Specifically, in-time and out-of-time targets were presented at least 83ms and 50ms after latest saccade offset respectively, which was 483 and 450ms after cue to saccade respectively. Subjects had to land their saccade between the cross colour change + 333 ms and 400 ms (i.e. within a 67ms time-window), to ensure that trials are outside the saccadic suppression window of 50 ms after saccade onset44. Our analysis revealed in-time and out-of-time targets were presented 159.4 ms and 144.8 ms after saccade onset in the apparent motion experiment, and in-time and out-of-time targets were presented 168.8 ms and 147.8 ms after saccade onset in the flicker experiment respectively. These latencies are beyond the range of saccadic suppression41,42,43,44. We excluded trials where a saccade did not meet the criterion. In the main fMRI experiment one subject and one run from two other subjects who showed less than 20 trials per run with a correct saccade were excluded (Figure S5b). Two subjects were excluded from the flicker control experiment as the saccade criterion reduced their successful saccade trials to less than 20 per run. We removed a further two subjects and 1 run from one other subject due to less than 20 successful trials per run in the replication experiments.
The functional and anatomical data were analyzed using Brainvoyager QX® software (Version 2.4). We discarded the first two volumes of each functional run to avoid saturation effects. To remove low-frequency noise and drift, we performed high-pass filtering at 6 sines/cosines during the 3D-motion correction for each run. After preprocessing, we aligned the functional data with the high-resolution anatomical data and transformed into Talairach space (applying each subject’s brain in to a common space along the AC-PC plane). We created 3D aligned time courses (VTCs) for each run after the intra-session anatomical was aligned with the high resolution anatomical. Each subject’s cortex was inflated into a surface model using the manually inhomogeneity corrected high resolution anatomical.
Once we removed invalid eye-tracking trials, we performed single subject and group deconvolution analysis. We contrasted beta-weight values over 3–7 seconds after stimulus onset in the target processing region and apparent motion inducer processing region of left V1 separately, corresponding to the time when the targets were presented in the apparent motion trials compensating for BOLD lag. We performed the same analysis in the left V2, right V5, and left V5. We normalised our data to baseline (fixation only) before performing our analyses. We contrasted the response amplitudes (expressed in beta-weight values of a general linear model (GLM) analysis) for in-time and out-of-time target trials using a serial correlation corrected comparison to determine the activation difference. Response amplitudes were also contrasted for flicker with in-time targets and flicker with out-of-time targets in the control analysis. We used a deconvolution analysis in order to analyse the BOLD responses between different conditions. The same analyses were performed in the replication experiments.
Retinotopically defining regions of interest (ROI)
The region of interest in left V1 was identified using a GLM contrast of target > lower inducing stimulus (mean (SD) Talairach co-ordinates: x = −11.33 (4.4), y = −89.67 (2.7), z = −6.3 (7.5); FDR = 0.05; Fig. 2). Left V2 was defined using the target > lower GLM contrast (mean (SD) Talairach co-ordinates: x = −18.11 (5.6), y = −94 (3.9), z = 1.89 (6.4); FDR = 0.05). The target V1 ROI was selected along the calcarine sulcus for each participant. V2 was located dorsally to the V1 ROI. We determined V1 and V2 boundaries using independent data measuring the cortical representation of phase-encoded polar angle mapping. The mirrored representation of V1 and V2 allowed us to identify the gradient inversion of phase encoded polar angle activation maps. The ROIs for right V5 and left V5 and right V1 were defined using apparent motion without target condition > baseline GLM contrast as no mapping data were collected for these regions (mean (SD) Talairach co-ordinates for right V5: x = 43.22 (4.4), y = −65.78 (4.1), z = 0.89(5.3); left V5: x = −45.22 (6.0), y = −70.44 (3.2), z = −0.89 (4.2); right V1: x = 7.78 (3.3), y = −80.89 (6.9), z = −1.78 (7.0); FDR = 0.05). The apparent motion inducer processing region of left V1 was also selected for control analysis using lower inducer > target contrast (mean(SD) Talairach co-ordinates: x = −5.89(5.3), y = −90.67 (6.2), z = 0.22(6.9); FDR = 0.05). For the flicker experiment, the target processing region in V1 was also defined using target > lower inducing stimulus (mean (SD) Talariach co-ordinates: x = −11.6 (4), y = −86.4 (3.4), z = −5.2 (7.2). We defined all ROIs with the same contrasts in the replication experiments.
We recorded behavioural data using Neurobehavioral Systems Presentation® software during the fMRI runs. We excluded one subject in the main fMRI experiment due to data recording issues. We conducted the analysis after applying the saccade criteria. Target detection accuracy was averaged per subject and then across all subjects. Analysis focused on accurate detection difference of in-time versus out-of-time targets.
Psychophysical control experiment
It was our objective to have the target prediction effect generated by pre-saccadic processing and not by post -saccadic, post-diction mechanisms. For this reason, the target always appeared in the post -saccadic stimulated right visual hemi-field, directly after saccade. The pre-saccadic illusion perception primed the visual system for prediction effects prior to subjects performing the saccade. After subjects executed the saccade, the post-saccadic stimulated hemisphere processed only the target and the lower apparent motion inducer. It is the top-down cross-hemispheric prediction of motion that leads to the difference between in-time and out-of-time target presentations. In 20% of trials, no target appeared but in 40% of the trials an in-time or an out-of-time target (20%/20%) appeared, allowing us to measure brain-activity differences between these conditions. In the remaining 40% of trials, we presented the mapping conditions and baseline. The target trials (40%) were effective in inducing different BOLD activations but did not create any behavioural effects in terms of target detection. This may be because subjects knew the target would be presented after saccade. We have shown robust behavioural effects previously6,15,20,21,22 using a paradigm in which targets were also presented mid-fixation (thus not always after saccade). We aimed to replicate this behavioural detection advantage in the current experiment. We ran an extra -session behavioural experiment using a continuous apparent motion paradigm to enable target presentation both after saccade and during fixation.
Nine subjects (6 male; 19–28 years) who participated in fMRI experiment 2 also completed the psychophysical counterpart. We excluded one of these subjects from analysis using the same saccade criterion used for trial exclusion during fMRI (Figure S5). The ethics committee of the College of Science and Engineering, University of Glasgow provided ethical approval.
We presented the paradigm using Neurobehavioral Systems Presentation® (Version 14.9) software, using identical parameters for the three apparent motion conditions as in the fMRI experiments (see Apparent Motion Stimulus). However, the apparent motion stimulation was continuous (onset and offset of each trial were not detectable) and no mapping conditions were presented. Subjects were cued to saccade across the apparent motion illusion every 2.66 s. Trials consisted of 10 cycles of apparent motion, and targets were presented in the cycle directly after saccade (at the same time as in the fMRI experiment) or during fixation.
We presented the three apparent motion conditions at random within 5 runs of the experiment. In 60% of the 152 trials per run, we presented targets directly after saccade in the same time-period as was used for the fMRI experiment. We presented a target mid-trial during fixation in 20% of trials, and in the remaining 20% of trials we did not present a target. This design decreased the probability of targets appearing after the saccades. Every 25 trials, the apparent motion was interrupted for 20 s with a natural scene to allow subjects to rest their eyes and prevent apparent motion breakdown45.
Task & Procedure
Subjects were seated 70 cm from a 16 inch Sony Trinitron CRT Monitor (1024 × 768; 60 Hz), upon which we presented the stimuli. Subjects’ heads were supported using chin and forehead rests. We recorded subjects’ eye-movements continuously throughout the experiment (EyeLink 1000, SR Research; acquisition as in fMRI method). We instructed subjects to fixate on the red fixation cross and move their eyes across the illusion when the red cross alternated with the green. We instructed subjects to indicate target detection with a button press.
The behavioural data was recorded using Presentation® software. We applied the same eye-tracking criterion for the fMRI data analysis to each trial of the psychophysical experiment. Detection of targets was only included if the button press occurred within 150 and 1200 ms after target onset. Target detection accuracy was averaged per subject and then across all subjects. Analysis focused on accurate detection difference of in-time versus out-of-time targets.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was supported by BBSRC grant (BB/G005044/1), ERC grant StG 2012_311751- Brain reading of contextual feedback and predictions’) and Human Brain Project grant from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No. 720270 (HBP SGA1).(‘Context-sensitive multisensory object recognition: a deep network model constrained by multi-level, multi-species data’).