Visual working memory directly alters perception

Abstract

Visual working memory (VWM), the ability to temporarily maintain and manipulate information, underlies a variety of critical high-level behaviours from directing attention1,2,3,4 to making complex decisions5. Here we show that its impact extends to even the most basic levels of perceptual processing, directly interacting with and even distorting the physical appearance of visual features. This interference results from and can be predicted by the recruitment of posterior perceptual cortices to maintain information in VWM6,7,8,9, which causes an overlap with the neuronal populations supporting perceptual processing10,11,12,13,14,15. Across three sets of experiments, we demonstrated bidirectional interference between VWM and low-level perception. Specifically, for both maintained colours and orientations, presenting a distractor created bias in VWM representation depending on the similarity between incoming and maintained information, consistent with the known tuning curves for these features. Moreover, holding an item in mind directly altered the appearance of new stimuli, demonstrated by changes in psychophysical discrimination thresholds. Thus, as a consequence of sharing the early visual cortices, what you see and what you are holding in mind are intertwined at even the most fundamental stages of processing.

Main

Traditionally, visual working memory (VWM) is thought to influence perception via its interaction with selective attention16,17,18,19,20,21, which arises from the neural substrate they share in the frontoparietal network22,23,24,25,26. However, recent studies report that information in VWM is maintained within the perceptual cortices in which it was initially processed, including orientations6,7, motion directions8,9, visual spatial patterns14 and visual objects15, suggesting a sensory recruitment model for VWM maintenance10,11,12,13. Further, transcranial magnetic stimulation of posterior parietal and occipital sites can lead to domain-specific impairments for object-based and spatial VWM27, as well as changes in the precision of remembered motion directions and orientations28,29,30. However, the strength of the transcranial magnetic stimulation evidence and whether the representation in the sensory areas is behaviourally meaningful has been questioned31,32,33. Here we test a key behavioural prediction of the sensory recruitment account. If neuronal resources in sensory areas are shared between VWM and bottom-up perceptual processing, the contents of VWM should interact directly with the perception of ongoing stimuli, simultaneously altering their appearance and being altered themselves.

Behaviourally, the maintenance of information in early visual areas predicts stimulus-specific and bidirectional interference between VWM and perception. Previous research has demonstrated this type of bidirectional interference in the context of visual attention and VWM. VWM content biases visual search1,2,3,4,18,34 and, in turn, distracting visual information interferes with VWM performance in change detection tasks35,36,37 and changes later memory report38,39. However, a few direct predictions from the common sensory recruitment by both perception and VWM remain untested. First, this overlap in early visual cortex would predict that ongoing perceptual processing can, in and of itself, change the contents of VWM even absent attention. In turn, VWM should be able to directly alter the physical appearance of current stimuli. Moreover, if this interference originates from primary sensory regions, it should reflect their known tuning properties40,41,42 for the features processed in those regions. Specifically, the amount of interference should reflect the similarity between the maintained and incoming stimuli along the feature dimension that defines the tuning, reflecting the extent of overlap in the recruitment of the neuronal population.

Across three different experimental paradigms, this study provides strong psychophysical evidence that this bidirectional interference between VWM and perception does occur and, moreover, its strength is predictable from the known tuning curves for both colour and orientation. Experiment 1 showed bidirectional interference between VWM and attention1,2,3,4 in a visual search task. The more similar a distractor was to the maintained content, the more effective it was at drawing visual attention and biasing subsequent VWM report towards itself. Experiment 2 removed the strong attentional demand created by visual search and showed that passively presenting the distractors in the background was sufficient to create the same graded interference in the memory report, providing further evidence for an early locus of interference between perception and VWM. Experiment 3 used a perceptual discrimination task to directly show that information maintained in VWM altered perception, predictably making stimuli appear either more similar or distinct depending on their relationship to the maintained information. Taken together, these results show that maintaining information in early visual cortices likely leads to direct perceptual consequences, opening a new domain in which to study top-down effects and suggesting that VWM may be more fundamental to perception than previously believed.

The first experiment tested whether the influence of VWM on selective attention and the impact of distracting stimuli of VWM are constrained by the known properties of tuning curves in the early visual areas. For basic visual features such as orientation and colour, in non-human primates, neurons show a monotonic decrease in firing rate with distance from the preferred value in the feature space40,41,42,43. If this tuning defines the population responses to both maintained and incoming perceptual information, the interference between the two should be modulated by their similarity in the feature space. Previous studies have reported that distractors and targets that match the content maintained in VWM capture attention more strongly than non-matching distractors1,2,3,4. Here we test whether attentional capture also creates distortion in the memory representation and whether these effects are graded with respect to similarity (such similarity effect has also been reported in similar visual search paradigms4,44).

While holding a colour (experiment 1a; Fig. 1a) or an orientation (experiment 1b; Fig. 1b) in mind, participants performed a search task in which the target was defined by an orthogonal feature (oriented line for experiment 1a and coloured circle for experiment 1b). Each search display included a distractor singleton that had varying similarity to the memory cue (0°, 15°, 35°, 55° or 75° in the CIE L*a*b* colour or orientation space; see Fig. 1c,d). For search performance, the extent of capture by the singleton was modulated by its similarity to the item in VWM (Fig. 1e,f and Supplementary Fig. 1), as revealed by significant one-way repeated measures analyses of variance (ANOVAs) with the factor similarity (five levels) conducted separately for colour (F(4,196) = 9.28, P < 0.001, partial eta squared \(\eta_{\mathrm{p}}^2\) = 0.159, 90% confidence interval (CI) \(\eta_{\mathrm{p}}^2\) = (0.076, 0.222) and orientation (F(4,196) = 3.135, P = 0.016, \(\eta_{\mathrm{p}}^2\) = 0.06, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.006, 0.105)).

Fig. 1: Bidirectional interference between VWM and selective attention.
figure1

a, Procedure in experiment 1a. Participants first memorized a colour, then searched for a tilted line (45° tilted to left or right) and reported its direction, then reported the remembered colour by clicking on a colour wheel. A colour disk would appear at the centre of the colour wheel with the clicked colour. The colour disk in the search display served as the critical distractor. In this example, the memory–distractor distance is 35°. ISI, interstimulus interval. b, Distractors in the search task differed in similarity from the colour cue by (0°), 15°, 35°, 55° or 75° in the colour space. c, Procedure in experiment 1b. While memorizing an orientation, participants searched for a red or green circle and ignored the critical distractor (oriented line). When they clicked on the empty circle to report the memorized orientation, a line would appear corresponding to the clicked location. d, The search distractors (oriented line) had varying similarity to the orientation cue. e, RT difference from the match condition as a function of memory–distractor distance in experiment 1a (N = 50). The more similar the colour distractor was to the memory cue, the larger the capture (slower RT) (F(4, 196) = 9.28, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.159, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.076, 0.222)). f, RT difference in experiment 1b (N = 50). The more similar distractors created larger attentional capture ((F(4, 196) = 3.135, P = 0.016, \(\eta_{\mathrm{p}}^2\) = 0.06, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.006, 0.105)). g, Normalized bias in the colour report in experiment 1a decreased with decreasing similarity between cue and distractor (F(3, 147) = 14.65, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.23, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.126, 0.311)). Positive value indicates bias towards the distractor. Only the four non-matching conditions were included. h, Normalized bias in experiment 1b decreased with decreasing similarity (F(3, 147) = 28.83, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.37, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.260, 0.448)). The orange tick marks indicate the condition where pairwise comparisons were conducted against. *P < 0.05. Error bars represent s.e.m. Supplementary Tables 1 and 2 list the full statistics.

Critically, participants also completed a subsequent continuous memory report to precisely measure the impact of the singleton on the VWM representation45,46. Consistent with the predicted extent of overlap in populational responses based on the known tuning curves, the degree of bias (error towards the distractor value and away from the testing value) was proportional to the similarity of the singleton to the item in VWM (Fig. 1g,h), as revealed by two separate repeated measures ANOVAs for both colour (F(3,147) = 14.65, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.23, 90%CI \(\eta_{\mathrm{p}}^2\) = (0.126, 0.311)) and orientation (F(3,147) = 28.83, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.37, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.260, 0.448)). The same pattern was found in the precision of the memory report (see Supplementary Fig. 2).

These results demonstrate that the contents of VWM affect selective attention and are themselves affected by the attended stimuli relative to the similarity between the distractors and the maintained information. Thus, these results verify the prediction of graded bidirectional interference between VWM and selective attention defined by similarity. Further, they raise the possibility that the common recruitment of early visual rather than the frontoparietal areas may be the neural mechanism underlying the VWM guided attentional capture effect1,2,3,4. For this effect to be originating from the frontoparietal network, it would require feature selectivity and tuning on par with that known to exist in perceptual cortices, which is unlikely. However, with the strong attentional demands inherent in visual search paradigms, there remains the possibility the effect is dependent on the engagement of the frontoparietal attentional circuit20,21. The next experiment demonstrates that the impact of visual processing on VWM occurs even in a task without a strong attentional demand.

In experiment 2, the embedded visual search task was replaced with an rapid serial visual presentation (RSVP) task wherein the interfering stimuli were presented as irrelevant backgrounds (Fig. 2a,b). Participants monitored a stream of black letters while holding a colour (experiment 2a) or orientation (experiment 2b) in mind and responded when they detected a white letter. Simultaneously, an irrelevant colour disk or oriented Gabor patch was presented in the background with varying similarity to the item held in memory. This task is known to suppress attention47 and therefore should minimize the engagement of attention on the interfering stimuli. Under the sensory recruitment account, the processing of the background stimuli in early visual cortices should be sufficient to alter the memory representation, regardless of whether it is attended. Alternatively, if the engagement of the attentional control network is responsible for the effect, the graded interference observed in experiment 1 should be greatly diminished or absent.

Fig. 2: Ongoing perception alters VWM representation.
figure2

a, The task in experiment 2a (N = 50) consisted of a colour working memory task and an orthogonal RSVP task embedded in between the memory cue and the probe. There were 8 displays in the RSVP stream (0.2 s per display) and participants monitored the colour change of the central letter and responded to the white letter. Here the target is the white ‘S’. Throughout the RSVP stream, there was a colour disk passively presented as the background and its similarity to the memory cue was manipulated. b, Experiment 2b (N = 50) started with an orientation memory cue, followed by a RSVP task with an orientation Gabor at the background, and the memory report at the end. The similarity between the memory orientation and the background Gabor was manipulated. c,d, We observed once again that similar distractors biased the mean memory report towards itself to a greater extent for both colour (c; F(3, 147) = 17.79, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.266, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.159, 0.347)) and orientation (d; F(3, 147) = 6.56, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.12, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.037, 0.189)) dimensions when distractors were perceived merely passively. *P < 0.05 for pairwise comparisons. The orange tick marks indicate the condition against which pairwise comparisons were conducted. Error bars represent s.e.m. Supplementary Table 3 lists the detailed statistics for all the pairwise comparisons.

As in experiment 1, the amount of bias introduced in the memory report showed a monotonic relationship with the similarity between the maintained item and the background distractor for remembered colours (Fig. 2c; F(3,147) = 17.79, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.266, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.159, 0.347)) and orientations (Fig. 2d; F(3,147) = 6.56, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.12, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.037, 0.189)). The same tuning curve patterns were replicated in the precision of the VWM report (see Supplementary Fig. 2). We further conducted a mixed factor ANOVA with similarity as the within-subject variable and experiment version (experiment 1 versus 2) as the between-subject variable to compare the magnitude of the observed bias in the current task from experiment 1. The main effect of experiment version was not significant for orientation VWM (F(1,98) = 0.988, P = 0.323, \(\eta_{\mathrm{p}}^2\) = 0.01, 90% CI \(\eta_{\mathrm{p}}^2\) = (0, 0.065)). Interestingly, the magnitude of bias in experiment 2 was even greater than experiment 1 for colour VWM (F(1,98) = 37.84, P < 0.001, \(\eta_{\mathrm{p}}^2\) = 0.28, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.159, 0.386)). Therefore, even in a task likely to suppress the interfering stimuli and with no required shifts of attention, perceiving the distractor created a significant amount of bias, consistent with these effects arising from the early visual cortices. The increase in the strength of the effect observed for colour may be due to the match in the retinotopic position of the remembered item and interfering stimulus, consistent with the known retinotopy in early visual cortices48,49.

These results show that the impact of ongoing perception on VWM is not dependent on a task requiring shifts of attention, which argues against this interaction arising entirely within the attentional control network. These results are highly suggestive of interference occurring within the perceptual cortices; however, neither experiment 1 nor 2 has yet shown a direct impact of VWM on perception, only a bias in attentional selection and an influence of ongoing perception on VWM. Experiment 3 directly tested whether VWM can change perceptual discrimination.

If VWM content is maintained within the perceptual cortices, it should directly alter ongoing perception, just as visual attention is known to change tuning curves in V450, preactivate portions of visual field maps in early visual areas51, and change behavioural sensitivity to contrast, motion and colour52,53,54. The following experiment leveraged the behavioural paradigms used to show the impact of attention on perception to test the hypothesis that VWM can facilitate or impair the discriminability of subtle stimulus differences depending on the relationship between the maintained and incoming stimuli.

Presented with a coloured Gabor patch, participants memorized either its colour or orientation. During the memory delay, they performed a discrimination task on colour or orientation, resulting in four combinations (between-subject; Fig. 3a) in which the memorized and discriminated features either matched or differed. We expected to observe the hypothesized effect of VWM on perceptual discrimination thresholds (lower thresholds in the ‘side’ than ‘middle’ condition) only when the critical feature in both tasks matched (colour discrimination: experiment 3a; orientation discrimination: experiment 3b). In fact, the stimuli in all four of the conditions were identical and the conditions differed only in the instructions given to the participants. This design allowed a direct comparison of the effects of maintaining a stimulus in VWM (match conditions) versus merely perceiving a task-irrelevant stimulus (non-match conditions) during the memory phase, ruling out explanations such as perceptual priming and adaptation for any observed effects.

Fig. 3: Testing the impact from VWM to perception in experiment 3.
figure3

a, Experiment 3 consisted of a colour or orientation memory task and an embedded colour or orientation discrimination task. In the match conditions, the feature dimension of the memory task matched with the discrimination dimension (colour memory and colour discrimination; orientation memory and orientation discrimination). The non-match conditions were implemented as controls in which the dimensions for the two tasks did not match (orientation memory and colour discrimination; colour memory and orientation discrimination). The displays were exactly the same for these four combinations and what varied was the task instruction. b, Manipulation of the relationship between memory cue and the discrimination stimuli using colour as an example. In the middle condition, the memory colour (grey line) was in the middle of the two discrimination stimuli (black lines). The underlying tuning curve of the memory colour would pull the population responses of the discrimination colours towards itself. The perceived colours would appear to be more similar. In the side condition, the response of the closer discrimination colour would be pulled towards the memory colour to a greater extent and increase the perceived difference between the two colours.

Critically, in all four conditions, we manipulated the relationship between the remembered item and the discrimination stimuli in the colour/orientation space. The item could either be in the middle of the two discrimination stimuli or on one side to them, ‘middle’ and ‘side’ conditions, respectively (Fig. 3b). The colour and orientation of the Gabor were manipulated simultaneously and set in opposition, such that, for example, when the colour relationship was ‘side’, the orientation relationship was always ‘middle’.

First, assuming that VWM content is maintained in the same neuronal populations that are processing current stimuli, the remembered stimulus should bias the population responses of the discrimination stimuli towards itself. In the middle condition, the perceived colour difference would be smaller because both responses are strongly pulled towards the remembered colour (Fig. 3b, top). In contrast, in the side condition, the closer colour would be biased towards the memory colour to a larger extent than the farther colour, resulting in a bigger perceived difference, consistent with the known population tuning curves and the previous results (Fig. 3b, bottom). Therefore, the discrimination threshold should be lower for the side condition than the middle condition. The second prediction was that this effect of VWM on perception should be contingent on the match of the remembered and discriminated feature, with qualitatively stronger effects when they matched than when they differed. Finally, consistent with experiment 2, we also expected to observe bias in VWM report regardless of task relevance in the side, but not middle, condition, as the discrimination stimuli in the middle condition should introduce equivalent and opposite going biases.

The results showed that VWM content changed the discrimination threshold as predicted with lower discrimination thresholds in the side condition when the remembered and discriminated feature matched than when they did not (Fig. 4a). When examining the psychometric curves in detail, in the match conditions for both colour and orientation, there was a shift to the left in the psychometric function in the side condition compared with the middle condition (Fig. 4b,c). For colour, the fitted threshold from a Weibull function55 for 65% reporting ‘different’ was significantly lower in the side condition than the middle condition, revealed by the pairwise comparison (t(29) = 4.20, P < 0.001 (two-tailed), Cohen’s d = 0.77, 95% CId = (0.353, 1.170)). This shift in threshold reflected significant pairwise differences between the conditions for colour differences of 9° (t(29) = 2.91, P = 0.007 (two-tailed), Cohen’s d = 0.531, 95% CId = (0.144, 0.910)), 12° (t(29) = 2.59, P = 0.015 (two-tailed), Cohen’s d = 0.473, 95% CId = (0.091, 0.847)) and 15° (t(29) = 2.23, P = 0.033 (two-tailed), Cohen’s d = 0.407, 95% CId = (0.032, 0.777)). For orientation, the fitted threshold was significantly lower in the side than the middle condition (t(29) = 4.594, P < 0.001 (two-tailed), Cohen’s d = 0.839, 95% CId = (0.416, 1.251)). Simple pairwise comparisons without corrections showed significant differences for 4° (t(29) = 3.39, P = 0.002 (two-tailed), Cohen’s d = 0.619, 95% CId = (0.223, 1.005)), 6° (t(29) = 3.42, P = 0.002 (two-tailed), Cohen’s d = 0.625, 95% CId = (0.229, 1.012)), 8° (t(29) = 3.73, P = 0.001 (two-tailed), Cohen’s d = 0.680, 95% CId = (0.277, 1.074)) and 10° (t(29) = 3.48, P = 0.002 (two-tailed), Cohen’s d = 0.635, 95% CId = (0.237, 1.023)) orientation differences.

Fig. 4: VWM changes the appearance of colour and orientation, reflected by the changes in discrimination thresholds.
figure4

a, Summary figure: the difference in discrimination thresholds between side and middle conditions plotted as a function of memory and discrimination dimensions. In general, there was a greater difference when the two task dimensions matched than when they were non-matched. b, In the colour-match condition (N = 30) when memory and discrimination dimensions were both colour, the colour discrimination threshold was lower in the side condition than the middle condition (t(29) = 4.20, P < 0.001, Cohen’s d = 0.77, 95% CId = (0.353, 1.170)). c, Consistently, the orientation discrimination threshold in the match condition (N = 30) was lower in the side condition (t(29) = 4.594, P < 0.001, Cohen’s d = 0.839, 95% CId = (0.416, 1.251)). d, In the non-match condition with orientation memory and colour discrimination (N = 30), although the stimuli were exactly the same as in the match condition, the difference in the threshold failed to reach significance (t(29) = 0.100, P = 0.921, Cohen’s d = 0.018, 95% CId = (0, 0.376)). e, In the non-match condition (N = 30) with colour memory and orientation discrimination, the difference in the threshold was significant (t(29) = 2.62, P = 0.014, Cohen’s d = 0.510, 95% CId = (0.096, 0.853)), but significantly smaller than in the match condition (F(1,58) = 4.42, P = 0.040, \(\eta_{\mathrm{p}}^2\) = 0.071, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.002, 0.191)). f, We replicated the colour-match condition (N = 48) and found a lower threshold in the side condition (t(47) = 3.81, P < 0.001, Cohen’s d = 0.550, 95% CId = (0.244, 0.851)). g, For the replication of the non-match condition with orientation memory and colour discrimination (N = 48), merely perceiving the colour of the memory cue did not create a significant difference (t(47) = 0.184, P = 0.855, Cohen’s d = 0.027, 95% CId = (0, 0.309)). *P < 0.05 for pairwise comparisons (all two-tailed, see Supplementary Tables 4 and 5 for full statistics). Error bars represent s.e.m.

Critically, this effect of VWM on perception was modulated by task relevance, with greater differences observed when the memory dimension and the discrimination dimension matched. When remembering orientation and performing a colour discrimination, whether the colour of the memory cue was side or middle to the discrimination stimuli did not influence the colour discrimination threshold (Fig. 4d). The pairwise comparison on the fitted thresholds between side and middle conditions did not reach significance (t(29) = 0.100, P = 0.921 (two-tailed), Cohen’s d = 0.018, 95% CId = (0, 0.376)). None of the pairwise comparisons on the raw data reached significance (see Supplementary Table 4 for statistical reports). The interaction between the effect of middle versus side and match versus non-match did not reach significance at the chosen alpha level of 0.05 (F(1,58) = 3.73, P = 0.055, \(\eta_{\mathrm{p}}^2\) = 0.062, 90% CI \(\eta_{\mathrm{p}}^2\) = (0, 0.177)). To further confirm the reliability and more firmly quantify the strength of this effect, we conducted a direct replication with a new group of participants for both the match and non-match conditions with colour discrimination. Again, there was a significant difference in fitted threshold between the middle and side conditions when the memory and discrimination dimensions matched (Fig. 4f) (t(47) = 3.81, P < 0.001(two-tailed), Cohen’s d = 0.550, 95% CId = (0.244, 0.851)). Simple pairwise comparisons showed a significant difference between middle and side for 6° (t(47) = 2.62, P = 0.012 (two-tailed), Cohen’s d = 0.379, 95% CId = (0.084, 0.670)), 9° (t(47) = 3.65, P = 0.001 (two-tailed), Cohen’s d = 0.527, 95% CId = (0.222, 0.826)), 12° (t(47) = 2.59, P = 0.013 (two-tailed), Cohen’s d = 0.374, 95% CId = (0.080, 0.665)) and 15° (t(47) = 2.16, P = 0.036 (two-tailed), Cohen’s d = 0.312, 95% CId = (0.020, 0.600)) colour differences. For the non-match condition, the difference in fitted threshold between middle and side did not reach significance (Fig. 4g) (t(47) = 0.184, P = 0.855 (two-tailed), Cohen’s d = 0.027, 95% CId = (0, 0.309)) nor did any of the pairwise comparisons (all P > 0.17, see Supplementary Table 4). Further, when conducting a mixed factor ANOVA with middle versus side as the within-subject factor and match versus non-match as the between-subject factor, we observed a significant interaction between the two factors on fitted thresholds (F(1,94) = 6.04, P = 0.016, \(\eta_{\mathrm{p}}^2\) = 0.06, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.006, 0.150)).

For orientation discrimination with colour memory, the same pattern of results was observed (Fig. 4e). Only the pairwise comparison for 8° difference was significant, t(29) = 3.53, P = 0.001 (two-tailed), Cohen’s d = 0.644, 95% CId = (0.245, 1.033)). Supplementary Table 5 presents the full statistics for all the orientation gaps. Although there was a significantly lower threshold in the side than the middle condition (t(29) = 2.62, P = 0.014 (two-tailed), Cohen’s d = 0.510, 95% CId = (0.096, 0.853)), the difference was significantly larger in the match (0.88°) than non-match condition (0.39°) (F(1,58) = 4.42, P = 0.040, \(\eta_{\mathrm{p}}^2\) = 0.071, 90% CI \(\eta_{\mathrm{p}}^2\) = (0.002, 0.191)) (Fig. 4e) as revealed by the mixed factor ANOVA with middle versus side as the within-subject factor and match versus non-match as the between-subject factor.

Finally, consistent with experiment 2, perceiving the discrimination stimuli affected the memory representation regardless of their task relevance. There was significant bias towards the discrimination stimuli in the mean memory report in the side condition regardless of whether the remembered and discriminated stimuli matched or differed (all P < 0.001 for all four conditions, Supplementary Fig. 3; the pattern held for the replication, Supplementary Figs. 4 and 5). When the memory cue was in the middle of the discrimination stimuli, the interference from each side was equivalent and no bias was observed in any of the conditions (all P > 0.16, see Supplementary Fig. 3). As predicted, the degree of bias did not vary with the relevancy of the memory dimension to the discrimination task for colour (F(1,58) = 1.66, P = 0.20, \(\eta_{\mathrm{p}}^2\) = 0.03, 90% CI \(\eta_{\mathrm{p}}^2\) = (0, 0.126) or orientation (F(1,58) = 0.52, P = 0.47, \(\eta_{\mathrm{p}}^2\) = 0.01, 90% CI \(\eta_{\mathrm{p}}^2\) = (0, 0.085)).

These results show, critically, that content maintained in VWM can change the appearance of colours/orientations and alter discrimination thresholds. Consistent with an early visual cortical locus of interference, perception altered the VWM representation regardless whether the memory dimension of the discrimination stimuli was task relevant or not. Taken together, these results provide support for the hypothesis of the local maintenance of VWM content within perceptual cortices.

Building on recent neuroimaging studies that have highlighted the presence of maintained VWM information in perceptual cortices6,7,8,9,14,15, across three different paradigms and two independent features, the current study demonstrated the predicted behavioural consequences of the common use of early visual areas by VWM and ongoing perception. We found that the interaction between the processes is (1) bidirectional, with VWM distorting ongoing perception and perceptual input changing the content of VWM, (2) qualitatively follows the known tuning curve properties of orientation and colour in the visual cortex, and (3) is present even the absence of strong attentional demands. These results are direct demonstration of the perceptual consequences of this sensory recruitment mechanism10,11,12,13 of VWM maintenance. These results converge with neuroimaging evidence39,56 suggesting that the sensory representation of VWM directly corresponds to behaviour. Further, these results are also consistent with studies reporting that visual imagery evokes a qualitatively similar activation pattern as bottom-up visual processing57 and changes detection sensitivity for perceptual input58.

Thus, VWM should be viewed as a distributed mechanism that efficiently recruits areas that initially process information during a non-working memory context10,11,12,13,59 for maintenance. It should be noted that these results do not rule out a contribution of the prefrontal cortex (PFC) in VWM maintenance. Elevated delay-period activity in the PFC may serve as top-down input for the upkeep of sustained patterns in sensory cortices9,60 and the region has also been shown to represent task demand9,61,62 and directly maintain abstract categorical information15. However, it is unlikely that lower-level VWM content is stored in the PFC, given the unsuccessful decoding of visual details in the PFC6,8,9,15 and our current findings. Further study is needed to elucidate the exact mechanisms that support VWM maintenance in perceptual cortices, whether through low-level of distributed activity6,8,9 or activity-silent synaptic weight changes29,63.

These findings also suggest a possible reinterpretation of the observed impact of VWM on attention that is often described as top down or goal directed64, rooted in the traditional view that this biasing signal originates from high-order cortices. The fact that the memory interference persisted in absence of strong attentional demands implies that this effect can happen with minimal engagement of the frontoparietal network. Further, given our results showing a direct impact of VWM content on perception and that memory-guided attentional capture effects qualitatively mirror tuning curve properties, the previously described capture effects1,2,3,4 could be, at least in part, a product of local interactions in perceptual cortices. If this is the case, it is not surprising that memory-guided effect can override the saliency of stimuli65, suggesting a lower-level cortical locus contributes to the effects of VWM on attentional selection. Considering their common recruitment of perceptual cortices and the mutual interference, our results provide empirical support for the view that working memory and selective attention rely on related mechanisms in the perceptual cortices17,66.

More generally, these results open up a new domain in which to compare and contrast goal-directed and stimulus-driven visual effects. The two processes have generally been considered segregated, with top-down processing heavily relying on the frontoparietal network and bottom-up processing originating from occipital visual areas20. Nevertheless, if the perceptual specifics of the task (for example, the light pink colour you are looking for) are actually maintained in the visual cortex, then it may operate via a similar mechanism as perceptual priming or cueing within the visual cortex as an extended hysteresis in neural activation, boosting the strength of the representation of relevant information. Moreover, this framework suggests that in a task context that relies on low-level visual information, different cognitive processes, such as visual imagery, VWM, attention, perceptual priming and adaptation, might all create similar and perhaps interacting changes in the perceptual cortices. This mechanism does not fully equate these processes, but does provide a context in which they can be directly compared: the dynamics of the perceptual cortices. For example, perceptual priming and adaptation are known to have effects that vary systematically with feature similarity (as in this study), retinotopic distance and timing, which are thought to depend on the early visual cortex67. These same dimensions can now be interrogated in the context of top-down effects like VWM to elucidate their precision and strength relative to bottom-up processing, potentially yielding insights into how top-down signals interact with local visual representation.

In conclusion, the current work uncovered a bidirectional interaction between the VWM and ongoing perception that likely originates in their shared recruitment of early visual areas. With the demonstration that what we are holding in mind can change what we see, it is not hard to imagine how our current mental state and past experience may shape our subjective perception at multiple levels, from early (for example, colour/orientation) to complex processing (for example, scenes, facial recognition and semantic processing). These results are a critical step in advancing our understanding of the interplay and distinctions between bottom-up and top-down processing.

Methods

Participants

Participants were recruited remotely from Amazon Mechanical Turk with compensation of US$7 per hour. Participants were given informed consent and reported normal colour vision and normal or corrected-to-normal visual acuity. All experimental procedures were approved by the institutional review board of The George Washington University. The locations of the Turk workers were limited to the United States and their lifetime approval rates on the Mechanical Turk site were restricted to higher than 95%.

For experiment 1a, 50 participants were recruited with an average age of 38.7 ± 11.3, including 26 females and 22 males (2 participants refused to disclose the demographic information). Experiment 1b included 50 participants with an average age of 37 ± 11.6, 25 females and 24 males (1 participant refused to disclose the demographic information). Experiment 2 had 50 participants for 2a (age of 34.1 ± 9.7; 28 females and 22 males) and a separate set of 50 participants for 2b (age of 32.6 ± 14.7; 26 females and 23 males; 1 participant refused to disclose the demographic information). In Experiment 3a, two separate sets of 30 participants were recruited for the match and non-match conditions. The match condition had an average age of 38.4 ± 20.7 and included 17 females; the non-match condition had an average age of 39.5 ± 10.1 and included 14 females. Experiment 3b also had two separate sets of 30 participants for the two conditions. The match condition had an average age of 39 ± 12.4 and included 15 females and 14 males (1 participant refused to disclose the demographic information); the non-match condition had an average age of 32.6 ± 7.1 and included 13 females and 15 males (2 participants refused to disclose the demographic information). There was no overlap in participants across experiments. Data collection and analysis were not performed blind to the conditions of the experiments.

For experiments 1 and 2, the exclusion criterion was defined as lower than 60% accurate in the search task or the RSVP task, which was an orthogonal measurement to the critical measurement of interest and served as an indication of accurate execution of the task. For experiment 1a, 8 participants were below 60% search accuracy and were excluded for any future analysis; 9 participants were excluded from experiment 1b due to low search accuracy. In experiment 2a, 6 participants were excluded for low accuracy in the RSVP task; 8 participants were excluded from experiment 2b for low RSVP accuracy. For experiment 3, participants whose data in the discrimination task failed to fit the Weibull function were excluded; 14 participants were excluded from experiment 3a and 16 participants were excluded from experiment 3b.

In general, the sample size was predetermined based on the variability and effect sizes in pilot studies of these three experiments. After applying a slightly conservative inflation, a subject size of 50 for experiments 1 and 2, and 36 for experiment 3 was derived. Once the sample size was determined, we collected the full set of data and only checked the results after the completion of data collection. In some cases, participants with poor performance (see exclusion criterion above) were removed but were immediately replaced to get to the predetermined participant count. We did not include any of the pilot data in the final reports and analyses.

For the replication of the core findings in experiment 3, an a priori power analysis for sample size estimation was conducted based on the effect sizes reported in similar studies in the literature54,68. With the goal of achieving a power of 0.95 and an alpha level of 0.05 with an expectation of a medium–large effect size, the projected minimum sample size needed is 40 participants for each condition. Therefore, to be certain of detecting at least medium-sized effects, we collected a total of 96 participants’ data (48 for the match condition and another 48 for the non-match condition). The match condition had an average age of 38.9 ± 11.3 and included 26 females and 21 males (1 participant refused to disclose the demographic information); the non-match condition had an average age of 36.6 ± 10.3 and included 24 females and 24 males.

Stimuli and procedure

All the experiments were presented with PsiTurk69, a package to interface with the Mechanical Turk website. To address potential concerns with reaction time (RT) measurement with online experiments, we recorded the RTs locally on the participants’ computer and collected them at the end of the experiment to avoid lag in timing in Internet connection and therefore the RTs should have comparable precision to in-lab experiments. In the literature, various studies have implemented RT measurements70,71 and WM-based tasks70,72,73,74,75 on Mechanical Turk and obtained high-quality and stable data comparable to in-lab experiments.

In experiment 1a (Fig. 1a), the memory cue was a colour disk (radius of 50 pixels) located at the centre. Its colour was drawn from a set of 40 colours evenly sampled (9° steps through the full 360°) from the CIE L*a*b* colour space. The centre of the space was set as L* of 70, a* of 20 and b* of 38, with a radius of 60. The search display consisted of 5 white circles (radius of 50 pixels, line thickness of 10 pixels) and 1 colour disk (radius of 50 pixels) arrayed around an invisible isoeccentric circle with a radius of 115 pixels. For the memory report, a colour wheel (radius of 405 pixes) appeared at the centre of the display corresponding to the CIE L*a*b* space where the memory cues were chosen from. The colour wheel was rotated randomly on each trial to avoid the use of spatial memory. When participants clicked on the colour wheel, a colour disk would appear at the centre with the same colour being clicked on the wheel (radius of 50 pixels). Each participant completed 20 practice trials and four blocks of 100 experimental trials. There were 40 memory cue colours, five memory–distractor similarity conditions and two rotation directions (clockwise and anticlockwise), which resulted in 400 trials. Since the two rotation directions did not apply to the match condition, the number of trials for the match condition was doubled to 80 trials to make sure it had the same power as the other conditions.

The stimuli and procedure of experiment 1b were similar to those in experiment 1a. The oriented memory cue was 100 pixels in length and 10 pixels in width with varying orientations. The 20 possible orientations of the memory cue were evenly sampled from the 180° orientation space with a 9° interval. In the search display, 6 circles (50 pixels in radius) were evenly spaced around an imaginary circle with a radius of 215 pixels. The target was the circle in red (R:255, G:88, B:109) or blue (R:117, G:188, B:94) colour. The other five task-irrelevant circles were coloured in either orange (R: 238, G: 153, B: 0) or purple (R: 184, G: 162, B: 211) with a line width of 10 pixels. Each participant completed 20 practice trials and four blocks of 100 experimental trials.

In experiment 2, the memory cue was a coloured disk (80 pixels in radius) with its colour selected from the same set of 40 colours as experiment 1a, or an oriented Gabor patch (48 pixels with 10 cycles in black and white colour). In the RSVP display, a stream of 8 letters were presented continuously on top of a grey disk (20 pixels in radius) at the centre of the display. The letters were 40 pixels in height and were randomly selected from the 26 letters without repetition on each trial. The grey disk was included to maintain a constant contrast between the letter and background regardless of condition. The target letter in the stream was defined by colour, that is, the target letter was coloured white whereas the non-target letters were coloured black. There could be one or two white target letter(s) on each trial. The target letter(s) could never be the first or the last one in the stream. On the trials with two targets, the two letters were never presented back to back. The interfering background of the RSVP stream was a colour disk or oriented Gabor with the same size as the memory disk/Gabor. The similarity of the background disk to the memory cue was manipulated systematically as in experiment 1. The frequency of the RSVP stream was 5 letters per second so that each letter was presented for 200 ms and the whole stream lasted 1,600 ms.

In experiment 3, the coloured Gabor patches for the memory cue and discrimination stimuli were 125 pixels with 5 cycles. The colour of the memory cue was selected from 18 colours at a 20° interval in the colour space. The orientation of the cue was selected from 18 orientations with a 5° interval from the range of 25° to 65° and the range of 115° to 155°, to make sure that the cue and discrimination stimuli were in the same quadrant. For the discrimination stimuli, the distance between the two discrimination colours were systematically manipulated colour (0°, 4°, 7°, 10°, 13°, 16°, 19°) or orientation space (0°, 2°, 4°, 6°, 8°, 10°, 12°) to obtain the psychometric function of colour or orientation discrimination. In the middle condition, the memory colour/orientation was always in the middle of the two discrimination stimuli, with a varying gap between the two discrimination Gabors; in the side condition, the distance from the memory cue to the closer discrimination stimuli was always set to 7° in colour or orientation space, with a varying distance between the two discrimination Gabors.

It has been noted in the literature that VWM being tested on each trial might trigger an intentional strategy of paying attention to the memory-matching distractor76. Here matching trials only constituted 20% of the trials for experiments 1 and 2 and 7% for experiment 3 and therefore paying attention to the distractor would not be generally beneficial to the memory performance. Further, most of the key analyses were focused on the trials when distractors did not fully match the memory probe, such as the memory bias and the discrimination threshold analyses.

VWM report analysis

The analysis of the memory report was focused on the precision and the bias of the response distribution. For each similarity condition, we calculated the standard deviation of the response distribution and took the inverse of the standard deviation as a measure of precision. The precision indicates the amount of variation in the distribution. The less variation (the narrower the distribution curve), the more stable the response and the higher the precision. Another critical measure was the bias towards the distractor. We calculated how much the centre of the distribution was shifted away from the memory cue colour/orientation towards the distractor colour/orientation. It was defined as the difference between the mean of the actual report and the value of the memory cue. Normalized bias is calculated by taking the amount of bias divided by the memory–distractor distance (percentage of change in relation to the memory–distractor distance), \({\mathrm{Normalized}}\,\,{\mathrm{bias}} = \frac{{{\mathrm{Bias}}}}{{{\mathrm{Distance}}}} \times 100{\mathrm{\% }}\). The direction of the difference demonstrates whether the bias is towards (positive) or away (negative) from the distractor.

Discrimination threshold analysis

The discrimination threshold values in experiment 3 were estimated by fitting each individual’s data to the Weibull function through the Palamedes toolbox55 in MATLAB:

$$\psi = \gamma + \left( {1 - \gamma - \lambda } \right) \times \left( {1 - {\mathrm{e}}^{ - \left( {\frac{x}{\alpha }} \right)^\beta }} \right)$$

in which ψ represents the proportion of response reporting ‘different’ at colour/orientation difference x; γ is the lower asymptote of the psychometric function (guess rate) and was fixed to 0.1; λ is the lapse rate and was fixed to 0.02. α and β are the fit parameters, where α represents the threshold and β represents the slope of the psychometric function. For each individual’s data, threshold values of about 65% reporting ‘different’ were derived separately for the middle and side conditions.

Statistical analyses

The ANOVAs reported in experiments 1 and 2 were all one-way repeated measures ANOVAs. Two-tailed paired t-tests were conducted for further pairwise comparisons. For experiment 3, two-tailed paired t-tests were employed for the fitted threshold and the probability at each colour or orientation difference. To test the effect of task relevancy on changes in discrimination threshold, a mixed-design ANOVA with the task demand (match or non-match) as the between-subject factor and the relation between the memory and discrimination stimuli (middle or side) as the within-subject factor were conducted separately for each dimension (colour and orientation). To compare the original data with the replication data, we conducted a mixed-design ANOVA on the colour discrimination threshold with replication versus original as an additional between-subject factor and tested whether it interacted with task demand (match or non-match), the relationship (middle or side) or both. The same analysis was conducted on the memory report as well for the match (colour memory) and non-match (orientation memory) conditions. The CIs were reported for all effect sizes—90% CIs of \(\eta_{\mathrm{p}}^2\) were reported for the ANOVAs and 95% CIs of Cohen’s d were reported for the t-tests. For all analyses, the assumptions of normality and equality of variances were formally tested and confirmed77.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Data availability

The data that support the findings of this study are available in the Open Science Framework at https://osf.io/j7nv2/.

Code availability

The custom code developed for this study is available in the Open Science Framework at https://osf.io/j7nv2/.

References

  1. 1.

    Downing, P. E. & Dodds, C. M. Competition in visual working memory for control of search. Vis. Cogn. 11, 689–703 (2004).

    Article  Google Scholar 

  2. 2.

    Olivers, C. N. L., Meijer, F. & Theeuwes, J. Feature-based memory-driven attentional capture: visual working memory content affects visual attention. J. Exp. Psychol. Hum. Percept. Perform. 32, 1243–1265 (2006).

    Article  Google Scholar 

  3. 3.

    Soto, D., Hodsoll, J., Rotshtein, P. & Humphreys, G. W. Automatic guidance of attention from working memory. Trends Cogn. Sci. 12, 342–348 (2008).

    Article  Google Scholar 

  4. 4.

    Kiyonaga, A. & Egner, T. Center-surround inhibition in working memory. Curr. Biol. 26, 1–5 (2016).

    Article  Google Scholar 

  5. 5.

    Baddeley, A. Working Memory, Thought, and Action. (OUP Oxford, 2007).

  6. 6.

    Harrison, S. A. & Tong, F. Decoding reveals the contents of visual working memory in early visual areas. Nature 458, 632–635 (2009).

    CAS  Article  Google Scholar 

  7. 7.

    Serences, J., Ester, E., Vogel, E. K. & Awh, E. Stimulus-specific delay activity in human primary visual cortex. Psychol. Sci. 20, 207–214 (2009).

    Article  Google Scholar 

  8. 8.

    Emrich, S. M., Riggall, A. C., Larocque, J. J. & Postle, B. R. Distributed patterns of activity in sensory cortex feflect the memory. J. Neurosci. 33, 6516–6523 (2013).

    CAS  Article  Google Scholar 

  9. 9.

    Riggall, A. C. & Postle, B. R. The relationship between working memory storage and elevated activity as measured with functional magnetic resonance imaging. J. Neurosci. 32, 12990–12998 (2012).

    CAS  Article  Google Scholar 

  10. 10.

    Pasternak, T. & Greenlee, M. W. Working memory in primate sensory systems. Nat. Rev. Neurosci. 6, 97–107 (2005).

    CAS  Article  Google Scholar 

  11. 11.

    D’Esposito, M. & Postle, B. R. The cognitive neuroscience of working memory. Annu. Rev. Psychol. 66, 115–142 (2015).

    Article  Google Scholar 

  12. 12.

    Postle, B. R. Working memory as an emergent property of the mind and brain. Neuroscience 139, 23–38 (2006).

    CAS  Article  Google Scholar 

  13. 13.

    Postle, B. R. The cognitive neuroscience of visual short-term memory. Curr. Opin. Behav. Sci. 1, 40–46 (2015).

    Article  Google Scholar 

  14. 14.

    Christophel, T. B., Hebart, M. N. & Haynes, J.-D. Decoding the contents of visual short-term memory from human visual and parietal cortex. J. Neurosci. 32, 12983–12989 (2012).

    CAS  Article  Google Scholar 

  15. 15.

    Lee, S. H., Kravitz, D. J. & Baker, C. I. Goal-dependent dissociation of visual and prefrontal cortices during working memory. Nat. Neurosci. 16, 997–999 (2013).

    CAS  Article  Google Scholar 

  16. 16.

    Awh, E., Vogel, E. K. & Oh, S. H. Interactions between attention and working memory. Neuroscience 139, 201–208 (2006).

    CAS  Article  Google Scholar 

  17. 17.

    Chun, M. M., Golomb, J. D. & Turk-Browne, N. B. A taxonomy of external and internal attention. Annu. Rev. Psychol. 62, 73–101 (2010).

    Article  Google Scholar 

  18. 18.

    Lavie, N. & De Fockert, J. The role of working memory in attentional capture. Psychon. Bull. Rev. 12, 669–674 (2005).

    Article  Google Scholar 

  19. 19.

    Cowan, N. An embedded-processes model of working memory. in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control (ed. Miyake, A. & Shah, P.) 62–102 (Cambridge University Press, 1999).

  20. 20.

    Corbetta, M. & Shulman, G. L. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215 (2002).

    CAS  Article  Google Scholar 

  21. 21.

    Lepsien, J. & Nobre, A. C. Cognitive control of attention in the human brain: insights from orienting attention to mental representations. Brain. Res. 1105, 20–31 (2006).

    CAS  Article  Google Scholar 

  22. 22.

    Courtney, S. M., Ungerleider, L. G., Keil, K. & Haxby, J. V. Transient and sustained activity in a distributed neural system for human working memory. Nature 386, 608–611 (1997).

    CAS  Article  Google Scholar 

  23. 23.

    Fuster, J. The prefrontal cortex—an update: time is of the essence. Neuron 30, 319–333 (2001).

    CAS  Article  Google Scholar 

  24. 24.

    Goldman-Rakic Cellular basis of working memory review. Neuron 14, 477–485 (1995).

    CAS  Article  Google Scholar 

  25. 25.

    Miller, E. K., Erickson, C. & Desimone, R. Neural mechanisms of visual working memory in prefrontal cortex of the macaque. J. Neurosci. 16, 5154–5167 (1996).

    CAS  Article  Google Scholar 

  26. 26.

    Pessoa, L. & Ungerleider, L. G. Top-Down Mechanisms for Working Memory and Attentional Processes. in The Cognitive Neurosciences (ed. Gazzaniga, M. S.) 919–930 (MIT Press, 2004).

  27. 27.

    Oliveri, M. et al. Parieto-frontal interactions in visual-object and visual-spatial working memory: evidence from transcranial magnetic stimulation. Cereb. Cortex 11, 606–618 (2001).

    CAS  Article  Google Scholar 

  28. 28.

    Rademaker, R. L., Van De Ven, V. G., Tong, F. & Sack, A. T. The impact of early visual cortex transcranial magnetic stimulation on visual working memory precision and guess rate. PLoS One 12, e0175230 (2017).

    Article  Google Scholar 

  29. 29.

    Rose, N. S. et al. Reactivation of latent working memories with transcranial magnetic stimulation. Science 354, 1136–1139 (2016).

    CAS  Article  Google Scholar 

  30. 30.

    Zokaei, N., Manohar, S., Husain, M. & Feredoes, E. Causal dvidence for a privileged working memory state in early visual cortex. J. Neurosci. 34, 158–162 (2014).

    CAS  Article  Google Scholar 

  31. 31.

    Xu, Y. Sensory cortex is nonessential in working memory storage. Trends Cogn. Sci. 22, 192–193 (2018).

    Article  Google Scholar 

  32. 32.

    Xu, Y. Reevaluating the sensory account of visual working memory storage. Trends Cogn. Sci. 21, 794–815 (2017).

    Article  Google Scholar 

  33. 33.

    Bettencourt, K. C. & Xu, Y. Decoding the content of visual short-term memory under distraction in occipital and parietal areas. Nat. Neurosci. 19, 150–157 (2016).

    CAS  Article  Google Scholar 

  34. 34.

    Awh, E. & Jonides, J. Overlapping mechansims of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126 (2001).

    CAS  Article  Google Scholar 

  35. 35.

    Magnussen, S. & Greenlee, M. W. Retention and disruption of motion information in visual short-term memory. J. Exp. Psychol. Learn. Mem. Cogn. 18, 151–156 (1992).

    CAS  Article  Google Scholar 

  36. 36.

    Wildegger, T., Myers, N. E., Humphreys, G. & Nobre, A. C. Supraliminal but not subliminal distracters bias working memory recall. J. Exp. Psychol. Hum. Percept. Perform. 41, 826–839 (2015).

    Article  Google Scholar 

  37. 37.

    Nemes, V. A., Parry, N. R. A., Whitaker, D. & McKeefry, D. J. The retention and disruption of color information in human short-term visual memory. J. Vis. 12, 26 (2012).

    Article  Google Scholar 

  38. 38.

    Rademaker, R. L., Bloem, I. M., De Weerd, P. & Sack, A. T. The impact of interference on short-term memory for visual orientation. J. Exp. Psychol. Hum. Percept. Perform. 41, 1650–1665 (2015).

    Article  Google Scholar 

  39. 39.

    Lorenc, E. S., Sreenivasan, K. K., Nee, D. E., Vandenbroucke, A. R. E. & D’Esposito, M. Flexible coding of visual working memory representations during distraction. J. Neurosci. 38, 5267–5276 (2018).

    CAS  Article  Google Scholar 

  40. 40.

    Mansfield, R. J. W. & Ronner, S. F. Orientation anisotropy in monkey visual cortex. Brain Res. 149, 229–234 (1978).

    CAS  Article  Google Scholar 

  41. 41.

    Ringach, D. L., Shapley, R. M. & Hawken, M. J. Orientation selectivity in macaque V1: diversity and laminar dependence. J. Neurosci. 22, 5639–5651 (2002).

    CAS  Article  Google Scholar 

  42. 42.

    Shapley, R., Hawken, M. & Ringach, D. L. Dynamics of orientation selectivity in the primary visual cortex and the importance of cortical inhibition. Neuron 38, 689–699 (2003).

    CAS  Article  Google Scholar 

  43. 43.

    Bohon, K. S., Hermann, K. L., Hansen, T. & Conway, B. R. Representation of perceptual color space in macaque posterior inferior temporal cortex (the V4 complex). eNeuro 3, ENEURO.0039–16.2016 (2016).

    Article  Google Scholar 

  44. 44.

    Geng, J. J., Di Quattro, N. E. & Helm, J. Distractor probability changes the shape of the attentional template. J. Exp. Psychol. Hum. Percept. Perform. 43, 1993–2007 (2017).

    Article  Google Scholar 

  45. 45.

    Zhang, W. & Luck, S. J. Discrete fixed-resolution representations in visual working memory. Nature 453, 233–235 (2008).

    CAS  Article  Google Scholar 

  46. 46.

    Bays, P. M. & Husain, M. Dynamic shifts of limited working memory resources in human vision. Science 321, 851–854 (2008).

    CAS  Article  Google Scholar 

  47. 47.

    Joseph, J. S., Chun, M. M. & Nakayama, K. Attentional requirements in a ‘preattentive’ feature search task. Nature 387, 805–807 (1997).

    CAS  Article  Google Scholar 

  48. 48.

    Horton, J. C. & Hoyt, W. F. The representation of the visual field in human striate cortex. Arch. Opthalmology 109, 816–824 (1991).

    CAS  Article  Google Scholar 

  49. 49.

    Engel, S. A., Glover, G. H. & Wandell, B. A. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex 7, 181–192 (1997).

    CAS  Article  Google Scholar 

  50. 50.

    McAdams, C. J. & Maunsell, J. H. Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4. J. Neurosci. 19, 431–441 (1999).

    CAS  Article  Google Scholar 

  51. 51.

    Müller, N. G. & Kleinschmidt, A. Dynamic interaction of object- and space-based attention in retinotopic visual areas. J. Neurosci. 23, 9812–9816 (2003).

    Article  Google Scholar 

  52. 52.

    Carrasco, M., Ling, S. & Read, S. Attention alters appearance. Nat. Neurosci. 7, 308–313 (2004).

    CAS  Article  Google Scholar 

  53. 53.

    Fuller, S. & Carrasco, M. Exogenous attention and color perception: performance and appearance of saturation and hue. Vis. Res. 46, 4032–4047 (2006).

    Article  Google Scholar 

  54. 54.

    Liu, T., Fuller, S. & Carrasco, M. Attention alters the appearance of motion coherence. Psychon. Bull. Rev. 13, 1091–1096 (2006).

    Article  Google Scholar 

  55. 55.

    Prins, N. & Kingdom, F. A. A. Applying the model-comparison approach to test specific research hypotheses in psychophysical research using the Palamedes toolbox. Front. Psychol. 9, 1250 (2018).

    Article  Google Scholar 

  56. 56.

    Scimeca, J. M., Kiyonaga, A. & D’Esposito, M. Reaffirming the sensory recruitment account of working memory. Trends Cogn. Sci. 22, 190–192 (2018).

    Article  Google Scholar 

  57. 57.

    Albers, A. M. et al. Supplemental information shared representations for working memory and mental imagery in early visual cortex. Curr. Biol. 23, 1427–1431 (2013).

    CAS  Article  Google Scholar 

  58. 58.

    Farah, M. J. Psychophysical evidence for a shared representational medium for mental images and percepts. J. Exp. Psychol. Gen. 114, 91–103 (1985).

    CAS  Article  Google Scholar 

  59. 59.

    Christophel, T. B., Klink, P. C., Spitzer, B., Roelfsema, P. R. & Haynes, J. D. The distributed nature of working memory. Trends Cogn. Sci. 21, 111–124 (2017).

    Article  Google Scholar 

  60. 60.

    Sreenivasan, K. K., Curtis, C. E. & D’Esposito, M. Revisiting the role of persistent neural activity during working memory. Trends Cogn. Sci. 18, 82–89 (2014).

    Article  Google Scholar 

  61. 61.

    Miller, E. K. & Cohen, J. D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202 (2001).

    CAS  Article  Google Scholar 

  62. 62.

    Woolgar, A., Hampshire, A., Thompson, R. & Duncan, J. Adaptive coding of task-relevant information in human frontoparietal cortex. J. Neurosci. 31, 14592–14599 (2011).

    CAS  Article  Google Scholar 

  63. 63.

    Mongillo, G., Barak, O. & Tsodyks, M. Synaptic theory of working memory. Science 319, 1543–1546 (2008).

    CAS  Article  Google Scholar 

  64. 64.

    Gazzaley, A. & Nobre, A. C. Top-down modulation: bridging selective attention and working memory. Trends Cogn. Sci. 16, 129–135 (2012).

    Article  Google Scholar 

  65. 65.

    Dowd, E. W. & Mitroff, S. R. Attentional guidance by working memory overrides salience cues in visual search. J. Exp. Psychol. Hum. Percept. Perform. 39, 1786–1796 (2013).

    Article  Google Scholar 

  66. 66.

    Kiyonaga, A. & Egner, T. Working memory as internal attention: toward an integrative account of internal and external selection processes. Psychon. Bull. Rev. 20, 228–242 (2013).

    Article  Google Scholar 

  67. 67.

    Kohn, A. Visual adaptation: physiology, mechanisms, and functional benefits. J. Neurophysiol. 10461, 3155–3164 (2007).

    Article  Google Scholar 

  68. 68.

    Montagna, B., Pestilli, F. & Carrasco, M. Attention trades off spatial acuity. Vis. Res. 49, 735–745 (2009).

    Article  Google Scholar 

  69. 69.

    Gureckis, T. M. et al. psiTurk: an open-source framework for conducting replicable behavioral experiments online. Behav. Res. Methods 48, 829–842 (2016).

    Article  Google Scholar 

  70. 70.

    Kiyonaga, A. & Egner, T. The working memory stroop effect: when internal representations clash with external stimuli. Psychol. Sci. 25, 1619–1629 (2014).

    Article  Google Scholar 

  71. 71.

    Crump, M. J. C., McDonnell, J. V. & Gureckis, T. M. Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS One 8, e57410 (2013).

    CAS  Article  Google Scholar 

  72. 72.

    Brady, T. F. & Alvarez, G. A. Hierarchical encoding in visual working memory: ensemble statistics bias memory for individual items. Psychol. Sci. 22, 384–392 (2011).

    Article  Google Scholar 

  73. 73.

    Brady, T. F. & Alvarez, G. A. No evidence for a fixed object limit in working memory: spatial ensemble representations inflate estimates of working memory capacity for complex objects. J. Exp. Psychol. Learn. Mem. Cogn. 41, 921–929 (2015).

    Article  Google Scholar 

  74. 74.

    Lew, T. F. & Vul, E. Ensemble clustering in visual working memory biases location memories and reduces the Weber noise of relative positions. J. Vis. 15, 10 (2015).

    Article  Google Scholar 

  75. 75.

    Shin, H. & Ma, W. J. Crowdsourced single-trial probes of visual working memory for irrelevant features. J. Vis. 16, 10 (2016).

    Article  Google Scholar 

  76. 76.

    Woodman, G. F. & Luck, S. J. Do the contents of visual working memory automatically influence attentional selection during visual search? J. Exp. Psychol. Hum. Percept. Perform. 33, 363 (2007).

    Article  Google Scholar 

  77. 77.

    Lakens, D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front. Psychol. 4, 863 (2013).

    Article  Google Scholar 

Download references

Acknowledgements

We thank the colleagues in the psychology department of George Washington University for their helpful discussion. Especially, we thank S. Shomstein, S. Mitroff and M. Behrmann for their insightful comments on this manuscript. This research project was supported by start-up funds granted to D.J.K. by GWU. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

Authors

Contributions

C.T. and D.J.K. designed the experiments. C.T. collected the data and performed analysis. C.T. and D.J.K. wrote the manuscript.

Corresponding author

Correspondence to Chunyue Teng.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information: Primary Handling Editor: Marike Schiffer.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–5 and Supplementary Tables 1–5.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Teng, C., Kravitz, D.J. Visual working memory directly alters perception. Nat Hum Behav 3, 827–836 (2019). https://doi.org/10.1038/s41562-019-0640-4

Download citation

Further reading