Separation or Integration of Sequential Arrays in VWM Depends On the interval Between Them

The visual information could be stored in either the active state or the passive state for a short time in visual working memory (VWM). Catering to the dynamic nature of visual world, we explored how the temporally dynamic input was stored in VWM by employing sequentially presented arrays rather than static arrays in which all items are presented simultaneously, and the contralateral delay activity (CDA), an electrophysiological measure was used to identify whether the memory items had been transferred into the passive state. In the current study, participants were instructed to encode two sequential arrays and retrieve them respectively, with two conditions of interval across the two arrays: 400ms and 800ms. These results provided strong evidence that two sequential arrays could be stored in VWM by forming two state-separated images if the interval between them was long enough, or by becoming an integration if the interval was relatively short. This conclusion was valid only when the participants encountered the task for the first time. Once participants have formed their mindset, they would apply the same memory mode to the subsequently extended or shortened interval condition. Therefore, the finding indicated that the extended interval could allow the storage transformation for the leading array, separated from the trailing array stored in the active state. Thus we proposed that the interval between two memory arrays influenced whether participants stored the two sequential arrays as parts of a single extended event or two independent episodes.


Introduction
The visual working memory (VWM) system is responsible for the transient retention and manipulation of visual information from the dynamic visual environment (Baddeley & Hitch, 1974;Olivers, Peters, Houtkamp & Roelfsema, 2011;Luck & Vogel, 2013). It helps to bridge across changes and create temporal continuity from multiple, successively occurring displays in the service of advanced cognition. A fundamental property of this system is its limited capacity. Recent studies considered that approximately four objects can be retained at once in VWM, and this information is encoded in the form of integrated visual features, rather than as a collection of disconnected visual features (Jiang, Olson, & Chun, 2000;Luck & Vogel, 1997;. These studies have employed static displays to present all representations simultaneously in VWM. However, visual events evolve over time in many cognitive activities. Concerning the dynamic nature of the visual environment, investigating VWM for sequential arrays could comprehensively explain the representations of the real world. Recently research efforts have focused on dynamic visual processing to make great advances in scientific knowledge about memory. For example, Jiang and Kumar (2004) explored whether the two sequentially presented arrays, separated by a variable interstimulus interval (ISI), were represented as an integrated image or two separated images in visual short-term memory (VSTM). They proposed that two sequential visual arrays were held separately at an ISI of 500 ms or longer (Jiang, Y. & Kumar, A., 2004). Subsequently they employed the same paradigm to further explore the dynamics of visual information, suggesting that VSTM has a constant capacity independent of the stimulus onset asynchrony (SOA) (Kumar, A. & Jiang, Y., 2005). Additionally, Gorgoraptis el at. (2011) further designed sequential and simultaneous presentation condition to examine the dynamics and flexibility of WM resources. Their study revealed that WM resources could be dynamically and flexibly updated as new items had to be stored, but redistribution of resources with the addition of new items was associated with misbinding object features (Gorgoraptis, N., Catalao, R., Bays, P., & Husain, M., 2011). But, for the most part, these studies all adopted an integration probe task in which the to-be-probed items were randomly chosen from previously memory arrays. The specific task potentially compelled participants to integrate all memory information in their mind, which corresponded well to the dynamically cognitive activities, such as chess game which requires memory updating gathered from many instances, retaining information across space and time. Nevertheless, dynamic visual processing also involved non-integrated retrieval. For example, a learner modeled a dance following a dancer's performance. Because the dance is a particular series of graceful movements, the learner must remember these dynamically appeared actions, and then performed them serially and accurately. Thus, the dynamics of visual information should endow not only the presentation of memory stimuli, but also the retrieval of memory content. Here, we made efforts to explore how the representations of two sequential arrays were stored in VWM when performing a sequential-encoding-sequential-retrieval task in which two memory arrays were encoded sequentially and then retrieved respectively.
The multi-state storage mechanism of VWM has been extensively recognized. One form of evidence comes from "activity-silent" synaptic mechanisms by which humans can hold information in WM (Rose et al. 2016). The "activity-silent" model proposed that the memory representations could be stored in either "active state" that was accompanied stationary activity and could be measured in typical experiments, or "passive (or silent) state" that could not be directly detected by recording techniques, but still maintained the relevant information (Stokes, 2015). In a memory-guided saccade task (MGS), it could be observed a "ramp-up" that indicated the robust activity emerged only at the late delay period for upcoming response cue, and a silent moment during which the robust delay activity was absent between memory stimuli and WM-behavioral response (Watanabe & Funahashi, 2007;Stokes, 2015). Similarly, when adding a dual task to the classic MGS task, WM-specific delay activity was abolished during the dual task period. But once the attention task was completed, the WM delay activity was "reawakened", presumably in time for WM-guided behavior (Watanabe & Funahashi 2014;Stokes, 2015). The two studies exactly supported the "activity-silent" model, showing that persistent neural activity was not critical for the maintenance of WM content but instead reflected task relevance of memoranda. Thereby, it could be known that whether the memory representations were maintained in the active state or the passive state was determined by the current task relevance.
According to "activity-silent" model, it could be reasoned that, in the sequential-encoding-sequential-retrieval task, the representations of currently task-irrelevant array would be held in the passive state, while the others should be held in the active state for the current encoding or the imminent probe.
Within the field of electroencephalography, researchers found a sustained contralateral negativity over posterior electrode sites during the retention period of VWM tasks through the event-related potential studies. It has been proven that the amplitude of contralateral delay activity (CDA) increases as the number of memory items increases, up to the individual's working memory capacity limit (Luria, Balaban, Awh & Vogel, 2016;Todd and Marois 2004). CDA is widely used as an effective marker to identify the number of items in the active state (Feldmann-Wustefeld, T., Vogel, E. K., & Awh, E. 2018). That is, the CDA amplitude would decrease quickly and disappeared when the memory items were transferred from the active state to the passive state. In addition, the CDA was not modulated by perceptual requirements and the number of currently attended locations (Ikkai, McCollough & Vogel, 2010;Luria, R., & Vogel, E. K. 2014;Balaban and Luria, 2015;Balaban and Luria, 2016). Hence, if the CDA was applied to the dynamic sequential-encoding-sequential-retrieval task, then we could explicitly reckon whether the first array was transferred into the passive state in the course of storage in VWM.
The sequential-encoding-sequential-retrieval task was rarely exploited to explore the storage of sequential arrays, but it has been adopted to determine what aspect of WM performance the CDA reflects (Ikkai et al., 2010). However, Ikkai's results suggested that the CDA amplitude during the encoding of the second array doubled compared with that following the first arrays, which implied that the two sequential arrays naturally became an integration following the second array. These results were obviously inconsistent with the prediction of the "activity-silent" model.
The brevity of the interval between two sequential arrays might be a plausible account of this contradiction. Because temporal context between two arrays was assumed a powerful factor in storage transformation for items stored in VWM. In the previous studies on the retro-cue effect in VWM, participants could maintain a relevant item in the active state by using a retro-cue, and transfer other items into the passive state. Moorselaar et al. (2015) studied the timing course of the effect of the retro-cue. They used a mask to interfere with the processing after the disappeared of the retro-cue, and manipulated the interval between the retro-cue and mask (Moorselaar et al., 2015). The appearance of the mask would prevent the transformation of VWM representations from the active state to the passive state. They found when the interval was less than 650ms, the appearance of the mask would significantly reduce the effect of retro-cue; when the interval was 650ms, the mask would not affect the retro-cue effect. These results suggested that it took about 650 ms for participants to transfer memory representations from the active state to the passive state. However, in Ikkai's study, the SOA of two memory arrays was only 500ms. So items of M1 in the active state did not have enough time to be transferred into the passive state in such short interval, and then they had to be continually represented in the active state together with the second memory array. Additionally, Jiang and Kumar (2004) conducted a paradigm in which two dot arrays, separated by an interstimulus interval (ISI) of 0, 200, 500, or 1,500ms, were sequentially presented. They found that the leading array was consolidated better and separated from the trailing array as the ISI increased to 500ms or longer (Jiang, Y. & Kumar, A. 2004). So we could deduce that extending the interval between the two sequential arrays would achieve the storage transformation for the currently task-irrelevant items to the passive state from activity in VWM, resulting in a separation of two dynamic arrays.
In the current study, we adopted the sequential-encoding-sequential-retrieval task to examine how the two sequentially presented arrays were stored (integration or separation) in VWM, and the CDA was employed to identify whether or not the memory representations were stored in the active state. According to the reasoning above, we predicted that if the interval between the two memory arrays was extended enough, the two arrays would be stored separately in different states; In contrast, if the interval still stayed short as Ikkai's experiment 2, then we would replicated the results that items of the first array became integrated with the second array to be stored during the encoding of the second array.

Method
There were two experimental conditions as depicted in Figure 1: short interval condition (400ms) and long interval condition (800ms) to test whether participants could potentially succeed in the two interval conditions by integrating the two arrays as an integration, or by forming two separated images. If the CDA amplitude following the second array have no increase compared to that following the first array in the long condition (800ms), we could confirm that the representations of two arrays were stored separately; If an integration of the CDA amplitude could be observed during the encoding of the second array when confronting relatively short interval period (400ms), we then deduced that the two arrays become an integration to be stored. The current experiment was a within-subjects design. All participants performed two conditions of 4 blocks each. The order of condition was balanced across subjects and each of them completed a total of 8 blocks of 64 trials each, resulting in 526 trials in total.

Participants
Twenty-two participants (18-25 years old) from Liaoning Normal University participated in the experiment. All participants had a normal or corrected-to-normal vision, and had no history of neurological disorders or color blindness. Participants were paid CNY 50 for their participation. Participants provided informed consent when they arrived at the lab. The experiment would last about 2.5 hours. Participants with eye-blink or eye-movement artifacts above 25% of trials were excluded from further analysis.
All participants were distinguished into two groups according to the order of experimental condition: those (11 participants) who performed short condition first and then long condition were called short-long group, and those (11 participants) who performed in converse order were called long-short group.

Stimuli and procedure
In the experiment, the stimuli were presented with e-prime software on a CRT screen in a light-bright room. Items were presented within 4 × 7.3°Rectangular regions bilaterally, centered 3°to the left and right of the middle of the screen. A black fixation cross was presented in the center of the screen throughout the trial against the gray background. An arrow was presented above the fixation point. Colored squares (0.65 × 0.65°) were randomly chosen without replacement within a trial from a set of seven colors (red, orange, blue, yellow, green, indigo and purple).
The schematic of a trial was illustrated in Figure1. Participants were instructed to fixate the black cross from 70 cm of viewing distance. Each trial consisted of an arrow cue, two memory arrays, retention period, and two test arrays. Participants attended to the cued visual field and remembered the colors of the memory array items. A fixation cross presented in the middle of the screen for 1500ms before the onset of the cue arrow, and then two memory arrays (M1 M2) were presented serially with a 400ms-interval in short condition and an 800ms-interval in long condition after a random retention (100-300ms). Participants were instructed to remember the items (two items in each memory array). After an 800ms-interval period, the first test array (T1) and the second test array (T2) appeared. At the onset of the test arrays, participants responded whether the test arrays (T1 T2) was respectively identical to the memory arrays (M1 M2) correspondingly by a button press (same vs. different). If they were identical, participants were instructed to press the "F" in the keyboard, or press 'J'. The test arrays did not disappear until the participants made a response. Through all trials, participants were informed to make a button press as accurately as possible. Item positions were randomized between the trials with a constraint that no square was present within 2°of one another. Before the experiment, all participants were asked to practice at least 16 trials and reached 75% accuracy to understand the experimental procedure.

Electrophysiology (EEG) recording and analyses
Sixty-four tin recording electrodes were mounted on an elastic cap to record EEG during the task. Electrode placements followed the International 10/20 system; Fp1, Fp2, Fpz, AF3, AF4, GND, AF7, AF8, F1, F2, F3, F4, F5, F6, F7, F8, Fz, FT7, FT8, FC1, FC2, FC3, FC4, FC5, FC6, FCz, T7, T8, C1, C2, C3, C4, C5, C6, Cz, TP7, TP8, CP1, CP2, CP3, CP4, CP5, CP6, CPz, P1, P2, P3, P4, P5, P6, P7, P8, Pz, PO3, PO4, PO7, PO8, POz, O1, O2 ,Oz and left-and right mastoids of these electrodes, CPz is the on-line reference, and the GND is the ground electrode. The off-line signals were analyzed with EEGLAB Toolbox and MATLAB, and the data were only filtered with a low-pass of 40 Hz, and re-referenced to the average of the left-and right mastoids. A horizontal electrooculogram (EOG) was recorded from electrodes placed next to each eye, and vertical EOG was recorded from an electrode placed below and above the left eye. EEG and EOG were amplified by SA Instrumentation amplifier, and data were collected at a sampling rate of 500 Hz. The recorded signals were segmented from 200ms before to 1400ms (in short interval condition) or 1800ms (in long interval condition) after the first memory array onset. The 200ms prior to the first memory array onset was used for performing the baseline correction. EOG was scanned for artifacts related to blinks and eye movements using an algorithm that detected large (80 μV) peak-to-peak deflections. All trials containing these artifacts were excluded from further analysis. Participants with trial rejection rates that exceeded 25% were excluded from the analyses.
We selected the P7\P8 to calculate the CDA amplitude. ERPs were time-locked at the onset of the memory array (onset of the first memory array and the second memory array) and recorded throughout the retention period. CDA mean amplitude was analyzed using the mean amplitude of difference wave (contralateral -ipsilateral) using a time window of between 300 and 500ms from the onset of the 1st memory array and the 2nd memory array.

Behavioral results
A 2(condition: short condition vs. long condition) × 2 (test 1 vs. test 2) two-way repeated measures ANOVA was analyzed on the performance accuracy. As shown in Figure 2. The main effect of condition was significant (F (1, 21) =16.088, p=0.01). The main effect of test was also significant (F (1, 21) =55.450, p<0.01). There was no significant interaction between condition and test (F (1, 21) =1.402, p=0.250). We further analyzed the change detection accuracy by paired t-test. Accuracy was significantly better for long condition (test1:91%; test 2:80%) than short condition (test1: 88%; test 2: 75%), as well as better for the first test arrays than second test arrays in both conditions. p<0.05.
Considering that two distinct groups have been divided according to the order of experimental condition, we examined the behavioral performance across the short-long group (complete the short conditions before the long conditions, N=11) and the long-short group (complete the short conditions after the long conditions, N=11). Both groups had similar performance to all participants. For the short-long group, a 2 (short condition vs. long condition) × 2 (test 1 vs. test 2) two-way repeated measures ANOVA showed that the main effect of condition was significant (F (1, 10) =8.361, p=0.016). The main effect of test was also significant (F (1, 10) =27.282, p<0.01). There was no significant interaction between condition and test (F (1, 10) =0.106, p=0.751). Additionally analyzing the behavioral accuracy by paired t-test. Accuracy was significantly better for long condition (test1:91%; test2:81%) than short condition (test1:88%; test 2: 77% ), as well as better for the first test arrays than second test arrays in both conditions, p<0.05. while for the long-short group, a 2 (short condition\long condition) × 2 (test 1\ test2) two-way repeated measures ANOVA also showed that the main effect of condition was significant (F(1,10)=7.119, p=0.024). The main effect of test was also significant (F (1, 10) =30.125, p<0.01). There was no significant interaction between condition and test (F (1, 10) =2.524, p=0.143). these data were further analyzed by paired t-test, and the statistics result showed that accuracy was significantly better for long condition (test1:91%; test 2:78%) than short condition (test1:88 %; test 2: 73% ), as well as better for the first test arrays than second test arrays in both conditions, p<0.05.

Electrophysiological results
CDA amplitude of short and long conditions were plotted across all participants as depicted in Figure 3A which showed mean difference waves (contralateral minus ipsilateral) for each condition. Beginning around 250ms, strong negative contralateral wave arose over posterior electrode sites and continued throughout the retention period. CDA amplitude of all participants in both conditions was obviously larger during the learning of the second memory array than that following the first array. We compared the rise in both the short interval condition and the long interval condition by measuring the amplitude during two time windows, early (300-500ms following memory array 1) and late (300-500ms following memory array 2). We found a highly significant main effect of time window (the early is greater than the late; short interval condition: p=0.025, long interval condition: p=0.006). Moreover, the results of repeated measures ANOVA showed that there was no significant main effect of condition, but the main effect of time window was significant (p=0.007), and no interaction between these factors. That is, even in long condition, the CDA amplitude during the second delay was highly larger than the first delay period, which was identical to the short condition.
Notably, priming could greatly affect the interpretation of incoming input in VWM. The results of Balaban's experiment 4 suggested that the individuation context that was primed before the main experimental phase could influence the interpretation that first appeared during the main experimental phase . So it was necessary to take priming of condition into consideration. Given that there were two conditions with a within-subjects design, and the order of condition was balanced across them, the difference of CDA wave might be observed in different conditions across the short-long group and the long-short group.
Across the short-long group, analyzing the differences of CDA amplitude was conducted by a 2(condition: short vs. long) x 2 (delay period: early vs. late) repeated measures ANOVA. A significant main effect of delay period was observed, F (1, 10) =36.672, p < .000, ηp ² = .786, but the main effect of condition was not significant. The interaction between condition and delay period was significant. F (1, 10) =24.338, p = .001, ηp ² = .709. In short condition, the mean amplitude in early delay and late delay period were -1.62 and -2.06; SD were 0.632 and 0.688, respectively, t(10)=2.720, p= .022. While in long condition, mean of amplitude in early delay and late delay period were -1.50 and -2.57; SD were 1.092 and 0.904, respectively, t(10)=9.297,p< .000. These results demonstrated that items from the two memory arrays were likely to be integrated more absolutely as the participants completed more trials. In contrast, across the long-short group, differences of CDA amplitude were also analyzed in a 2(condition: short vs. long) x 2 (delay period: early vs. late) repeated measures ANOVA. The main effect of the condition (F (1, 10) = .700, p = .422, ηp ² = .065) and delay period (F (1, 10) = .871, p = .373, ηp ² = .080) was not significant, nor was there a significant interaction between condition and delay period (F (1, 10) = .143, p = .713, ηp ² = .014). In long condition, mean of amplitude in early delay and late delay period were -2.228 and -2.496; SD were 0.785 and 1.711, respectively. While in short condition, mean of amplitude in early delay and late delay period were -1.913 and -2.267; SD were 0.788 and 1.325, respectively, suggesting that participants of long-short group stored the items from the two sequential arrays separately. By analyzing the two distinct groups, we found that the memory mode adopted in the latter condition was always the same as that in the initial condition for each group. Therefore, it could be known that priming indeed existed.
The current study aimed to test whether the two sequentially presented arrays were stored separately in the long interval condition. To abolish the priming effect, we only explored the short condition across the short-long group and the long condition across the long-short group. The results of short condition replicated the findings of Ikkai's study: CDA amplitude following the M2 was significantly larger than that following M1, suggesting that items of M1 integrated with items of M2 to be maintained in the active state together. Whereas in the long condition, there was no significant difference in CDA amplitude between two delay periods, which indicated that items of M1 were transferred into the passive state before the presence of M2, and thus the two memory arrays were stored separately. In other words, when removing the priming, the results of the current study were consistent with our hypothesis that the two sequential arrays would be stored separately when the interval between two sequential arrays was long enough.
In addition, considering the different results when processing the EEG data in a distinct way, the results indeed showed a prime effect of memory mode as shown in Figure 3B and 3C. For the long-short group, the CDA amplitude after two memory arrays were not significantly different in neither long condition nor short condition; while for short-long group, the CDA amplitude of the late delay was significantly higher than that of the early delay period in both the long condition (p<0.01) and the short condition (p=0.022). Specifically, the memory mode that could be indeed primed before another interval condition trials began could influence the subsequent mode, no matter what the next interval condition was short or long. Participants of the short-long group still employed an integrated manner to perform the task in the long condition; similarly, participants of the long-short group even applied a separated manner to the short condition. To sum up, whether two sequential arrays were stored as an integrated image or as two separated displays also depended on their mindset which was formed in the initial condition and played a dominate role in subsequent condition regardless of the interval was long or short.

Discussion
The primary goal of this study was to explore how VWM stored the memory information over a more dynamic, sequential input and retrieval. To accomplish this, we adopted a sequential-encoding-sequential-retrieval task in which we modulated the interval (400ms vs. 800ms) between two memory arrays. And the CDA was employed to identify whether the items were maintained in the active state. The results showed that the CDA amplitude during the encoding of the second array was significantly higher than that following the first array in both the short and the long interval condition across all participants, indicating that the two arrays were stored as an integration in the active state independent of the interval. Given that the present study was within-subjects design, and the order of experimental condition was balanced across subjects, all participants were divided into two groups: short-long group and long-short group. It was found that the CDA amplitude during the encoding of the second array was significantly larger than that following the first array in both short and long conditions across the short-long group, suggesting that the two memory arrays were maintained as one integration in the active state in the course of storage; whereas there was no significant difference of CDA amplitude between the early and late retention period in both conditions across long-short group, suggesting that the two arrays were stored separately because of the storage transformation of the first array. The results from the analysis of short-long group in the short condition and the long-short group in the long condition revealed that the two sequential arrays were stored as one integration when the interval was relatively short; but the two were maintained separately when the interval was extended enough, such that was valid only when participants initially encountered the task. The behavioral results suggested that the two memory modes would generate comparable VWM performance. Therefore, this study provided strong evidence that the temporal context between two sequential arrays made a great contribution to memory modes (integration and separation). The separation of two sequential arrays stored in VWM may occur if the interval (e.g. not less than 800ms) across two memory arrays was sufficient to transfer the first array into the passive state before the onset of the second array. Whereas if the shortage of interval (e.g. no more than 400ms) prevented the storage transformation of the first array, the two temporally segregated arrays would be stored in VWM as an integration which still contained the independent component of each array.
Moreover, the current results have revealed the role of mindset in the memory mode through the data of short-long group and long-short group in both conditions, manifested by priming effect. The findings demonstrated that the short-long group integrated two arrays to maintain even in the long interval condition, while the long-short group stored two arrays in a separated manner even in the short interval condition. Such results allowed us to conclude that the memory mode at the early stage of the experiment would be fixed and stay the same even in subsequently different condition. That provided a plausible account of why the results of all participants were not consistent with our hypothesis. What's more, it could be known that it was a top-down voluntary process rather than a stimulus-driven process for participants to decide whether to transfer VWM representations into the passive state. When the transformation could optimize cognitive resource allocation, participants would transfer the VWM representations. However, if the shortage of time restricted the storage transformation, then the favored strategy was to keep the VWM representations in the active state until making responses. This kind of transformation strategy might become a mindset in similar processing. Once the participants formed their mindsets which does not need to transfer the VWM representations to the passive state, the transformation of VWM representations would not occur anymore. Therefore, the mindset of representation transformation could also affect whether the two sequential arrays were stored as an integration or separation.
Recently the sequential presentation paradigm was widely adopted to explore the nature of contralateral delay activity (CDA). Researchers found that CDA tracked the current focus of spatial attention, exhibiting the absence of the CDA integration if the probe display and the initial sequential displays were spatially incompatible (Berggren, N., & Eimer, M., 2016). However, CDA amplitude instead indexed the total memory load across both displays if the format of the probe display was spatially compatible with the initial displays, indicated by an increase of CDA amplitude following the second array (Feldmann-Wüstefeld, T., Vogel, E., & Awh, E., 2018). It could be interpreted as that, in a sequential-encoding-simultaneous-retrieval task, spatial compatibility biased the participants to store two sequential arrays as one integration, but the spatial incompatibility made the first arrays transferred into the passive state, resulting in the separation of two sequential arrays even in a 500ms-interval condition. These studies further suggested that the storage transformation of WM representations was voluntarily modulated by cognitive resources.
VWM plays an important role in linking the preceding and following scenes. Beyond much understanding of VWM from the static displays, we attempt to explore the storage manner of VWM representation by revealing robust temporal dynamics over a very rapid timescale. Individuals need constant updating of information gathered from multiple successive occasions, so it could be energetically expensive to hold memory information in active state all the time, especially if WM is in near-constant use constructing and maintaining an up-to-date world model of the content of the environment (Stokes, M. G. 2015). Therefore, the neural economy may promote the separation of the two sequential arrays in VWM when encountering a relatively long interval. To sum up, whether the separation or integration of two sequentially presented arrays in VWM is dependent on the interval between them. But the temporal interval has no effect once individuals' mindset of transformation has formed.
The different maintenance requirements could possibly impact the response profile, reflecting the dynamic storage of representations in VWM. Thus, it remains to explore how the two dynamic arrays were stored in VWM when performing a dynamic recall report task. In addition, given that a fundamental property of the VWM system is limited-capacity, it is still unclear whether the capacity could expand by transferring the currently task-irrelevant stimuli into the passive state. Further studies need to explore these questions to provide new insights of the VWM system.