Robust spatial ventriloquism effect and aftereffect under memory interference

Our brain adapts to discrepancies in the sensory inputs. One example is provided by the ventriloquism effect, experienced when the sight and sound of an object are displaced. Here the discrepant multisensory stimuli not only result in a biased localization of the sound, but also recalibrate the perception of subsequent unisensory acoustic information in the so-called ventriloquism aftereffect. This aftereffect has been linked to memory-related processes based on its parallels to general sequential effects in perceptual decision making experiments and insights obtained in neuroimaging studies. For example, we have recently implied memory-related medial parietal regions in the trial-by-trial ventriloquism aftereffect. Here, we tested the hypothesis that the ventriloquism aftereffect is indeed susceptible to manipulations interfering with working memory. Across three experiments we systematically manipulated the temporal delays between stimuli and response for either the ventriloquism or the aftereffect trials, or added a sensory-motor masking trial in between. Our data reveal no significant impact of either of these manipulations on the aftereffect, suggesting that the recalibration reflected by the ventriloquism aftereffect is surprisingly resilient to manipulations interfering with memory-related processes.

hypothesis that the ventriloquism aftereffect is indeed susceptible to manipulations interfering 23 with working memory. Across three experiments we systematically manipulated the temporal 24 delays between stimuli and response for either the ventriloquism or the aftereffect trials, or 25 added a sensory-motor masking trial in between. Our data reveal no significant impact of either 26 of these manipulations on the aftereffect, suggesting that the recalibration reflected by the 27 ventriloquism aftereffect is surprisingly resilient to manipulations interfering with memory-28 related processes. 29

INTRODUCTION 31
Sensory recalibration is a mechanism by which the brain continuously adapts to apparent 32 discrepancies in our sensory environment, such as the displaced figure and voice of an actor 33 in a movie watched over headphones 1,2 . One example of adaptive recalibration is the 34 ventriloquism aftereffect, a frequently studied paradigm for multisensory perception in the 35 laboratory. To reveal this aftereffect, participants are first (in an audio-visual trial) presented 36 with spatially discrepant audio-visual stimuli, which give rise to a biased localization of the 37 sound. This bias reflects the partial fusion of the discrepant audio-visual informationthe so 38 called ventriloquism effect 3 . In a subsequent trial, participants are then asked to localize a 39 unisensory sound, which they often misjudge in the direction established by the previous 40 multisensory discrepancy [4][5][6][7] . For example, when the light is to the left of the sound in the audio-41 visual trial, the subsequent sound is misjudged to the left. This aftereffect, or recalibration bias, 42 is observed following prolonged exposure to consistent multisensory discrepancies 8,9 , but also 43 following single trial exposure, the so called immediate or trial-by-trial recalibration effect 4,7 . In 44 either case, the resulting aftereffect bias reflects the persistent influence of previously received 45 multisensory evidence on subsequent behavior. 46 47 underlying the trial-wise ventriloquism aftereffect 7 . We found that medial parietal cortices 48 reflect the persistent encoding of previous multisensory stimuli and are predictive of the trial-49 wise aftereffect 7 . This led us to speculate that brain regions traditionally implied in spatial and 50 working memory 10-13 contribute to the ventriloquism aftereffect, for example by maintaining a 51 representation of previous sensory evidence between trials and mediating its influence on the 52 perception of subsequent stimuli. Such a role of parietal regions in the ventriloquism aftereffect 53 has also been suggested by other studies, and possibly the same parietal processes 54 contribute to both the immediate and long term effects 14,15 . 55 That brain regions involved in short-term memory may contribute to the ventriloquism 56 aftereffect is similarly predicted by studies on other types of serial dependencies in perceptual 57 decision making. Statistical dependencies between judgements made in consecutive trials are 58 seen ubiquitously in sensory and cognitive tasks [16][17][18] . While in many paradigms such 59 dependencies could in principle arise from changes in early sensory representations, the 60 emerging consensus seems to be that these arise from memory-related processes 17,19 . In 61 support of this, recent studies showed that experimental manipulations known to affect 62 memory processes 20,21 , such as changing the delay between stimulus and response, alter 63 serial dependencies during judgements of visual orientations 22 , the accumulation of the 64 ventriloquism aftereffect 9 , and longer reaction times reduce perceptual bias in visual 65 discrimination 23 . Collectively, the functional analogy of the trial-wise ventriloquism aftereffect 66 with serial dependencies in perceptual decision making and the neuroimaging studies implying 67 medial parietal regions in the aftereffect, make a strong case for a memory-related component 68 in the ventriloquism aftereffect. 69 We set out to test the hypothesis that the trial-wise ventriloquism aftereffect is related to 70 memory processes, and hence susceptible to manipulations known to interfere with working 71 memory. In three experiments we i) manipulated the delay between the inducing audio-visual 72 (ventriloquist) stimulus and the associated response, ii) manipulated the delay between 73 stimulus and response in the auditory trial, or iii) used a masker trial in between the audio-74 visual and the auditory trial to interfere with mnemonic processes. We found that none of the 75 manipulations led to a consistent and robust change in the aftereffect bias, suggesting that 76 the ventriloquism aftereffect is more robust to memory-manipulations as expected from similar 77 studies on serial dependencies in serial perception. 78

Multisensory response biases 80
In three experiments we probed participants' judgments of sound location in audio-visual (AV) 81 trials and subsequent auditory (A) trials (Figure 1). In the AV trials, spatially localized sounds 82 were accompanied with spatially localized random-dot patterns presented at either the same 83 location or a range of spatial discrepancies (ΔVA). This allowed us to quantify the 84 ventriloquism effect, reflecting the bias induced by the visual stimulus on the perceived 85 location of the simultaneous sound. The responses in the subsequent A trials allowed us to 86 probe the trial-wise ventriloquism aftereffect, reflecting the persistent influence of the 87 multisensory discrepancy experienced in the AV trial on the judgement of a subsequent 88 unisensory sound 4,7 . Each experiment manipulated the sequence of AV-A trials in a different 89 manner: experiment 1 induced a variable delay before the response in the AV trial, experiment 90 2 induced a variable delay before the response in the A trial, and experiment 3 introduced a 91 sensory-motor masker stimulus in between AV and A trials (

103
In a first step we determined the dependency of each bias on the multisensory discrepancy, 104 ΔVA. For this we combined the data across all three experiments and, following previous 105 studies 14,24,25 , compared linear and non-linear models in their predictive power for each bias 106 (eq. 1-3). This revealed 'very strong' evidence that the ventriloquism bias was best explained 107 by a combined linear and non-linear dependency on ΔVA: relative group-level BIC values = 108 87, 74, 0, for models 1-3 (c.f. Materials and Methods). In contrast, the aftereffect was best 109 described as a linear dependency on ΔVA: rel-BIC = 0, 11, 11 (models 1-3). In the following 110 we hence focused on the combined linear and nonlinear model (eq. 3) for the ventriloquism 111 bias and a linear model (eq. 1) for the aftereffect to probe whether these are affected by the 112 experimental manipulations. 113

Manipulating the delay within audio-visual trials 114
In the first experiment (n=20) we manipulated the temporal delay between the audio-visual 115 stimulus and participant's response in the AV trial, which could take one of the five average 116 values (0.5s, 1s, 2s, 4s, and 8s; ± 200 ms uniform random jitter in each trial). This manipulation 117 could in principle affect both the ventriloquism bias and the aftereffect bias. Figure 2A shows 118 the resulting biases (as participant-averaged data) for the two extreme values of the delay (0.5 119 and 8s). 120 We implemented two separate analyses to probe whether the biases differed as a function of 121 delay. In a first approach, we fit a GLMM across all single trial biases, conditions and 122 participants. Extending model 3 by the delay as an additional factor provided "very strong" 123 evidence in favor of no effect of delay (BIC 55940 without and 55963 with including the delay; 124 ΔBIC=23). The model parameters for the full model including the delay and its interactions 125 revealed no significant effect for delay ( Table 1). 126 In a second approach, we fit the participant and condition-wise trial-averaged biases using 127 individual regression models and investigated whether these two slopes (linear, nonlinear) 128 differed as a function of delay using a non-parametric test ( Figure 2B): neither slope revealed 129 an effect of delay (Friedman's nonparametric ANOVA, reporting FDR corrected p-values; 130 linear term: chi(4,99)=4.5, pfdr =0.95, quadratic term: chi(4,99)=3.1, pfdr =1.2). Hence, our data 131 offer no evidence that manipulating the delay between the AV stimulus and the associated 132 response affects the strength of the ventriloquism bias. 133 Table 1  The same manipulation also did not affect the ventriloquism aftereffect (Figure 2A). The 140 addition of the delay in model 1 resulted in a reduced fit (BIC without 52322 and with delay 141 52339; ΔBIC=17 providing "very strong" evidence in favor of no effect) and in the combined 142 model neither the effect of delay nor its interaction with ΔVA were significant ( Table 1). The 143 analysis of participant-and condition-wise biases led to the same conclusion ( Figure 1C; 144 linear term: chi(4,99)=6.8, pfdr = 0.6; Figure 2B).

152
Manipulating the delay within auditory trials 153 In a second experiment (n=21) we tested whether adding a similar temporal delay between 154 the auditory stimulus and the response in the A trial would affect the two biases ( Figure 3A). 155 First, and as expected given that the manipulation was specific to the A trial, we found that the 156 ventriloquism bias was not affected: adding the delay as factor did not improve model fit (BIC 157 without 57840 and with delay as factor 57865; ΔBIC=25 providing "very strong" evidence in 158 favor of no effect) and the factor delay and its interactions were not significant ( Table 2). The 159 condition-and participant-wise biases were also not significantly different between delays 160 (linear term: chi(4,104)=1.4, pfdr = 1.7, quadratic term: chi(4,104)=1.6, pfdr =1.7; Figure 3B).  Interestingly, also the aftereffect did not change with the delay in this experiment: the addition 168 of the delay did not improve the model fit (BIC 54617 vs. 54633; ΔBIC=15 providing "very 169 strong" evidence in favor of no effect) and the interaction terms were not significant ( Table 2). 170 The same conclusion was supported by the participant-and condition-wise biases (linear term: 171 chi(4,104)=8.1, pfdr = 0.4; Figure 3B). and auditory modalities (a spatially diffuse sound) and required the participants to make a 185 motor response to also mask potential memory traces of the preceding motor response in the 186 AV trial. For comparison, participants performed blocks with the interleaved masking trial and 187 without. We ensured that the overall temporal delay between the AV and A trials was 188 comparable across these two conditions. 189 As in the two preceding experiments, we observed robust ventriloquism and aftereffect biases 190 in the AV and A trials ( Figure 4A). As expected given the experimental design, the 191 ventriloquism effect did not differ significantly between conditions. The addition of the masking 192 condition as factor did not improve model fit (BIC without 60978 and with masker 60998; 193 ΔBIC=20 providing "very strong" evidence in favor of no effect) and the interaction terms were 194 not significant ( Table 3)  Interestingly the masking manipulation did not affect the ventriloquism aftereffect. The addition 205 of masking condition did not improve the model fit (BIC without 58439 and with delay 58455; 206 ΔBIC=15 providing "very strong" evidence in favor of no effect) and the model parameters 207 revealed no significant contribution of condition (Table 3). Finally, the analysis of individual 208 participant data revealed no significant difference in slope ( Fig. 4; chi(1,43) We have previously shown that parietal representations of multisensory spatial information 231 are maintained between trials in the ventriloquism paradigm, and are predictive of the 232 aftereffect bias 7 . Hence, a role of short-term memory in the aftereffect is directly suggested by 233 neuroimaging results. 234 Second, previous work on serial dependencies in unisensory perceptual tasks has suggested 235 that these dependencies do not arise from sensory-level affects but rather reflect higher 236 cognitive processes such as memory or the use of remembered information for subsequent 237 decisions 16,18,22 . For example, a study on serial dependencies in visual judgements has used 238 very similar mnemonic manipulations of temporal delays to show that the trial-wise biases are 239 affected by the delay manipulation 22 . Hence, the observation that sensory and meta-cognitive 240 variables carry over between trials even in simple laboratory paradigms also suggests a role 241 of memory-related processes in the ventriloquism aftereffect. 242 While we did not find a dissipating effect of temporal delays or sensory maskers on the 243 ventriloquism aftereffect, a previous study suggested that intervening audio-visual trials before 244 the auditory trial lead to a reduction of the trial-wise aftereffect 4 . This suggests that 245 multisensory information that bears the very same task-relevance can reduce the aftereffect, 246 while a masking stimulus that comprises distinct audio-visual features and pertains to a 247 different task does not, as seen in the present study. In addition, one study used repetitive AV 248 trials to induce the ventriloquism aftereffect and found that this accumulates over repetitions 249 but also dissipates over delays of 5 s and 20 s when no sensory interference is present 9 . 250 Hence, the combined evidence suggests that the trial-wise aftereffect and that induced by 251 prolonged and repetitive exposure to a consistent audio-visual discrepancy differ in their 252 sensitivity to memory interference. Still, future work is needed to directly test this hypothesis 253 within the same participants and experimental design. cursor. Each trial started with a fixation period (Exp1,2: uniform 1100 ms -1500 ms; Exp3: 318 1000 ms -1200 ms) followed by the stimulus (50 ms). After a random post-stimulus period 319 (see below) the response cue emerged, which was a horizontal bar along which participants 320 could move a cursor. A letter 'T' was displayed on the cursor for 'tone' in the AV or A trials, 321 and 'V' for the V trials to indicate which stimulus participants had to localize. There were no 322 constraints on response times, however the participants were instructed to respond intuitively, 323 and to not dwell too much on their response. Inter-trial intervals varied randomly (see below). 324 A typical sequence of trials is depicted in Figure 1. and hence devoid of spatial information (Figure 1B, bottom). The motor masking task was to 346 bring the cursor appearing randomly along the horizontal line to the middle red target box 347 ( Figure 1B, bottom). Experiment 3 comprised equal numbers of masking trials and no-348 masking (control) trials per level of ΔVA. Given that the stimulus and response in the masking 349 trials required additional time, we extended the inter-trial interval between AV and A trials in 350 the no-masking condition so that the average duration between the response in the preceding 351 AV trial and the subsequent stimulus of the A trial was comparable between AV-A sequences 352 with and without the masking trial (Figure 1B, bottom). Bias ~ β0 + β1⋅ΔVA + (1/subj) (eq. 1) 376 Bias ~ β0 + β1⋅(ΔVA) ½ + (1/subj) (eq. 2) 377 Bias ~ β0 + β1⋅ΔVA + β2⋅(ΔVA) ½ + (1/subj) (eq. 3) 378 Here, and in the following, (ΔVA) ½ stands for the signed square-root of the magnitude of ΔVA 379 (i.e. sign(ΔVA) * sqrt(abs(ΔVA)), bias stands for the single-trial bias (ve, or vae), and subj 380 stands for the participant id. The specific form of nonlinear dependency was chosen based on 381 previous work 14,24,25 . Models were fit using a maximum likelihood procedure using the Laplace 382 method in Matlab R2017a (fitglme.m). These models were compared based on their 383 respective BIC. Interpretations of differences in BIC's were based on established criteria, with 384 values larger than 6 corresponding to "strong" and those larger than 10 to "very strong" 385 evidence 36 . This revealed (see Results) that model 3 provided the best fit for the ventriloquism 386 bias and model 1 for the aftereffect. 387 We then used two approaches to probe whether the ventriloquism bias or the aftereffect are 388 significantly affected by the manipulations of the delays or the masking condition. Each 389 experiment was analyzed separately in the following two ways. In a parametric approach, we 390 extended the above models (model 1 for the vae; model 3 for the ve) by the trial-specific delay 391 (in milliseconds) or the masking condition as additional (continuous or categorical) factors, 392 including their interaction with the linear (and possibly also nonlinear) ΔVA-dependencies. 393 Again we compared BIC values between the respective model without delay (masking 394 condition) and the model including these. In addition, we investigated the respective model 395 parameters and their confidence intervals (Tables 1-3). 396 In the second approach we asked whether the distribution of the condition-wise and