Phasic and sustained interactions of multisensory interplay and temporal expectation

Every moment organisms are confronted with complex streams of information which they use to generate a reliable mental model of the world. There is converging evidence for several optimization mechanisms instrumental in integrating (or segregating) incoming information; among them are multisensory interplay (MSI) and temporal expectation (TE). Both mechanisms can account for enhanced perceptual sensitivity and are well studied in isolation; how these two mechanisms interact is currently less well-known. Here, we tested in a series of four psychophysical experiments for TE effects in uni- and multisensory contexts with different levels of modality-related and spatial uncertainty. We found that TE enhanced perceptual sensitivity for the multisensory relative to the best unisensory condition (i.e. multisensory facilitation according to the max-criterion). In the latter TE effects even vanished if stimulus-related spatial uncertainty was increased. Accordingly, computational modelling indicated that TE, modality-related and spatial uncertainty predict multisensory facilitation. Finally, the analysis of stimulus history revealed that matching expectation at trial n-1 selectively improves multisensory performance irrespective of stimulus-related uncertainty. Together, our results indicate that benefits of multisensory stimulation are enhanced by TE especially in noisy environments, which allows for more robust information extraction to boost performance on both short and sustained time ranges.


Supplement I: Beta estimates for all models
Supplementary Figure 1: Here we show all beta estimates for the d' models (top) and the RT models (bottom) reported in our publication. Beta estimates are shown for each factor (columns) as well as the intercept. Error bars depict standard errors. Significance values are indicated as follows: *** p<.001, ** p<.01, * p<.05, n.s. implies p>.1.

mean RT in ms
However, we did not expect that the interaction of TE and modality would result in a large effect size. A large interaction effect size would imply that almost all participants show multisensory enhancement and a larger enhancement in the expect condition across all experiments. Given the manipulation of experimental context (i.e. spatial and target uncertainty) across experiments this is highly unlikely. For example, in the low target and low spatial uncertainty experiment (Exp.1), task difficulty was minimised compared to all other experiments. Hence, multisensory stimulation might be less beneficial. In accord with this notion, 14 out of 30 participants consistently did not show multisensory facilitation there. Furthermore, 7 participants in this experiment showed multisensory enhancement only in the unexpected condition, i.e. the more difficult condition due to missing preparedness. In all 4 experiments combined, 24 of 120 participants showed no sign of multisensory facilitation and 16 participants showed no improvement due to TE (4 people did neither show TE nor multisensory enhancement). For a total of 15 out of 120 participants, multisensory enhancement was restricted to unexpected trials. However, the majority of 81 participants showed multisensory facilitation in the expected condition especially in the high spatial uncertainty experiments which gave rise to the reported interaction effect size.
Based on our previous report, we could have restricted our analysis to those experiments with a robust overall interaction (i.e. experiments with high spatial uncertainty) as this interaction is the main focus of our manuscript. A reduced ANOVA approach would have indeed resulted in a highly significant 'TE*modality' interactions with a pointedly larger effect size (i.e. η 2 = .12). However, we chosein the interest of the readershipto report the full scope of our results, thereby reducing effect sizes.
Finally, effect sizes might have also been affected by the ratio of early and late target trials in the 'expect early' and 'expect late' blocks (86%-14% and 43%-57%, respectively). The reason behind our decision to use more early trials in the expect-late blocks was to have a more robust estimate of unexpected early trials. Performance in the expected early condition is based on 144 trials and in the unexpected early condition on 72 trials. If we would fully reverse probabilities, unexpected early performance would be based on 24 trials which significantly lowers reliability of the performance measure. However, it most likely would have increased the overall TE effect (see Exp. 5 1 ) and could have also affected the interaction term.

Supplement V: Late targets
As for early targets, the auditory modality was the preferred modality when targets were presented late (dʹ: 72 of 120, RT: 66 of 120).
The graphs below illustrate that under high uncertainty, RTs increased in the best unisensory condition (Supplementary Supplementary Figure 2  As in our previous report 1 , the late target results support the notion that late targets are always expected and that TE effects are restricted to scenarios with temporal uncertainty. Yet audiovisual stimulation can still enhance target perception (as indicated by the d' effect). However, the pattern of results found for RTs rather indicates that differences were driven by decision processes. If e.g. perceptual latency was truly shortened, we would expect that reaction times decrease in the expected condition (as for AV trials, see Supplementary Figure 2 right). However, here RTs decreased in the unexpected condition indicating that participants possibly lowered their response threshold in the unexpected unisensory condition. This condition was most likely less often perceived than the audiovisual condition, hence, participants guessed more often fast than slow.