Abstract
Children with autism spectrum disorders (ASD) exhibit alterations in multisensory processing, which may contribute to the prevalence of social and communicative deficits in this population. Resolution of multisensory deficits has been observed in teenagers with ASD for complex, social speech stimuli; however, whether this resolution extends to more basic multisensory processing deficits remains unclear. Here, in a cohort of 364 participants we show using simple, non-social audiovisual stimuli that deficits in multisensory processing observed in high-functioning children and teenagers with ASD are not evident in adults with the disorder. Computational modelling indicated that multisensory processing transitions from a default state of competition to one of facilitation, and that this transition is delayed in ASD. Further analysis revealed group differences in how sensory channels are weighted, and how this is impacted by preceding cross-sensory inputs. Our findings indicate that there is a complex and dynamic interplay among the sensory systems that differs considerably in individuals with ASD.
Similar content being viewed by others
Introduction
Biological events tend to be multisensory, emanating or reflecting multiple forms of energy (e.g., photons, airborne vibrations, volatilised molecules, etc.). Humans have evolved a highly specialised set of sensory receptors that enable us to sample these different forms of energy concurrently, optimising how we perceive ecologically relevant information. For instance, processing redundant audiovisual cues often leads to faster response times (RTs) than processing the same information separately, a phenomenon known as the redundant signals effect (RSE)1,2,3. The RSE can be explained by parallel processing models such as the popular race model, which predicts that a response is triggered independently by the faster sensory channel4. However, the observed RSE typically exceeds the benefit predicted by mere statistical facilitation5. This multisensory effect has been demonstrated using bisensory detection tasks for several decades and is widely interpreted as reflecting the speed-up in processing time due to multisensory integration6,7,8,9,10,11,12,13,14,15.
Whereas multisensory processing clearly influences how we perceive most biological events, particularly in instances when sensory evidence is ambiguous16,17,18,19, individuals with autism spectrum disorders (ASD) often do not benefit from the availability of multisensory information to the same extent as their neurotypical (NT) peers20,21,22,23,24,25,26,27. We and others have suggested that impaired multisensory processing in ASD contributes to some of the commonly associated phenotypes such as atypical responses to sensory stimulation, and may even have detrimental effects on higher-order cognitive processes such as social interaction and communication28,29,30,31,32,33,34.
Previous studies using audiovisual (AV) speech stimuli have demonstrated that multisensory processing deficits observed in high-functioning children with ASD are not evident in teenagers with the disorder25,35. In contrast, when performing a simple AV detection task with flashes and beeps, high-functioning teenagers with ASD failed to show the same levels of multisensory gain as their NT peers36. Recent theoretical32 and computational37 perspectives have suggested that the constant exposure to AV speech during maturation may serve to train multisensory speech function, leading to resolution of AV speech deficits in ASD at an earlier age. However, the empirical question remains as to whether the resolution of multisensory deficits in ASD is specific to complex, social stimuli, or if instead it generalizes to more basic deficits in multisensory processing. This has obvious implications for understanding the basis of impairments in higher-order cognitive processes in ASD (e.g., social communication), as well as for neurobiological theories of ASD. Here, using the same bisensory detection task36,38, we tested the hypothesis that basic multisensory deficits observed in high-functioning children and teenagers with ASD would not be evident in adults with the disorder.
Electrophysiological studies in the cat superior colliculus have shown that the ability of neurons to integrate multisensory inputs is not present at birth39,40, but rather emerges and matures in the immature nervous system with exposure to multisensory experiences41,42,43,44. Computational modelling suggests that multisensory signals interact by default in a competitive manner, inhibiting effective processing of such stimuli45. Considerable postnatal exposure to multisensory cues is thought to strengthen excitatory cross-sensory projections, promoting processing of a facilitative nature46,47,48. While numerous developmental studies in humans have reported reduced multisensory ability in young children49,50, there is little empirical evidence of such competitive multisensory processing other than that reported in adults51,52,53. Here, we used a computational modelling framework to directly test whether behaviour in children reflected multisensory processing of a competitive or facilitative nature. The type of information processing was determined by how accurately hypothesis-driven models of multisensory behaviour could predict empirical multisensory benefits15,54. The same modelling approach was used to quantify age-related changes in sensory dominance and any potential group differences therein. We expected that such inherent sensory weighting would have a greater impact on multisensory processing of a competitive nature.
Multisensory experiments typically require participants to sequentially process different sensory stimuli in quick succession, and can thus be thought of as a task-switching paradigm. When switching from one modality to another, average response times are slower on trials preceded by a different sensory modality (switch trials) compared to trials preceded by the same modality (repeat trials)55,56,57. These so-called modality switch effects (MSEs) are inherent to any bisensory detection task that uses an intermixed stimulus presentation design58,59 and have been shown to systematically contribute to the RSE because they are typically larger on unisensory trials than on multisensory trials60,61,62. Moreover, data suggest that children with high-functioning ASD incur a greater cost when switching from auditory to visual stimuli than their NT peers63. We therefore examined potential group differences in MSEs and quantified their contribution to the RSE. Using a computational framework64, we modelled the trial-to-trial dependency between RTs on different sensory channels to gain deeper insight into how this process could be linked to inhibitory cross-sensory switching effects53,65. We discuss the implications of task-switching on the interpretation of the RSE, and how the interplay between multisensory integration and task-switching may contribute differentially as a function of age in NT and ASD individuals.
Results
Redundant signals effect
In all, 225 neurotypical (NT) individuals (age range: 6–36 years; 115 females) and 139 individuals with a diagnosis of ASD (age range: 6–39 years; 34 females) performed a speeded bisensory detection task36,38 in which they were required to respond as fast as possible with a button press to randomly alternating auditory (A; 1 kHz tone), visual (V; red disk) and audiovisual (AV; disk/tone pair) stimuli, each with a duration of 60 ms and an interstimulus interval (ISI) of 1000–3000 ms (see Supplementary Fig. 1). To examine potential age-related effects, age was treated either as a continuous variable, or participants were separated into four cross-sectional age groups: 6–9, 10–12, 13–17 and 18–40 years (see Table 1 for group sample sizes and demographics). A linear mixed-effects analysis was used to examine the effects of group, age, and stimulus condition on response times (R2adj = 0.53). ISI, preceding modality and subject were included as random factors, and the slopes of the latter 2 were adjusted for condition66 (see Methods section for further details). Participants with ASD responded slower to stimuli than their NT peers (β = 34.45, SE = 11.59, p = 0.003; Fig. 1a). There was an effect of age, with older participants responding faster than younger participants (β = −8.98, SE = 0.79, p = 1 × 10−29). Responses to multisensory stimuli were faster than those to both audio (β = 53.89, SE = 10.53, p = 3 × 10−7) and visual (β = 69.64, SE = 6.49, p = 8 × 10−27) stimuli, indicating the presence of an RSE. However, there was an interaction between age and RSE (RSEA: β = −0.58, SE = 0.21, p = 0.005; RSEV: β = −0.56, SE = 0.18, p = 0.002).
To examine the RSE further, we measured the proportion of RTs that were faster on multisensory trials relative to the faster of the unisensory conditions, i.e., multisensory benefit54. For this, RT distributions were converted to cumulative distribution functions (CDFs) to enable analysis of the whole RT distribution67,68. Empirical benefits were quantified by the area between the CDFs of the multisensory condition and the faster of the unisensory conditions (Eq. 4; Fig. 1b, right). Analogously, predicted benefits were quantified by the area between the CDFs of the race model and the faster of the unisensory conditions (Eq. 3; Fig. 1b, left)54. Here, we used Raab’s independent race model4, which assumes statistical independence between sensory channels (see Methods for further details). A linear regression model was constructed to quantify the effects of group and age on predicted (R2adj = 0.09, BF01 = 9 × 10−6) and empirical (R2adj = 0.25, BF01 = 2 × 10−21) benefits. Predicted benefits decreased as a function of age (β = −0.001, SE = 0.0001, p = 2 × 10−8, BF01 = 3 × 10−6) and were not significantly different in NT and ASD individuals (β = 0.002, SE = 0.0019, p = 0.28, BF01 = 9.1). Conversely, empirical benefits increased with age (β = 0.002, SE = 0.0002, p = 1 × 10−19, BF01 = 3 × 10−17) and were smaller in individuals with ASD (β = −0.013, SE = 0.003, p = 1 × 10−5, BF01 = 1 × 10−3). This suggests that the race model over-predicts empirical benefits for younger individuals and under-predicts them for older individuals (see Fig. 1c). Moreover, the race model does not predict the group differences observed in empirical benefits, suggesting a deficit specific to multisensory processing.
Testing the race model
To test whether the speed-up due to multisensory processing exceeded statistical facilitation, we quantified the proportion of audiovisual RTs that deviated from the predictions of Raab’s independent race model at every RT quantile. Positive deviations indicate multisensory facilitation and were detected using right-tailed permutation tests with tmax correction69. NT participants showed evidence of facilitation at one or more quantiles in every age group, the number of quantiles increasing as a function of age (p < 0.05, shaded area, Fig. 2a). The percentage of participants that exhibited facilitation is illustrated in Supplementary Fig. 2. Individuals with ASD showed no evidence of facilitation between the ages of 6–12 years (Fig. 2b). However, evidence of facilitation emerges in adolescence (first quantile) and becomes more evident in adulthood (first 3 quantiles; see Supplementary Table 2 for test statistics).
To compare deviations from the race model between NT and ASD individuals of different ages, we computed the root-mean-square error (RMSE) and correlation coefficient between each participant’s difference function and that of every other participant. Prior to assessing the Pearson correlation, a rank-based inverse normal (RIN) transformation was applied to the data70. Participants were split into 10 age bins separated by 2.5 years between the ages of 6–21 years. Similarity matrices containing RMSE and correlation values were obtained by averaging over the values within each age bin (Fig. 2c). The red lines indicate the NT groups that were most similar to each ASD group, and its divergence above the dotted midline suggests that multisensory behaviour in individuals with ASD corresponded more closely to that of younger NT individuals (i.e., a potential delay in age-related changes). Convergence of the red lines towards the midline in 18.5–21-year-olds suggests that this delay may resolve in adulthood, in line with our original hypothesis. This absence of multisensory deficits in adulthood is further examined in the following section.
Resolution of multisensory deficits in ASD
To examine this potential delay in age-related changes in ASD, we constructed a linear model to evaluate the effects of group and age on multisensory gain (R2adj = 0.39, BF01 = 1 × 10−35). Multisensory gain was quantified by the net proportion of multisensory RTs that were faster than the independent race model (Eq. 5; Fig. 3a). Multisensory gain exhibited a moderate increase as a function of age (β = 0.0025, SE = 0.0002, p = 4 × 10−21, BF01 = 0.98) but was significantly reduced in participants with ASD compared to NT individuals (β = −0.023, SE = 0.007, p = 0.001, BF01 = 6 × 10−5). The absence of an interaction between group and age suggests that this age-related effect was present in both groups (β = 0.0005, SE = 0.0004, p = 0.22, BF01 = 8.84). Group comparisons were conducted within each of the four age groups. For this, NT participants were sex, age and IQ-matched to each of the ASD participants and difference functions were compared at every quantile using two-tailed (unpaired) permutation tests (see Methods section for further details). Group differences were observed in the 10–12-year-olds at quantiles 4–6, and in the 13–17-year-olds at quantiles 2–5 (p < 0.05, shaded area, Fig. 3c). To determine whether this multisensory deficit in ASD can be said to resolve in adulthood, a Bayes factor analysis was used to provide evidence in favour of the absence of any group differences based on the net multisensory gain (Fig. 3d). Group differences were observed in participants aged 6–9 years (t(86) = 2.37, p = 0.021, Hedge’s g = 0.50, 95CI [0.09, 0.94], BF01 = 0.47), 10–12 years (t(64) = 3.48, p = 0.001, Hedge’s g = 0.85, 95CI [0.40, 1.36], BF01 = 0.03) and 13–17 years (t(56) = 3.13, p = 0.003, Hedge’s g = 0.81, 95CI [0.33, 1.37], BF01 = 0.08), but there was no evidence to suggest a group difference in participants aged 18–40 years (t(64) = 1.02, p = 0.310, Hedge’s g = 0.25, 95CI [−0.23, 0.77], BF01 = 3.34).
To examine these age-related changes in greater detail, we treated age as a continuous variable (Fig. 3e). Age was highly predictive of multisensory gain between 6–17 years (NT: R2 = 0.34, p = 2 × 10−5, BF01 = 7 × 10−13; ASD: R2 = 0.21, p = 2 × 10−5, BF01 = 8 × 10−5) but not between 18–40 years (NT: R2 = 0.005, p = 0.56, BF01 = 9.22; ASD: R2 = 0.015, p = 0.5, BF01 = 5.92), suggesting that progression of this process ceases in adulthood. To characterise the trajectory of multisensory processing as a function of age more precisely, we calculated the mean multisensory gain with a moving window k of 7 years in increments of 1 year between the ages of 6 and 26 years (Fig. 3f). Controls were sex, age and IQ-matched to ASD individuals within each 7-year window and compared using two-tailed permutation tests (FDR corrected) and Bayes factor analyses. In NT participants, multisensory gain increased steadily between the ages of 6–17 years. In individuals with ASD, the rate of increase was slower and the average gain was significantly lower than that of their NT peers between 6–17 years (p < 0.05, shaded area, Fig. 3f). In adults aged 18 years and over, there was no evidence to suggest any significant group differences (p > 0.05). Moreover, our Bayes factor analysis provided evidence in favour of the absence of an effect between the ages of 22–26 years (BF01 > 3; see Supplementary Fig. 3), suggesting that this deficit resolves in adulthood as originally hypothesised.
Modelling multisensory behaviour in ASD
Raab’s independent race model has been shown to provide strong predictions of the RSE in healthy adults, suggesting that the basic underlying neural architecture may follow that of the race model15,54,64. This is also consistent with the fact that the logic operations of the race model (i.e., logic OR gate) exactly match the tasks demands of a bisensory detection task (i.e., respond to the presence of a signal on channel A or channel V)59. To assess whether the independent race model could provide similar predictions of multisensory behaviour in children and individuals with ASD, one-way ANCOVAs were used to measure the correlation between predicted and empirical benefits in each age group by treating age group as a partialled out categorical variable71. Predicted benefits were correlated with empirical benefits in NT individuals (F(1,217) = 62.88, p = 1 × 10−13, R2 = 0.23, BF01 = 1 × 10−19) and less so in individuals with ASD (F(1,131) = 5.64, p = 0.019, R2 = 0.041, BF01 = 0.35), but an interaction between age group and predicted benefits in NT individuals suggested that this relationship was age-dependent (NT: F(3,217) = 5.4, p = 0.0013, R2 = 0.07, BF01 = 0.97; ASD: F(3,131) = 2.22, p = 0.088, R2 = 0.05, BF01 = 51.9). Figure 4a, b shows that the ability of the race model to predict empirical benefits increases dramatically as a function of age. While the race model predicted a significant proportion of the variance in the 18–40-year-olds (NT: R2 = 0.49, p = 2 × 10−5, BF01 = 6 × 10−10; ASD: R2 = 0.15, p = 0.028, BF01 = 0.66), it accounted for almost none of the variance in the 6–9-year-olds (NT: R2 = 0.008, p = 0.54, BF01 = 7.58; ASD: R2 = 0.002, p = 1.0, BF01 = 8.15).
In light of this result and recent findings, we wished to test the hypothesis that multisensory processing in early childhood is governed by a competition rather than facilitation45. While facilitation would likely follow the predictions of the race model, we hypothesized that competition would follow the predictions of the more dominant sensory modality. Thus, we tested the predictions of models that were biased towards either a specific modality (Models 1A and 1V; Eq. 6) or the previous modality (Models 2A and 2V; Eq. 7). The probability p of multisensory processing being facilitative (race model) or competitive (bias model) was parametrically varied between 0 and 1 in increments of 0.25 (Eq. 8; see Methods section for further details). In children with ASD aged 6–9 years, Model 2A was most accurate at predicting empirical benefits, suggesting that their responses were triggered by the previous modality with an auditory bias (Fig. 4c). In their NT peers, none of the bias models exceeded the performance of the race model considerably, but there was evidence in favour of an auditory bias once again. In ASD participants aged 10–12 years, Model 2V dramatically exceeded race model performance, suggesting that RTs were largely determined by the previous modality but this time with a visual bias (Fig. 4d). In their NT peers, there was no major improvement beyond the race model, but there was evidence of a visual bias as well. In teenagers and adults, none of the bias models outperformed the race model, suggesting that individuals with ASD transition from competition to facilitation during adolescence (Supplementary Fig. 4).
Age-related changes in sensory dominance
To further examine age-related patterns in sensory dominance, we tested Model 1A and Model 1V with the probability of a sensory-specific bias set to 1 (Eq. 8). Evaluating model performance as before, we noticed an auditory dominance in both groups at 6–9 years that shifted to a visual dominance by 10–12 years (Fig. 5a). In the NT group, this visual dominance appears to continue into adulthood in accordance with the well-known Colavita visual dominance effect72. However, in the ASD group, this sensory weighting appears to shift once again in adolescence, leading to an apparent auditory dominance in adulthood.
If such biases exist, one would expect to see larger switch costs on AV trials preceded by the less dominant modality than those preceded by the dominant modality. To test this hypothesis, we examined modality switch effects (MSEs) on AV trials, separating trials preceded by A and V trials (A→AV and V→AV, respectively). MSEs were quantified by deriving separate CDFs for switch and repeat trials and computing the area between them (Eq. 9; see Methods section for further details). Additionally, MSEs in each condition were normalized by that of the grouped condition (V/A→AV) to allow for meaningful comparison across age groups (this did not change the results qualitatively). Based on our modelling of sensory dominance, we expected to see greater MSEs on V→AV trials for NT children aged 6–9 years, and on A→AV trials for NT individuals aged 10 years and over. We expected something similar for ASD individuals with another shift in adolescence. The data in Fig. 5b suggest that, as predicted, MSEs were greater on V→AV trials for NT 6–9-year-olds and on A→AV trials for NT 10–40-year-olds. For ASD individuals, the data suggest the reverse, with greater MSEs on A→AV trials in children and teenagers (6–17 years) and on V→AV trials for adults (18–40 years).
Reduced switch costs in ASD
To detect potential group differences in the behavioural cost of switching sensory modality, we modelled the effects of diagnosis, age, and condition on MSEs (R2adj = 0.304, BF01 = 2 × 10−80). MSEs increased with age (β = 0.002, SE = 0.0002, p = 1 × 10−24, BF01 = 7 × 10−21) and, contrary to previous work63, were reduced in individuals with ASD compared to NT individuals (β = −0.012, SE = 0.002, p = 4 × 10−7, BF01 = 9 × 10−5). Compared to multisensory trials, MSEs were larger on both auditory trials (β = 0.048, SE = 0.003, p = 2 × 10−59, BF01 = 3 × 10−59) and visual trials (β = 0.037, SE = 0.003, p = 7 × 10−38, BF01 = 3 × 10−59). Follow-up permutation tests and Bayes factor analyses revealed that MSEs were only reduced in the adolescent ASD group when switching from auditory to visual stimuli (t(56) = 3.63, p = 0.001, Hedge’s g = 0.94, 95CI [0.46, 1.52], BF01 = 0.02; Fig. 6a). A more detailed examination using a moving mean estimate of MSE showed that these group differences emerged between the ages of 11–17 years (p < 0.05, shaded area, FDR corrected; Fig. 6b, right) and were strongest at around 16 years (Hedge’s g = 0.93, 95CI [0.48, 1.42], BF01 = 0.01). For visual to auditory switches, MSEs were once again reduced in individuals with ASD between 14 and 16 years of age (Hedge’s g = 0.68, 95CI [0.27, 1.14], BF01 = 0.07). For effect size and Bayes factor analyses, see Supplementary Fig. 5.
Contrary to our results, a study by Williams, et al.63 found that individuals with ASD between the ages of 8–15 years exhibited a greater cost to switching from auditory to visual stimuli than their age-matched NT peers. To test this directly, we performed a left-tailed permutation test and Bayes factor analysis on a group of sex, age and IQ-matched participants between the ages of 8–15 years (n = 79 per group) and used a similar measure of MSE based on the mean RT. However, this analysis did not suggest that MSEs were greater in individuals with ASD (NT: 30.0 ± 27.4 ms, ASD: 19.7 ± 42.2 ms; t(156) = 1.82, p = 0.964, Hedge’s g = 0.29, 95CI [−0.01, 0.60], BF01 = 23.58), confirming that the inconsistency between our studies was not a consequence of how MSE was quantified. The only remaining difference between our studies was that Williams, et al.63 used longer ISIs (3–5 s vs. 1–3 s). Thus, we repeated the test, but limited RTs to trials with preceding ISIs between 2.5–3 s. Once again, there was no evidence to suggest that MSEs were greater in individuals with ASD (t(156) = 1.30, p = 0.905, Hedge’s g = 0.21, 95CI [−0.10, 0.51], BF01 = 18.43). However, limiting the analysis to longer ISIs did result in a reduction in mean MSE in both NT individuals (18.5 ± 43.5 ms, t(78) = 3.01, p = 0.008, Hedge’s g = 0.32, 95CI [0.11, 0.58], BF01 = 0.16) and individuals with ASD (7.5 ± 60.7 ms, t(78) = 1.95, p = 0.102, Hedge’s g = 0.23, 95CI [0.00, 0.48], BF01 = 1.79), suggesting invocation of disparate processing underlying MSEs at shorter versus longer ISIs.
Modelling channel dependency and RT variability
To gain deeper insights into the factors that shape multisensory facilitation and switch costs, we adopted a computational modelling framework developed by Otto and Mamassian64. This framework extends Raab’s independent race model such that it includes 2 additional free parameters, η and ρ (Fig. 7b), that enable a more accurate fit to empirical multisensory RTs (see Methods for further details). The first parameter η accounts for the additional variability or noise typically observed in empirical multisensory RTs relative to that predicted by Raab’s race model; this increase could be due to the pooling of neuronal activity (and variability) during multisensory processing. The second parameter ρ allows us to quantify the trial-to-trial correlation between RTs on different sensory channels; this correlation typically presents as a negative channel dependency for healthy adults because neural processing on one channel likely happens at the expense of processing on the other5,64. We can visualise this negative dependency by plotting the average frequency of switch versus repeat trials per quantile in our NT adult group (see Fig. 7a). We hypothesized that the increase in RT variability would be larger for individuals with higher multisensory gain, and that channel dependency would be lower or more negatively correlated for individuals with greater MSEs. The best-fitting estimates of the noise parameter η increased with age (β = 0.005, SE = 0.0008, p = 2 × 10−10) but there was no evidence to suggest a group difference (β = −0.007, SE = 0.012, p = 0.58; R2adj = 0.105; Fig. 7c). The best-fitting estimates of the correlation parameter ρ decreased with age (β = −0.037, SE = 0.003, p = 5 × 10−33) and were more positive-going in ASD (β = 0.22, SE = 0.04, p = 2 × 10−7; R2adj = 0.382; Fig. 7d). Follow-up tests revealed group differences in participants aged 6–9 years (t(86) = 2.32, p = 0.021, Hedge’s g = 0.49, 95CI [0.08, 0.93], BF01 = 0.44), 10–12 years (t(64) = 3.27, p = 0.002, Hedge’s g = 0.80, 95CI [0.31, 1.41], BF01 = 0.05) and 13–17 years (t(56) = 3.20, p = 0.002, Hedge’s g = 0.83, 95CI [0.32, 1.44], BF01 = 0.06). As predicted, channel dependency was more negative for participants that exhibited larger MSEs in both NT individuals (rAV = −0.15, p = 0.02; rA = −0.47, p = 2 × 10−4; rV = −0.36, p = 2 × 10−4) and individuals with ASD (rAV = −0.15, p = 0.02; rA = −0.5, p = 2 × 10−4; rV = −0.29, p = 2 × 10−4; Fig. 7e).
Linking sequential and multisensory effects
To examine the relationship between modality switch effects and multisensory gain, we performed a series of partial correlations across participants, controlling for age (see Supplementary Table 3). As one might predict, there was a strong positive correlation between multisensory gain on switch trials and MSEs on unisensory trials (but not on multisensory trials). However, there was no significant correlation between multisensory gain on repeat trials and MSEs on unisensory trials, whereas there was a strong positive correlation with MSEs on multisensory trials (see Supplementary Fig. 6). This pattern, which was identical in both groups, confirms that MSEs on unisensory trials are more likely to contribute to multisensory gain. Figure 8a, b illustrates the impact of switching sensory modality on multisensory facilitation. 87% of NT individuals exhibited a larger multisensory gain on switch trials than on repeat trials (t(224) = 15.62, p = 0.0002, Hedge’s g = 0.84, 95CI [0.73, 0.96], BF01 = 4 × 10−35), with 83% of individuals with ASD showing the same (t(138) = 7.22, p = 0.0002, Hedge’s g = 0.51, 95CI [0.36, 0.66], BF01 = 4 × 10−9). Nevertheless, when we submitted RTs from the repeat trials to a race model test, both groups exhibited significant levels of deviation from race model (see Supplementary Table 4).
Having established the relationship between MSEs and multisensory gain, we wished to determine whether the contribution of the former was full or partial. To do this, we submitted the data to a mediation analysis73. Specifically, we tested whether MSEs mediated the relationship between participant age and multisensory gain (Fig. 8c, d; see Methods section for further details). First, we established that age was a reliable predictor of both MSE (NT: β = 0.5, SE = 0.06, p = 2 × 10−5; ASD: β = 0.43, SE = 0.09, p = 0.0001) and multisensory gain (NT: β = 0.58, SE = 0.06, p = 2 × 10−5; ASD: β = 0.59, SE = 0.08, p = 0.0002), meeting the first two criteria for mediation. MSE affected gain, controlling for age (NT: β = 0.26, SE = 0.06, p = 5 × 10−5; ASD: β = 0.28, SE = 0.08, p = 0.001) and the mediation effect was significant for both groups (NT: β = 0.13, SE = 0.03, p = 4 × 10−5; ASD: β = 0.12, SE = 0.04, p = 0.001). However, there was still a significant direct path between age and gain when controlling for MSE (NT: β = 0.45, SE = 0.06, p = 1 × 10−5; ASD: β = 0.47, SE = 0.09, p = 5 × 10−5), indicating that MSE only partially mediated the observed relationship between age and multisensory gain.
Discussion
Our data suggest that the resolution of multisensory deficits in ASD generalizes to the case of non-social AV stimuli, but that the trajectory is later than that observed in AV speech studies25,35. We hypothesized that this delay may be due to lack of environmental exposure to such ecologically-rare stimuli, as this would limit the opportunity to strengthen the necessary excitatory cross-sensory connections, which we know only emerge postnatally with considerable exposure to multisensory experiences41,42,43,44. Alternatively, delayed resolution of multisensory deficits could result from invocation of neural processes with longer developmental trajectories. Indeed, multisensory gain in NT individuals has been shown to reach full maturity much later for simple AV stimuli38 compared to AV speech stimuli74. This undoubtedly affects the average age that individuals with ASD will “catch up” to their NT peers.
The disparity in multisensory function trajectories for speech and non-speech stimuli may reflect the fact that multisensory processing occurs across distributed networks and that different stimuli and tasks tap into unique processes with varying trajectories75. The task employed in the current study required the speeded detection of simple AV stimuli, without discrimination, identification, or any other higher-order cognitive processing. Integration of such simple AV stimuli likely consists of early cross-sensory activation of visual and auditory cortical regions, enhancing detection of the incoming visual and auditory inputs, respectively6,76,77. In contrast, speech identification engages an extensive network of hierarchically-organised brain areas, mapping spectrotemporal representations to phonetic representations, and from there to lexical-semantic representations78,79. Moreover, integration of auditory and visual speech cues may act through multiple integrative mechanisms, including early visual activation of auditory cortex, increasing perceptual sensitivity80, and later integration of visual speech content (i.e., place and/or manner of articulation), reducing the density of phonemic and lexical neighbourhoods81,82. Clearly, task demands and stimuli play a major role in the patterns of multisensory deficits and their resolution that are observed for any given experimental paradigm.
Alternatively, differences in age-related changes in multisensory function could be caused by invocation of neural processes distinct from those under study. Phenomena such as modality switch effects, which contribute significantly to multisensory gain in a bisensory detection task, but not in an AV speech identification task, could prolong the perceived trajectory of multisensory processing. While this is consistent with the fact that age-related changes in MSEs (visual to auditory) extended well into adulthood (Fig. 6b, left), the trajectory of multisensory gain was qualitatively unchanged when the contribution of MSEs was reduced by focusing our analysis on the repeat trials (see Supplementary Fig. 7 and Supplementary Table 4). This, and the results of our mediation analysis, suggest that MSEs are not the sole driving factor behind the disparity in multisensory processing trajectories across the two paradigms.
Human behavioural studies have demonstrated the co-occurrence of multisensory competition and facilitation using RT measures51. The existence of a visual dominance in adults, i.e., the Colavita visual dominance effect72, means that directing participants to respond to either the auditory or visual component of an AV stimulus can have an inhibitive or facilitative effect on RTs, respectively52. However, in the same way that the race model is used as a threshold for detecting facilitative multisensory processing, an upper statistical bound should be used to quantify genuine competitive multisensory procesing64. Current neuro-computational perspectives of multisensory development suggest that competition is the default state of integration in the neonatal mammalian brain45. Intuitively, a competition scenario would likely favour the most effective sensory modality, which in our case would be either the modality that is most dominant due to an inherent sensory bias (e.g., Colavita effect), or the preceding modality due to prior allocation of attentional resources. To gain insight into the nature of multisensory processing in children and individuals with ASD, we tested two computational models that reflected the above hypothetical scenarios of multisensory competition. We examined fits between the empirical data and model behaviour that were based on a parametric weighting of the race model (facilitation) and each bias model (competition). In children with ASD aged 6–12 years, model fits suggested that the response to an AV stimulus was biased towards the previous sensory modality, potentially due to the presence of competitive processing, whereas in their NT peers, the same models provided only marginal improvements beyond probability summation. This suggests that NT individuals acquire the ability to process multisensory information in a facilitative manner at an earlier age than their ASD peers, who do not show evidence of facilitation until adolescence. This is consistent with the age at which we observe the emergence of multisensory facilitation within each group (see Fig. 2a, b).
Another interesting finding to emerge from our modelling analysis was that NT children aged 6–9 years appear to be biased towards the auditory modality during AV processing, but thereafter are biased towards the visual modality. The same pattern was demonstrated by a follow-up analysis that examined MSE patterns on multisensory trials. These findings are supported by previous studies that have reported an auditory dominance in infants and young children presented with AV stimuli83,84, as well as the abovementioned Colavita visual dominance effect commonly reported in adults72. Several studies have traced the transition from an auditory to a visual dominance over the course of childhood85,86 and, in line with our data, suggest that this sensory reweighting occurs at ~9–10 years of age86 (for a meta-analysis, see Hirst, et al.87). Sensory reweighting has also been shown to occur ~8–10 years of age for the visual and haptic modalities49. Our modelling analysis suggests that the same trend appears to emerge in children with ASD between the ages of 6–12 years, but then reverses once more during adolescence, preferencing the auditory modality in adulthood. Our MSE analysis suggests that a visual dominance exists initially in children with ASD, shifting to an auditory dominance in adulthood. Indeed, a visual dominance has been previously reported in children with ASD88,89, but its progression with age has not yet been documented to our knowledge. In support of a visual dominance in ASD, our MSE analysis revealed that children and teenagers with ASD found it easier to switch from auditory to the visual stimuli compared to their NT peers.
Prior work by our lab suggests that the neural processes underlying multisensory integration are impaired in children with autism36. Specifically, we found that EEG correlates of integration were weaker (of lower amplitude) and occurred later in the information processing hierarchy. Neural indices of integration over parieto-occipital scalp between 140 and 160 ms were predictive of race model violation in NT children but not in children with ASD. Using the same paradigm, we recorded intracranial electrophysiology in adults with epilepsy and demonstrated that visual stimulation influenced the phase of ongoing oscillations in auditory cortex77, and auditory stimulation influenced the phase of ongoing oscillations in visual cortex76, such that cross-sensory stimulation appears to prime ancillary sensory cortices to make them more receptive to their primary sensory input. The response to the primary sensory input (e.g., visual stimulation of visual cortex) is then enhanced for multisensory trials76, at least in a bisensory detection task such as the current one. Furthermore, neuro-oscillatory phase alignment across the sensorimotor network was significantly enhanced by multisensory stimulation, and was related to the speed of a response77.
Phase resetting of ongoing neural oscillations by functionally distinct and distant neuronal ensembles is thought to be fundamental to multisensory integration90,91,92,93,94. Impaired cross-sensory phase-resetting, as might be predicted by reduced subcortical and cortical connectivity, would likely result in impaired integrative abilities. In autism, there is evidence for such disrupted connectivity95,96, although these findings are mixed and somewhat inconclusive97. Nevertheless, disrupted connectivity could in turn lead to impaired cross-sensory phase-resetting and hence contribute to impaired multisensory processing in ASD. As previously mentioned, weaker cross-sensory inhibition might account for reduced MSEs in ASD65, possibly as a result of poorer brain connectivity. Alternatively, it may be that cross-sensory connectivity is fully intact in children with ASD, but integration of multisensory information has not yet transitioned from a state of competition, to one of facilitation, as discussed earlier46,47. Recent neurocomputational work by our lab has examined the link between such an inhibitory neural architecture and the empirical data present here53,98. Establishing the specific neural mechanisms that underlie impaired multisensory behaviour in children with ASD will likely require the use of more advanced neuroimaging techniques.
One of the unexpected findings to emerge from our MSE analysis was the reduced switch costs in adolescents with ASD. This ran contrary to a recent study by Williams et al.63 that reported larger switch costs (auditory to visual) in individuals with ASD of approximately the same age. Interestingly, a post hoc analysis of our data that focused on trials with longer ISIs (closer to that of Williams et al.63) led to a considerable reduction in mean MSEs but did not yield any evidence of a group difference. While we were unable to detect it empirically due to the limited range of ISIs used in our study (1–3 s), it is possible that there exists an interaction between group and ISI. A possible explanation for this potential interaction points to the so-called “trace theory” which originates from research on MSEs in individuals with schizophrenia99. This theory suggests that sensory information leaves traces of residual activity in different neuronal populations, facilitating the processing of subsequent stimuli of the same sensory modality and inhibiting the processing of stimuli of other modalities. Zubin99 predicted that these traces attenuate over time but persist longer in individuals with schizophrenia. If such an inhibitory cross-sensory mechanism were weaker in individuals with ASD, but persisted longer over time, it would explain the potential interaction and the inconsistency between our findings and that of Williams et al.63. Evidence in support of this theory comes from a recent study that demonstrated that individuals with ASD weight recent stimuli less heavily than NT individuals and that their perception is dominated by longer-term statistics100. While reduced cross-sensory inhibition would undoubtedly facilitate processing of subsequent inputs in other sensory systems and thus lead to lower MSEs in ASD, it would also likely result in greater susceptibility to distraction by task/sensory-irrelevant information. This is consistent with previous neurophysiological research by our lab that demonstrated increased susceptibility to distraction by task-irrelevant stimuli in children with ASD65. This behavioural deficit was accompanied by a reduced neural suppression of sensory-irrelevant information, as indexed by EEG recordings of alpha-band oscillatory activity. Thus, both behavioural and neurophysiological accounts of multisensory attention in ASD are consistent with a reduction in MSE.
Alternatively, reduced MSEs in ASD could be explained by differences in the ability to make predictions about the sensory environment. While individuals with ASD have been shown to utilise longer-term statistics to make predictions about their sensory environment100, other work suggests that they tend to overestimate the volatility of their environment at the expense of learning to build stable predictions101. In the current study, stimuli were presented in a random order with equal probability, meaning there was a 66.6% chance of the same unisensory input occurring on the next trial (including the AV condition). Based on these statistics, it is more efficient to predict the reoccurrence of the same signal (or part of it) on the next trial and to direct attention therein. If these statistics are not being actively used to build predictions about the modality of an upcoming stimulus, as may be the case in ASD, then the participant may be less likely to prepare for it and thus less averse to switching sensory modality. This fits well with the notion that individuals with autism rely more on bottom-up than top-down processing102.
It is well established that MSEs systematically contribute to multisensory facilitation in a bisensory detection task60,61,103. To determine the role of MSEs, we performed separate tests of the race model using switch and repeat trials. While we found that multisensory gain was much greater on switch trials than on repeat trials, there was still evidence of facilitation on repeat trials. However, it is important to consider that in the context of a mixed block design, responses on repeat trials are likely subject to residual switch effects from earlier trials (n−2, n−3, etc.). Furthermore, if we consider the impact that switching modality has on RTs, a mixed block design could be said to violate the assumption of context invariance. While it is unlikely that it would present the opportunity to change strategy from trial to trial in a top-down manner, it is conceivable that the continuously changing context (from switch to repeat conditions) could invoke disparate processing mechanisms in a bottom-up manner (for a detailed discussion, see Shaw et al.62). We also measured the correlation between multisensory gain and MSEs on unisensory and multisensory trials, partialling out the effects of age. There was a strong positive correlation for unisensory (but not multisensory) stimuli, as would be expected if MSEs were to impact multisensory gain systematically. This was followed up with a mediation analysis to determine whether MSEs mediated the observed relationship between age and multisensory gain. This analysis indicated only partial mediation, suggesting that neural processes other than MSEs (e.g., cross-sensory integration) were contributing to the observed multisensory gain.
Another way to examine the contribution of MSEs is to remove the presence of switch trials by using a blocked design. In another study by our lab62, we demonstrated that RTs to simple AV stimuli were not faster than race model predictions when the three conditions are presented in entirely separate blocks. Comparing the median RTs between blocked and mixed conditions revealed a slowing of the unisensory but not the multisensory RTs in the mixed condition that could be largely accounted for by increased RTs on switch trials. A similar null result was reported for an AV word detection task that used a blocked continuous speech paradigm104. In another study that employed a blocked design, Otto and Mamassian64 reported evidence of race model violation, but importantly, presented AV stimuli in background noise which are more likely to recruit integrative mechanisms during bisensory detection14,18,105,106. However, it is important to consider the theoretical implications of employing a blocked design. The race model test relies on the assumption of context invariance, because unisensory RTs distributions are used to model multisensory RTs distributions5,58,107. By randomly interleaving the conditions, the participant does not know which condition to expect and presumably processes, say, an auditory signal in the same way under unisensory and multisensory conditions. Thus, positive deviations from the race model are assumed to be caused by multisensory integration as opposed to differences in processing strategies. In contrast, when unisensory and multisensory stimuli are presented in separate blocks, there may be opportunity for the subject to employ different processing strategies to optimise task performance. Hence, it is inherently difficult to disentangle contributions of switching and integration when examining the RSE. Violation of the race model likely involves an interplay between integrative and switching processes that carry different weights in different contexts (mixed vs. blocked presentations) and under different stimulus conditions (clean vs. noisy).
We can draw several conclusions from the present study. (1) When assessed using the race model test, multisensory processing in individuals with ASD has “normalized” to neurotypical levels by adulthood. (2) Computational modelling suggested that multisensory processing in children with ASD takes longer to transition from the default mode of competition to facilitation. (3) Different age-related patterns in sensory dominance indicate fundamental alterations in how the nervous system of children with ASD responds in a dynamic multisensory environment. (4) The behavioural cost of switching from auditory to visual stimuli is smaller in teenagers with ASD, possibly due to altered cross-sensory inhibition or reduced influence of short-term statistics. The current findings also make clear that there is significant work ahead of us before we fully understand the neural processes that contribute to age-related changes in multisensory function and how it differs in children with ASD. Complicating this endeavour, we must begin to understand how such empirical evidence collected in high-functioning individual with ASD translates to those most affected by the disorder and how future work can strive to obtain meaningful data from low-functioning individuals with ASD. Here we set the stage for detailed characterisation of these processes and their interactions, to in turn understand potential roadblocks to the typical age-related changes in multisensory processing in ASD, and some of the factors that might contribute to sensory reactivity in both high- and low-functioning individuals within this group.
Methods
The present study is based on reanalysis of a large body of data collected as part of several previously published studies36,38,108, as well as unpublished data from more than 200 additional participants.
Participants
A total of 411 individuals participated in the experiment. The data of 47 participants (11.4% of the total sample, 34 ASD) were excluded from all analyses based on the following criteria: (1) they did not fall within the desired age range of 6–40 years (n = 9), (2) their performance IQ was below 80 or not recorded (n = 24), (3) their detection accuracy was <3 SDs below the sample’s mean (<65%, n = 6), (4) they had an excessive number of false alarms (>65%, n = 4), (5) they had a disproportionate number of hits on visual trials (excessive eye-closure) or on audio trials (not listening; <50% of other modality, n = 3), (6) the ISIs used were not within the desired range of 1–3 seconds (n = 1), or (7) they had less than 20 RTs per condition (n = 1; this can bias median RT estimates109,110 as well as race model estimates111). Of the remaining 364 participants, 225 met criteria for NT (age range: 6–36 years; 115 females) and 139 had a diagnosis of ASD (age range: 6–39 years; 34 females). For analysis purposes, age was treated either as a continuous variable or participants were cross-sectioned into four age groups: children (6–9 years), pre-adolescents (10–12 years), adolescents (13–17 years), adults (18–40 years). Mean age was not statistically different between NT and ASD participants in any of the four age groups (statistics reported in Supplementary Table 1). Participant demographics are presented in Table 1.
Individuals were excluded from participating in the experiment if they had a history of seizures or head trauma, or a known genetic disorder. Additionally, NT participants were excluded if they had a history of psychiatric, educational, attentional or other developmental difficulties (as assessed by a history questionnaire), a biological first-degree relative with a known developmental disorder, or if they or their legal guardians endorsed six or more items of inattention or hyperactivity on a DSM-IV checklist for attention deficit disorder. For the vast majority of participants, diagnoses of ASD were obtained by a trained clinical psychologist using the Autism Diagnostic Interview-Revised112 and the Autism Diagnostic Observation Schedule (ADOS)113. Diagnoses of the remaining individuals were made by a licensed clinical psychologist external to this study using the Diagnostic Criteria for Autistic Disorder from the DSM-IV-TR114. For more details regarding sub-phenotyping, medication and ethnic demographics, please refer to previous studies36,108.
Intelligence quotients for performance (PIQ), verbal (VIQ) and full-scale (FSIQ) were assessed in 65% of NT participants and 97% of ASD participants using the Wechsler Abbreviated Scales of Intelligence (WASI)115. We found no evidence to suggest that mean PIQ was statistically different between NT and ASD participants in any of the four age groups (statistics described in Supplementary Table 1). To ensure rigorous between-group comparisons, individuals within each subgroup were matched for sex, age and PIQ using a k-nearest neighbour search. The descriptive statistics for each of the subgroups are summarised in Table 1. Participants were formally screened for normal or corrected-to-normal vision using a Snellen eye test chart and audiometric threshold evaluation confirmed that all participants had within-normal-limits hearing. All procedures were approved by the institutional review boards of the City College of New York, Albert Einstein College of Medicine, and University of Rochester School of Medicine and Dentistry. All participants or legal guardians of participants provided written informed consent in accordance with the tenets of the 1964 Declaration of Helsinki.
Stimuli and procedure
The stimulus materials were identical to those described previously38. In brief, visual (V) stimuli consisted of a red disc (diameter: 3.2 cm; duration: 60 ms), located 0.4 cm above a central fixation crosshair on a black background. The disc subtended visual angles of 1.5° vertically and horizontally and the bottom of the disc subtended 0.9° vertically above the crosshair. Auditory (A) stimuli consisted of a 1-kHz pure tone, sampled at 44.1 kHz (duration: 60 ms; rise/fall time: 5 ms; intensity: 75 dB SPL). Audiovisual (AV) stimuli consisted of the combined simultaneous pairing of the auditory and visual stimuli described above (see Supplementary Fig. 1).
Participants performed a speeded bisensory detection task on a computer and were seated 122 cm from the visual display in a dimly lit, sound-attenuated booth. RTs were recorded during the simultaneous recording of electrophysiological (EEG) data, however, the EEG data are not reported in this study (please refer to previous studies36,38,108). To reduce predictability, the stimuli were presented in a completely randomised order with equal probability and the interstimulus interval (ISI) was randomly jittered between 1000 and 3000 ms according to a uniform, square-wave distribution. The task did not involve any catch trials or background noise/distractors. Stimulus presentation was controlled using Presentation® software (Neurobehavioral Systems, Inc., Berkeley, CA). Auditory stimuli were delivered binaurally at an intensity of 75 dB SPL via a single, centrally located loudspeaker (JBL Duet Speaker System, Harman Multimedia). Visual stimuli were presented at a resolution of 1280 × 1024 pixels on a 17-inch Flat Panel LCD monitor (Dell Ultrasharp 1704FTP). The auditory and visual stimuli were presented in close spatial proximity, with the speaker placed atop the monitor and aligned vertically to the visual stimulus. Participants were instructed to press a button on a response pad (Logitech Wingman Precision Gamepad) with their right thumb as soon as they perceived any of the three stimuli. Analogue triggers indicating the latencies of stimulus onsets and button presses were sent to the acquisition PC via Presentation® and stored digitally at a sampling rate of 512 Hz in a separate channel of the EEG data file using ActiView software (BioSemi™, Amsterdam, The Netherlands). Stimuli were presented in blocks of ~100 trials and participants typically completed 6–10 blocks in total.
Data analysis
To account for false alarms and excessive button pressing, detection accuracy was quantified by the harmonic mean of precision and recall, i.e., F1 score116. Response times were measured relative to the onset time of the preceding stimulus and analysed in MATLAB (The MathWorks, Inc., Natick, MA). Responses were excluded from all analyses if there was more than one response within a given trial (double-presses), they occurred within the first 3 trials of a block (considered training) or the preceding ISI was not between 1000 and 3000 ms (due to system errors). An outlier correction procedure was performed before the main RT analyses. First, RTs that did not fall within 100 and 2000 ms post-stimulus were removed. On average, fast outliers (<100 ms, considered anticipatory responses) made up 0.7% (±0.9) of trials and slow outliers (>2000 ms, considered misses) made up 0.4% (±0.6) of trials. Second, RTs outside the middle 95th percentile (2.5–97.5) of their respective conditions were removed. While not all RTs outside of this range are necessarily outliers, those within this range are most likely to come from the cognitive processes under consideration117. This approach minimises the impact of outliers with only negligible truncation biases118 and captures the range of RTs at an individual-participant level, an important factor when dealing with significant inter-subject variability.
Data analysis was conducted on the whole RT distribution by splitting it into discrete quantiles67,68. RTs were organised into 20 linearly-spaced quantiles between the 2.5 and 97.5 cutoffs used for outlier correction. Because outlier correction was performed separately for each condition, the lowest 2.5 and highest 97.5 percentiles were used for all three conditions to maintain the relationship between them. Cumulative distribution functions (CDFs) were obtained by calculating the cumulative probability of RTs occurring below time t of each quantile given a signal X, \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{X}}}}}}}\le t|{{{{{\rm{X}}}}}}\right)\). The resulting CDFs can be represented as a function of time t or quantiles in the unit interval [0, 1]. Here, we use quantiles for consistency across different analyses and in keeping with previous work119. We chose to quantify the CDFs as cumulative probabilities (the “vertical” method), as opposed to taking the additional step to interpolate them to reaction times (the “horizontal” method). This is particularly important when dealing with a highly heterogeneous population, as is the case here, because a speed-up indexed by the proportion of faster RTs can reveal relative effects that could otherwise be obscured when measured in absolute time (seconds). Note, this approach does not require there to be an equal number of RTs in each condition120.
Race model analysis
To obtain quantitative predictions of statistical facilitation, we used Raab’s independent race model4. Let \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}}\le t|{{{{{\rm{AV}}}}}}\right)\) and \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{V}}}}}}}\le t|{{{{{\rm{AV}}}}}}\right)\) represent the CDFs of the A and V components of an AV stimulus, respectively. Assuming the RT distributions of the A and V components overlap, the probability of either triggering a response can be represented using probability summation. To solve this analytically, we need to make two assumptions: (1) RTs to the A and V components of the AV signal follow the same distributions as the RTs to the unisensory A and V signals, such that \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}}\le t|{{{{{\rm{AV}}}}}}\right)=P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}}\le t|{{{{{\rm{A}}}}}}\right)\) and \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{V}}}}}}}\le t|{{{{{\rm{AV}}}}}}\right)=P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{V}}}}}}}\le t|{{{{{\rm{V}}}}}}\right)\), i.e., context invariance107,121,122; (2) RTs to the A and V components of the AV signal are statistically independent, such that their joint probability \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}\cap {{{{{\rm{V}}}}}}}\le t|{{{{{\rm{AV}}}}}}\right)\) can be calculated by the product of \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}}\le t|{{{{{\rm{A}}}}}}\right)\) and \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{V}}}}}}}\le t|{{{{{\rm{V}}}}}}\right)\)123. Simplifying \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}\cup {{{{{\rm{V}}}}}}}\le t|{{{{{\rm{AV}}}}}}\right)\) to FA∪V (t), \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{A}}}}}}}\le t|{{{{{\rm{A}}}}}}\right)\) to FA (t) and \(P\left({{{{{{\rm{RT}}}}}}}_{{{{{{\rm{V}}}}}}}\le t|{{{{{\rm{V}}}}}}\right)\) to FV (t), the race model can be represented as:
Note, the joint probability term is often omitted from Eq. 1 to produce an upper bound of statistical facilitation known as Miller’s bound or the race model inequality5, as the assumption of channel independence is poorly motivated; it is likely that responses to signals on different sensory channels compete for resources5,58,124,125,126. Assuming that the allocation of such resources is partially determined by the modality of the previous trial5, we separated the unisensory RTs by preceding sensory modality (A, V, AV) and computed individual race models per condition before averaging them:
where m is the preceding modality. This method captured some of the dependency between RTs to signals on different channels, resulting in an estimate of statistical facilitation that was less conservative on 90% of quantiles over the 364 participants (two-tailed permutation tests with tmax correction).
Multisensory benefits were quantified by the area between the CDFs in the multisensory condition and the most effective unisensory condition54. First, we computed the multisensory benefit predicted by the race model (Fig. 1b, left):
where the integral is taken over every quantile t from 0 to 1. \({\max }\left[{F}_{{{{{{\rm{A}}}}}}}\left(t\right),{F}_{{{{{{\rm{V}}}}}}}\left(t\right)\right]\) represents a lower bound on statistical facilitation, known as Grice’s bound127. Similarly, we computed empirical benefits based on the observed multisensory RTs (Fig. 1b, right):
To determine whether the empirical multisensory benefits exceeded the statistical facilitation predicted by the race model, we computed the difference between the CDFs of the multisensory condition and the race model at every quantile6. Positive deviations indicate quantiles where multisensory RTs were faster than predicted, i.e., facilitation. To obtain an overall index of multisensory gain, we calculated the area under the curve (AUC) by taking the integral over every quantile as before (Fig. 3a):
While it is common practice to interpret only the positive AUC as an index of multisensory gain12,128,129, Eq. 6 is equal to the sum of the positive and negative AUC130,131. This is mathematically equivalent to the difference between predicted and empirical benefits and represents the overall multisensory gain across the whole RT distribution. Qualitatively, this is equivalent to using only the positive AUC132, because the positive and negative AUCs are inversely proportional (see Fig. 3b). Moreover, many younger participants in this study did not exhibit a multisensory benefit that exceeded statistical facilitation, rendering a statistical analysis based on only the positive AUC less powerful. All race model analyses were conducted using the RaceModel (v1.0) toolbox (https://github.com/mickcrosse/RaceModel).
Modelling multisensory competition
To determine whether subject behaviour reflected multisensory processing of a facilitative or competitive nature, we attempted to model the logical coupling between parallel decision processes for both cases. Facilitation would likely follow the predictions of the race model as it exactly matches the task demands of a bisensory detection task, i.e., a logical disjunction59. Competition, on the other hand, would likely follow the predictions of the stronger sensory modality because of cross-sensory inhibitory mechanisms53, regardless of inherent processing speeds on either channel. Such biased channel coupling could manifest as a result of two potential scenarios: (1) the dominant sensory modality would generally prevail due to an inherent biological preference, thus triggering the response or (2) the modality of the previous trial would generally prevail due to pre-allocated attentional resources, thus triggering the response. The first scenario was modelled with a bias towards either the auditory modality (Model 1A) or the visual modality (Model 1V) as follows:
where m is the preceding modality (A, V, AV) and b is the modality that the system is biased towards (A or V). The second scenario was modelled with a bias towards the previous modality, except when the previous trial was an AV stimulus, where it was biased towards either the auditory modality (Model 2A) or the visual modality (Model 2V):
The models were used to obtain alternative measures of predicted benefits and assessed based on how accurately they predicted empirical benefits. To examine potential age-related transitions in multisensory processing, we parametrically varied the probability of such processing being facilitative (race model) or competitive (bias model) as follows:
where Fib (t) is the bias model and p is the probability of it triggering a response. When p = 0, processing is purely facilitative (race model), and when p = 1, processing is purely competitive (bias model). In addition, Model 1 was used to examine age-related changes in sensory dominance which are likely to have a significant impact during competitive multisensory processing.
Quantifying modality switch effects
To examine modality switch effects (MSEs), RTs were separated into trials preceded by the same modality (repeat trials) and those preceded by a different modality (switch trials). Unisensory trials preceded by multisensory trials (AV→A, AV→V) were excluded from this analysis as they were considered neither switches nor repeats (repeat trials: A→A, V→V, AV→AV; switch trials: V→A, A→V, V→AV, A→AV). Separate CDFs were obtained for switch and repeat trials within each condition. Trials belonging to the two multisensory switch conditions (A→AV, V→AV) were pooled to produce one multisensory switch condition (V/A→AV). MSEs were quantified by the area between the CDFs of the switch and repeat trials:
To examine the impact of switching sensory modality on the observed multisensory gain, separate tests of the race model were performed for switch and repeat trials.
Modelling channel dependency and RT variability
Seminal work by Otto and Mamassian64 has demonstrated that the basic architecture of race models can be used to accurately predict empirical multisensory RT data by including two additional free parameters to account for (1) the additional variability or noise η typically observed in empirical multisensory RTs relative to standard race model predictions, and (2) the correlation ρ between processing speeds on different sensory channels. Conceptually, Miller’s and Grice’s bounds assume a perfect negative and positive correlation respectively, whereas Raab’s model assumes zero correlation (i.e., independence). Otto’s model on the other hand makes no such assumptions, allowing the correlation parameter ρ to vary in a way that optimises the model’s fit to the empirical data. Using the RSE-box133 (v1.0) toolbox (https://github.com/tomotto/RSE-box), we examined the values of ρ and η that optimised the model fit for each participant. Gaussian functions were fit to the reciprocal of the unisensory RT distributions via the LATER model approach134, which assumes that the reciprocals of the RT distributions are normally distributed with mean μ and SD σ (see Fig. 7b). These parameters were then used to generate the probability density function of the maximum distribution \({f}_{A\cup V}(x)={f}_{A}(-x)+{f}_{V}(-x)\) (refs. 64,135).
Statistics and reproducibility
A linear mixed-effects model was used to determine which parameters influenced RTs, fit using the maximum likelihood criterion. Single-trial RTs were the continuous numeric dependent variable. Diagnosis was a contrast-coded fixed factor (NT, ASD), age was a continuous numeric fixed factor (6–40 years), and condition was a multi-level nominal fixed factor (AV, A, V). Subjects were included as a random factor, along with by-subject slope adjustments for condition66. ISI was included as another random factor, as well as preceding modality with slope adjustments for condition. Subsequent analyses employing standard linear models coded fixed effects as above.
A mediation analysis136 was conducted using the M3 Mediation (v1.0) toolbox (https://github.com/canlab/MediationToolbox) to establish whether the relationship between participants’ age and multisensory gain was mediated by a direct effect of age on MSE. For this analysis, MSEs were averaged across the two unisensory conditions (V→A, A→V), as we hypothesized that it was a slowing of unisensory RTs that was the cause of the observed RSE. We constructed a three-variable mediation model with age as the causal variable, gain as the outcome variable and MSE as the mediating variable (see Fig. 8c). All 3 variables were z-scored prior to conducting the analysis. For MSE to be considered a mediator, the following criteria must be met based on three separate regressions: (1) the causal variable must affect the outcome, (2) the causal variable must affect the mediator, and 3) the mediator must affect the outcome but the causal variable must either no longer affect the outcome (full mediation) or at least weaken the effect (partial mediation). Significance and SE of the associated path coefficients were bootstrapped (10,000 samples) and adjusted using the bias-corrected and accelerated percentile method73.
Significance testing was conducted via nonparametric permutation tests (10,000 permutations) using the PERMUTOOLS (v1.0) toolbox (https://github.com/mickcrosse/PERMUTOOLS). Multivariate tests were adjusted to control for family-wise error rate using the tmax correction method137,138,139. This method has been shown to control for Type 1 error at a desired level when performing tests of the race model at multiple quantiles69. Moving mean analyses were corrected for multiple comparisons using the false discovery rate (FDR) control procedure140. Bias-corrected effect sizes (i.e., Hedge’s g141) were calculated using the MES (v1.0) toolbox141 (https://github.com/hhentschke/measures-of-effect-size-toolbox). All confidence intervals were bootstrapped (10,000 samples) at the 95% confidence level and adjusted using the bias-corrected and accelerated percentile method142. Bayes factor (BF01) analyses were conducted using the Bayes Factor (v2.0) toolbox (https://github.com/klabhub/bayesFactor) to test for the absence of an effect by quantifying the relative likelihood of the data under the null versus the alternative hypothesis. Effect sizes were assumed to follow a Cauchy prior distribution with a scale parameter of 1 and were interpreted using the standard convention139.
Reporting summary
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
Data availability
The data underlying figures and corresponding demographic information that support the findings of the study are available to download in anonymised form from Figshare (https://figshare.com/s/d349bd507d419db8077f).
Code availability
The custom MATLAB scripts used to run the main and supplementary analyses of the study are available to download from GitHub (https://github.com/mickcrosse/AVSRT-Autism). Dependencies for conducting the main modelling and statistical analyses have been packaged by the authors as the abovementioned RaceModel and PERMUTOOLS toolboxes, respectively, and are available to download from GitHub.
References
Todd, J. W. Reaction to Multiple Stimuli (Science Press, 1912).
Hershenson, M. Reaction time as a measure of intersensory facilitation. J. Exp. Psychol. 63, 289 (1962).
Kinchla, R. Detecting target elements in multielement arrays: a confusability model. Percept. Psychophys. 15, 149–158 (1974).
Raab, D. H. Division of psychology: statistical facilitation of simple reaction times. Trans. N. Y. Acad. Sci. 24, 574–590 (1962).
Miller, J. Divided attention: evidence for coactivation with redundant signals. Cogn. Psychol. 14, 247–279 (1982).
Molholm, S. et al. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn. Brain Res. 14, 115–128 (2002).
Harrington, L. & Peck, C. Spatial disparity affects visual-auditory interactions in human sensorimotor processing. Exp. Brain Res. 122, 247–252 (1998).
Murray, M. M. et al. Grabbing your ear: rapid auditory–somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb. Cortex 15, 963–974 (2004).
Mahoney, J. R. et al. Keeping in touch with the visual system: spatial alignment and multisensory integration of visual-somatosensory inputs. Front. Psychol. 6, 1068 (2015).
Gielen, S. C., Schmidt, R. A. & Van Den Heuvel, P. J. On the nature of intersensory facilitation of reaction time. Percept. Psychophys. 34, 161–168 (1983).
Diederich, A. & Colonius, H. Intersensory facilitation in the motor component? Psychol. Res. 49, 23–29 (1987).
Miller, J. Timecourse of coactivation in bimodal divided attention. Percept. Psychophys. 40, 331–343 (1986).
Mégevand, P., Molholm, S., Nayak, A. & Foxe, J. J. Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS ONE 8, e71608 (2013).
Stevenson, R. A. et al. Inverse effectiveness and multisensory interactions in visual event-related potentials with audiovisual speech. Brain Topogr. 25, 308–326 (2012).
Innes, B. R. & Otto, T. U. A comparative analysis of response times shows that multisensory benefits and interactions are not equivalent. Sci. Rep. https://doi.org/10.1038/s41598-019-39924-6 (2019).
Sumby, W. H. & Pollack, I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26, 212–215 (1954).
Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C. & Foxe, J. J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environment. Cereb. Cortex 17, 1147–1153 (2007).
Crosse, M. J., Di Liberto, G. M. & Lalor, E. C. Eye can hear clearly now: inverse effectiveness in natural audiovisual speech processing relies on long-term crossmodal temporal integration. J. Neurosci. 36, 9888–9895 (2016).
Stevenson, R. A. & James, T. W. Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44, 1210–1223 (2009).
de Gelder, B., Vroomen, J. & Van der Heide, L. Face recognition and lip-reading in autism. Eur. J. Cogn. Psychol. 3, 69–86 (1991).
Smith, E. G. & Bennetto, L. Audiovisual speech integration and lipreading in autism. J. Child Psychol. Psychiatry 48, 813–821 (2007).
Bebko, J. M., Schroeder, J. H. & Weiss, J. A. The McGurk effect in children with autism and Asperger syndrome. Autism Res. 7, 50–59 (2014).
Silverman, L. B., Bennetto, L., Campana, E. & Tanenhaus, M. K. Speech-and-gesture integration in high functioning autism. Cognition 115, 380–393 (2010).
Irwin, J. R., Tornatore, L. A., Brancazio, L. & Whalen, D. Can children with autism spectrum disorders “hear” a speaking face? Child Dev. 82, 1397–1403 (2011).
Foxe, J. J. et al. Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cereb. Cortex 25, 298–312 (2015).
Stevenson, R. A. et al. Multisensory temporal integration in autism spectrum disorders. J. Neurosci. 34, 691–697 (2014).
Stevenson, R. A. et al. Evidence for diminished multisensory integration in autism spectrum disorders. J. Autism Dev. Disord. 44, 3161–3167 (2014).
Iarocci, G. & McDonald, J. Sensory integration and the perceptual experience of persons with autism. J. Autism Dev. Disord. 36, 77–90 (2006).
Foxe, J. J. & Molholm, S. Ten years at the multisensory forum: musings on the evolution of a field. Brain Topogr. 21, 149–154 (2009).
Ayres, A. J. & Tickle, L. S. Hyper-responsivity to touch and vestibular stimuli as a predictor of positive response to sensory integration procedures by autistic children. Am. J. Occup. Ther. 34, 375–381 (1980).
Martineau, J. et al. Electrophysiological evidence of different abilities to form cross-modal associations in children with autistic behavior. Electroencephalogr. Clin. Neurophysiol. 82, 60–66 (1992).
Beker, S., Foxe, J. J. & Molholm, S. Ripe for solution: delayed development of multisensory processing in autism and its remediation. Neurosci. Biobehav. Rev. 84, 182–192 (2018).
Stevenson, R. A. et al. The cascading influence of multisensory processing on speech perception in autism. Autism 22, 609–624 (2018).
Baum, S. H., Stevenson, R. A. & Wallace, M. T. Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. Prog. Neurobiol. 134, 140–160 (2015).
Taylor, N., Isaac, C. & Milne, E. A comparison of the development of audiovisual integration in children with autism spectrum disorders and typically developing children. J. Autism Dev. Disord. 40, 1403–1411 (2010).
Brandwein, A. B. et al. The development of multisensory integration in high-functioning autism: high-density electrical mapping and psychophysical measures reveal impairments in the processing of audiovisual inputs. Cereb. Cortex 23, 1329–1341 (2013).
Cuppini, C. et al. A computational analysis of neural mechanisms underlying the maturation of multisensory speech integration in neurotypical children and those on the autism spectrum. Front. Hum. Neurosci. 11, 518 (2017).
Brandwein, A. B. et al. The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. Cereb. Cortex 21, 1042–1055 (2011).
Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444 (1997).
Wallace, M. T. & Stein, B. E. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21, 8886–8894 (2001).
Wallace, M. T. & Stein, B. E. Early experience determines how the senses will interact. J. Neurophysiol. 97, 921–926 (2007).
Wallace, M. T., Perrault, T. J., Hairston, W. D. & Stein, B. E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24, 9580–9584 (2004).
Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. Noise‐rearing disrupts the maturation of multisensory integration. Eur. J. Neurosci. 39, 602–613 (2014).
Yu, L., Rowland, B. A. & Stein, B. E. Initiating the development of multisensory integration by manipulating sensory experience. J. Neurosci. 30, 4904–4913 (2010).
Yu, L., Cuppini, C., Xu, J., Rowland, B. A. & Stein, B. E. Cross-modal competition: the default computation for multisensory processing. J. Neurosci. 39, 1374–1385 (2019).
Cuppini, C., Stein, B. E. & Rowland, B. A. Development of the mechanisms governing midbrain multisensory integration. J. Neurosci. 38, 3453–3465 (2018).
Cuppini, C., Stein, B. E., Rowland, B. A., Magosso, E. & Ursino, M. A computational study of multisensory maturation in the superior colliculus (SC). Exp. Brain Res. 213, 341–349 (2011).
Cuppini, C., Magosso, E., Rowland, B., Stein, B. & Ursino, M. Hebbian mechanisms help explain development of multisensory integration in the superior colliculus: a neural network model. Biol. Cybern. 106, 691–713 (2012).
Gori, M., Del Viva, M., Sandini, G. & Burr, D. C. Young children do not integrate visual and haptic form information. Curr. Biol. 18, 694–698 (2008).
Barutchu, A., Crewther, D. P. & Crewther, S. G. The race that precedes coactivation: development of multisensory facilitation in children. Dev. Sci. 12, 464–473 (2009).
Sinnett, S., Soto-Faraco, S. & Spence, C. The co-occurrence of multisensory competition and facilitation. Acta Psychol. 128, 153–161 (2008).
Molholm, S., Ritter, W., Javitt, D. C. & Foxe, J. J. Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb. Cortex 14, 452–465 (2004).
Cuppini, C. et al. Cross-sensory Inhibition Or Unisensory Facilitation: The Neural Architecture Of Modality Switch Effects 99, 102438 (2020).
Otto, T. U., Dassy, B. & Mamassian, P. Principles of multisensory behavior. J. Neurosci. 33, 7463–7474 (2013).
Wundt, W. M. Grundzüge der physiologischen Psychologie. 4 edn, Vol. 2 (W. Engelmann, 1893).
Spence, C., Nicholls, M. E. & Driver, J. The cost of expecting events in the wrong sensory modality. Percept. Psychophys. 63, 330–336 (2001).
Sutton, S., Hakerem, G., Zubin, J. & Portnoy, M. The effect of shift of sensory modality on serial reaction-time: A comparison of schizophrenics and normals. Am. J. Psychol. 74, 224–232 (1961).
Gondan, M. & Minakata, K. A tutorial on testing the race model inequality. Atten. Percept. Psychophys. 78, 723–735 (2016).
Otto, T. U. & Mamassian, P. Multisensory decisions: the test of a race model, its logic, and power. Multisens. Res. 30, 1–24 (2017).
Gondan, M., Lange, K., Rösler, F. & Röder, B. The redundant target effect is affected by modality switch costs. Psychonomic Bull. Rev. 11, 307–313 (2004).
Van der Stoep, N., Van der Stigchel, S. & Nijboer, T. Exogenous spatial attention decreases audiovisual integration. Atten. Percept. Psychophys. 77, 464–482 (2015).
Shaw, L. H. et al. Operating in a multisensory context: assessing the interplay between multisensory reaction time facilitation and inter-sensory task-switching effects. Neuroscience 436, 122–135 (2020).
Williams, D. L., Goldstein, G. & Minshew, N. J. The modality shift experiment in adults and children with high functioning autism. J. Autism Dev. Disord. 43, 794–806 (2013).
Otto, T. U. & Mamassian, P. Noise and correlations in parallel perceptual decision making. Curr. Biol. 22, 1391–1396 (2012).
Murphy, J. W., Foxe, J. J., Peters, J. B. & Molholm, S. Susceptibility to distraction in autism spectrum disorder: Probing the integrity of oscillatory alpha‐band suppression mechanisms. Autism Res. 7, 442–458 (2014).
Barr, D. J., Levy, R., Scheepers, C. & Tily, H. J. Random effects structure for confirmatory hypothesis testing: Keep it maximal. J. Mem. Lang. 68, 255–278 (2013).
Whelan, R. Effective analysis of reaction time data. Psychol. Rec. 58, 475 (2008).
Ratcliff, R. Group reaction time distributions and an analysis of distribution statistics. Psychol. Bull. 86, 446 (1979).
Gondan, M. A permutation test for the race model inequality. Behav. Res. Methods 42, 23–28 (2010).
Bliss, C. I. Statistics in Biology. Vol. 1 (McGraw-Hill, 1967).
Bland, J. M. & Altman, D. G. Calculating correlation coefficients with repeated observations: part 2—correlation between subjects. BMJ 310, 633 (1995).
Colavita, F. B. Human sensory dominance. Percept. Psychophys. 16, 409–412 (1974).
Wager, T. D., Davidson, M. L., Hughes, B. L., Lindquist, M. A. & Ochsner, K. N. Prefrontal-subcortical pathways mediating successful emotion regulation. Neuron 59, 1037–1050 (2008).
Ross, L. A. et al. The development of multisensory speech perception continues into the late childhood years. Eur. J. Neurosci. 33, 2329–2337 (2011).
Chandrasekaran, C. Computational principles and models of multisensory integration. Curr. Opin. Neurobiol. 43, 25–34 (2017).
Mercier, M. R. et al. Auditory-driven phase reset in visual cortex: human electrocorticography reveals mechanisms of early multisensory integration. Neuroimage 79, 19–29 (2013).
Mercier, M. R. et al. Neuro-oscillatory phase alignment drives speeded multisensory response times: an electro-corticographic investigation. J. Neurosci. 35, 8546–8557 (2015).
Hickok, G. & Poeppel, D. The cortical organization of speech processing. Nat. Rev. Neurosci. 8, 393–402 (2007).
Peelle, J. E. The neural basis for auditory and audiovisual speech perception. In The Routledge Handbook of Phonetics (eds Katz, W. F. & Assmann, P. F.), pp. 193–216 (Routledge, 2019)
Mégevand, P. et al. Phase resetting in human auditory cortex to visual speech. bioRxiv https://doi.org/10.1101/405597 (2018).
Tye-Murray, N., Sommers, M. & Spehar, B. Auditory and visual lexical neighborhoods in audiovisual speech perception. Trends Amplif. 11, 233–241 (2007).
Peelle, J. E. & Sommers, M. S. Prediction and constraint in audiovisual speech perception. Cortex 68, 169–181 (2015).
Lewkowicz, D. J. Sensory dominance in infants: I. Six-month-old infants’ response to auditory-visual compounds. Dev. Psychol. 24, 155 (1988).
Lewkowicz, D. J. Sensory dominance in infants: II. Ten-month-old infants’ response to auditory-visual compounds. Dev. Psychol. 24, 172 (1988).
Robinson, C. W. & Sloutsky, V. M. Auditory dominance and its change in the course of development. Child Dev. 75, 1387–1401 (2004).
Nava, E. & Pavani, F. Changes in sensory dominance during childhood: Converging evidence from the Colavita effect and the sound‐induced flash illusion. Child Dev. 84, 604–616 (2013).
Hirst, R. J., Cragg, L. & Allen, H. A. Vision dominates audition in adults but not children: a meta-analysis of the Colavita effect. Neurosci. Biobehav. Rev. 94, 286–301 (2018).
O’Connor, N. & Hermelin, B. Sensory dominance: in autistic imbecile children and controls. Arch. Gen. Psychiatry 12, 99–103 (1965).
Hermelin, B. & O’Connor, N. Effects of sensory input and sensory domiinance of severely disturbed autistic children and on subnormal controls. Br. J. Psychol. 55, 201–206 (1964).
Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S. & Puce, A. Neuronal oscillations and visual amplification of speech. Trends Cogn. Sci. 12, 106–113 (2008).
Lakatos, P., Chen, C.-M., O’Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292 (2007).
Fiebelkorn, I. C. et al. Ready, set, reset: stimulus-locked periodicity in behavioral performance demonstrates the consequences of cross-sensory phase reset. J. Neurosci. 31, 9971–9981 (2011).
Fiebelkorn, I. C. et al. Cortical cross-frequency coupling predicts perceptual outcomes. Neuroimage 69, 126–137 (2013).
Senkowski, D., Schneider, T. R., Foxe, J. J. & Engel, A. K. Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci. 31, 401–409 (2008).
Zeng, K. et al. Disrupted brain network in children with autism spectrum disorder. Sci. Rep. 7, 16253 (2017).
Arnold Anteraper, S. et al. Disrupted cerebro-cerebellar intrinsic functional connectivity in young adults with high-functioning autism spectrum disorder: a data-driven, whole-brain, high temporal resolution fMRI study. Brain Connect. 9, 48–59 (2018).
Vasa, R. A., Mostofsky, S. H. & Ewen, J. B. The disrupted connectivity hypothesis of autism spectrum disorders: time for the next phase in research. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 1, 245–252 (2016).
Crosse, M. J., Cuppini, C., Foxe, J. J. & Molholm, S. In Computational and Systems Neuroscience (COSYNE) 2019 (COSYNE, 2019).
Zubin, J. Problem of attention in schizophrenia. Experimental approaches to psychopathology, 139-166 (1975).
Lieder, I. et al. Perceptual bias reveals slow-updating in autism and fast-forgetting in dyslexia. Nat. Neurosci. 22, 256–264 (2019).
Lawson, R. P., Mathys, C. & Rees, G. Adults with autism overestimate the volatility of the sensory environment. Nat. Neurosci. 20, 1293 (2017).
Maekawa, T. et al. Top-down and bottom-up visual information processing of non-social stimuli in high-functioning autism spectrum disorder. Res. Autism Spectr. Disord. 5, 201–209 (2011).
Shaw, L. H. et al. Operating in a multisensory context: assessing the interplay between multisensory reaction time facilitation and inter-sensory task-switching effects. Neuroscience 436, 122–135 (2020).
Crosse, M. J., Butler, J. S. & Lalor, E. C. Congruent visual speech enhances cortical entrainment to continuous auditory speech in noise-free conditions. J. Neurosci. 35, 14195–14204 (2015).
Wallace, M. T., Wilkinson, L. K. & Stein, B. E. Representation and integration of multiple sensory inputs in primate superior colliculus. J. Neurophysiol. 76, 1246–1266 (1996).
Senkowski, D., Saint-Amour, D., Höfle, M. & Foxe, J. J. Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. Neuroimage 56, 2200–2208 (2011).
Miller, J. Statistical facilitation and the redundant signals effect: What are race and coactivation models? Atten. Percept. Psychophys. 78, 516–519 (2016).
Brandwein, A. B. et al. Neurophysiological indices of atypical auditory processing and multisensory integration are associated with symptom severity in autism. J. Autism Dev. Disord. 45, 230–244 (2015).
Miller, J. A warning about median reaction time. J. Exp. Psychol. Hum. Percept. Perform. 14, 539 (1988).
Miller, J. Reaction time analysis with outlier exclusion: bias varies with sample size. Q. J. Exp. Psychol. 43, 907–912 (1991).
Kiesel, A., Miller, J. & Ulrich, R. Systematic biases and Type I error accumulation in tests of the race model inequality. Behav. Res. Methods 39, 539–551 (2007).
Lord, C., Rutter, M. & Le Couteur, A. Autism Diagnostic Interview-Revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J. Autism Dev. Disord. 24, 659–685 (1994).
Lord, C. et al. The Autism Diagnostic Observation Schedule—Generic: a standard measure of social and communication deficits associated with the spectrum of autism. J. Autism Dev. Disord. 30, 205–223 (2000).
Cooper, J. Diagnostic and statistical manual, 4th edn, Text Revision (DSM-IV-TR) (American Psychiatric Association, 2000).
Stano, J. Wechsler Abbreviated Scale Of Intelligence (The Psychological Corporation, 1999).
Van Rijsbergen, C. J. Information Retrieval (Butterworth-Heinemann, 1979).
Ratcliff, R. Methods for dealing with reaction time outliers. Psychol. Bull. 114, 510 (1993).
Ulrich, R. & Miller, J. Effects of truncation on reaction time analysis. J. Exp. Psychol. Gen. 123, 34 (1994).
Molholm, S. et al. Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. J. Neurophysiol. 96, 721–729 (2006).
Ulrich, R., Miller, J. & Schröter, H. Testing the race model inequality: an algorithm and computer programs. Behav. Res. Methods 39, 291–302 (2007).
Luce, R. D. Response Times: Their Role In Inferring Elementary Mental Organization (Oxford University Press on Demand, 1986).
Ashby, F. G. & Townsend, J. T. Varieties of perceptual independence. Psychol. Rev. 93, 154 (1986).
Meijers, L. & Eijkman, E. Distributions of simple RT with single and double stimuli. Percept. Psychophys. 22, 41–48 (1977).
Colonius, H. Possibly dependent probability summation of reaction time. J. Math. Psychol. 34, 253–275 (1990).
Colonius, H. Measuring channel dependence in separate activation models. Percept. Psychophys. 40, 251–255 (1986).
Miller, J. Multidimensional same–different judgments: evidence against independent comparisons of dimensions. J. Exp. Psychol. Hum. Percept. Perform. 4, 411 (1978).
Grice, G. R., Canham, L. & Gwynne, J. W. Absence of a redundant-signals effect in a reaction time task with divided attention. Percept. Psychophys. 36, 565–570 (1984).
Nozawa, G., Reuter-Lorenz, P. A. & Hughes, H. Parallel and serial processes in the human oculomotor system: bimodal integration and express saccades. Biol. Cybern. 72, 19–34 (1994).
Hughes, H. C., Nelson, M. D. & Aronchick, D. M. Spatial characteristics of visual-auditory summation in human saccades. Vis. Res. 38, 3955–3963 (1998).
Colonius, H. & Diederich, A. The race model inequality: interpreting a geometric measure of the amount of violation. Psychol. Rev. 113, 148–154 (2006).
Krueger Fister, J., Stevenson, R. A., Nidiffer, A. R., Barnett, Z. P. & Wallace, M. T. Stimulus intensity modulates multisensory temporal processing. Neuropsychologia 88, 92–100 (2016).
Nidiffer, A. R., Stevenson, R. A., Fister, J. K., Barnett, Z. P. & Wallace, M. T. Interactions between space and effectiveness in human multisensory performance. Neuropsychologia 88, 83–91 (2016).
Otto, T. U. RSE-box: an analysis and modelling package to study response times to multiple signals. The Quantitative Methods Psychol. 15, 112–133 (2019).
Noorani, I. & Carpenter, R. The LATER model of reaction time and decision. Neurosci. Biobehav. Rev. 64, 229–251 (2016).
Nadarajah, S. & Kotz, S. Exact distribution of the max/min of two Gaussian random variables. IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 16, 210–212 (2008).
Baron, R. M. & Kenny, D. A. The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. J. Pers. Soc. Psychol. 51, 1173 (1986).
Blair, R. C., Higgins, J. J., Karniski, W. & Kromrey, J. D. A study of multivariate permutation tests which may replace Hotelling’s T2 test in prescribed circumstances. Multivar. Behav. Res. 29, 141–163 (1994).
Westfall, P. H. & Young, S. S. Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment (Wiley Series in Probability and Statistics, 1993).
Schönbrodt, F. D. & Wagenmakers, E.-J. Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bull. Rev. 25, 128–142 (2018).
Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B (Methodol.) 57, 289–300 (1995).
Hedges, L. V. & Olkin, I. Statistical Methods for Meta Analysis. (Academic Press, 1985).
Davison, A. C. & Hinkley, D. V. Bootstrap methods and their application. Vol. 1 (Cambridge University Press, 1997).
Acknowledgements
The authors thank Juliana Bates, Natalie Russo, and the Human Clinical Phenotyping Core (HCP) for their careful clinical and cognitive phenotyping of our ASD cohort, Alice Brandwein who conducted the previous research for which most of these data were collected, Cristiano Cuppini for his helpful comments and discussions, Ana Francisco for her support with statistical analyses, Douwe Horsthuis for his technical support and data collection, and Elise Taverna and Danielle DeMaio for their technical assistance. We also extend our heartfelt gratitude to the participants and their families that have contributed their time to participate in this research. This work was supported by the National Institute of Mental Health of the National Institutes of Health (NIH) under award number R01MH085322 (S.M. and J.J.F.), and the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the NIH under award number U54HD090260, previously P30HD071593 (Rose F. Kennedy Intellectual and Developmental Disabilities Research Center). Participant recruitment and phenotyping for participants at the University of Rochester was conducted through the UR-IDDRC Human Clinical Phenotyping and Recruitment Core, supported by a centre grant from the Eunice Kennedy Shriver National Institute of Child Health and Human Development under the award number P50HD103536 (J.J.F.). Data collection for this project at the University of Rochester was supported though a pilot grant from the Harry T. Mangurian, Jr. Foundation (E.G.F., S.M. and J.J.F.).
Author information
Authors and Affiliations
Contributions
S.M. and J.J.F. designed the original experiment. M.J.C., S.M. and J.J.F. conceived of the current study. M.J.C. analysed the data and produced all illustrations in consultation with S.M. and J.J.F.; M.J.C. wrote the first substantive draft of the manuscript. S.M. and J.J.F. provided editorial input to M.J.C. on multiple subsequent drafts. E.G.F. recruited additional participants for the revised manuscript and K.T. collected the data.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Communications Biology thanks the anonymous reviewers for their contribution to the peer review of this work. Primary Handling Editors: Christian Beste, Karli Montague-Cardoso, and George Inglis.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Crosse, M.J., Foxe, J.J., Tarrit, K. et al. Resolution of impaired multisensory processing in autism and the cost of switching sensory modality. Commun Biol 5, 601 (2022). https://doi.org/10.1038/s42003-022-03519-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s42003-022-03519-1