The capacity to integrate information from different senses is central for coherent perception across the lifespan from infancy onwards. Later in life, multisensory processes are related to cognitive functions, such as speech or social communication. During learning, multisensory processes can in fact enhance subsequent recognition memory for unisensory objects. These benefits can even be predicted; adults’ recognition memory performance is shaped by earlier responses in the same task to multisensory – but not unisensory – information. Everyday environments where learning occurs, such as classrooms, are inherently multisensory in nature. Multisensory processes may therefore scaffold healthy cognitive development. Here, we provide the first evidence of a predictive relationship between multisensory benefits in simple detection and higher-level cognition that is present already in schoolchildren. Multiple regression analyses indicated that the extent to which a child (N = 68; aged 4.5–15years) exhibited multisensory benefits on a simple detection task not only predicted benefits on a continuous recognition task involving naturalistic objects (p = 0.009), even when controlling for age, but also the same relative multisensory benefit also predicted working memory scores (p = 0.023) and fluid intelligence scores (p = 0.033) as measured using age-standardised test batteries. By contrast, gains in unisensory detection did not show significant prediction of any of the above global cognition measures. Our findings show that low-level multisensory processes predict higher-order memory and cognition already during childhood, even if still subject to ongoing maturation. These results call for revision of traditional models of cognitive development (and likely also education) to account for the role of multisensory processing, while also opening exciting opportunities to facilitate early learning through multisensory programs. More generally, these data suggest that a simple detection task could provide direct insights into the integrity of global cognition in schoolchildren and could be further developed as a readily-implemented and cost-effective screening tool for neurodevelopmental disorders, particularly in cases when standard neuropsychological tests are infeasible or unavailable.
When a child wishes to cross the street, simply looking left and right for incoming cars is not always sufficient to make a safe choice. Sensitivity to additional cues, like the noise generated by an approaching car, will also guide their judgement, and may save their life. There are two aspects of this capacity to integrate information from different senses that are likely themselves synergistic. First, multisensory information may accelerate perceptual decision-making and result in faster and more accurate responses (reviewed in1,2,3,4). Second, multisensory information may provide a more efficient means for learning and memory than unisensory stimuli, which in turn can guide future behaviour (reviewed in5,6,7,8). Learning in multisensory contexts is thus of clear adaptive benefit during development and throughout the lifespan, particularly given the fact that multisensory contexts are reflective of naturalistic settings9. It thus logically follows that the gain afforded by multisensory processes may themselves provide a scaffold for improved higher-level cognitive functions such as learning, recognition memory, working memory, and fluid intelligence (among others) (reviewed in10). The aim of the present study was to assess the presence of such relationships in school-aged children.
Considerable research points to a general link between processing speed and measures of intelligence11,12,13,14, in adults as well as in school-aged children15. One potential consideration with that research is that the tasks used to evaluate processing speed were all visual in nature and thus did not assess the contribution of other sensory systems to cognitive abilities, including intelligence. At the same time, it stands to reason that individuals capable of capitalising on situations that improve processing speed (e.g. multisensory contexts) should also demonstrate stronger cognitive abilities; what Rose and colleagues refer to as a “cognitive cascade”2,16.
Longitudinal studies have linked cross-modal pattern matching in infants with their later reading abilities, such as in the seminal work of Birch and Belmont17,18,19 in 220 elementary (5–12 years old) scholars. This link was then extended to infants, including those born prematurely, by Rose and colleagues. An infant’s ability to match information (typically temporal patterns) between the senses is predictive of later reading skills. In particular, matching abilities between the senses has been shown to be a better predictor of reading skills than matching skills for patterns within a given sensory modality20. The capacity to establish sensory-independent or multisensory representations may be a core underlying skill for cognitive functions to develop and thus are indicative of core intelligence.
While the literature in infants and young children appears to support links between multisensory processes and higher-level cognition, establishing these links in school-aged children has proven more elusive. There is evidence that school-aged children (8–12 years-old) do benefit from multisensory compared to unisensory learning contexts, with facilitated later (unisensory) recognition memory21,22. Similar conclusions are garnered from the works of Broadbent and colleagues. These authors found incidental learning to be improved by multisensory cues23, and that retention of category learning over a 24‐hour delay to be significantly higher for multisensory cues than unisensory ones in 5–10 year-old schoolchildren24. This is consistent with literature in adults reporting evidence for links between processes subserving multisensory integration on the one hand and cognitive functions, including recognition memory, on the other hand. For example, Thelen et al.25 showed that individual performance on a continuous object recognition task could be predicted by brain responses to multisensory, but not unisensory, stimuli at initial encounters. Likewise, healthy elderly and those with mild cognitive impairment can be classified based on performance on a simple multisensory detection task, but not from unisensory performance alone, highlighting functional links between multisensory processes and memory (dys)functions26.
While there is evidence that children (and, later, adults) indeed garner benefits from multisensory contexts when performing memory tasks, the links between benefits of multisensory information during stimulus processing and measures of intelligence remain to be firmly established. For example, one study of 95 school children aged 6–11 years old27 compared performance on an auditory-visual simple detection task with scores from Raven’s Coloured Progressive Matrices28 and the Neal Analysis of Reading Ability29. In this work, there was no evidence of a statistically reliable link between multisensory facilitation of behaviour and these measures of cognitive function. Instead, those results provide evidence that multisensory processes, at least those indexed by violations of Miller’s race model inequality, remain immature in this age group. In a later study of 88 school children, Barutchu and colleagues observed a significant difference in full-scale IQ between those children whose facilitation of reaction times exceeded probability summation and those children whose multisensory facilitation could be explained by probability summation30. An additional more recent study of 38 8–11 years-old children reported no correlation between (absolute) multisensory reaction time facilitation and IQ scores. Instead, there was a significant positive correlation between raw multisensory reaction time and their working memory index31. It should be noted, though, that there was no evidence for a systematic correlation between measures of multisensory facilitation and IQ scores (In fact, there were positive correlations between IQ and unisensory RTs27,32; a pattern somewhat at odds with the notion of IQ being coupled with processing speed or with facilitation under multisensory conditions).
That multisensory processing capabilities are related in some manner or another to cognitive ones is certain, as is the evidence that this relationship develops (and perhaps modulates in its nature) over childhood and adolescence. As this relationship could potentially offer a long-term scaffold to improve a child’s scholastic outcomes, both in the case of typical development as well as in cases of neurodevelopmental disorders15,33,34, clarification seems important. Our prior work in adults would indeed suggest that the manner in which an individual detects multisensory stimuli in their environment is predictive of how well multisensory contexts will be beneficial for recognition memory functions25,35,36,37,38,39,40,41. One implication is that low-level multisensory processes may be predictive of higher-level cognitive functions, be it multisensory or more traditional and unisensory, and that such relationships may be formed during childhood (and perhaps earlier). To better understand the nature of these interactions, here, we collected data from both a simple detection task and a continuous recognition memory task, which we have used extensively in our research in adults25,35,36,37,38,39,40,41, together with standardised neuropsychological measures of working memory and fluid intelligence in school-age children.
Materials and Methods
In total, seventy-seven children (36 girls) from 4.6 to 15.5 years old (Mage = 8.1 years, SD = 3.0 years) partook in the experiment. All children had normal or corrected vision and reported no hearing loss. Moreover, Swiss children are all screened at age of 4 for sensory and learning disabilities. Any child with a reported suspicion of such disabilities was excluded from participating in our study. These individuals are the subset of participants from another study comparing pedagogical settings, and so information about schooling was also collected (Montessori and traditional). Nine schoolchildren were excluded from the study due to poor performance on the detection task (N = 3), defined as an accuracy rate lower than 30%, or due to missing data from technical issues (N = 6). The final sample included 68 children (32 girls), aged 4.6–15.5 years (Mage = 7.9 years, Median = 6.4 years, SD = 3.0 years). The study was conducted in accordance with the Declaration of Helsinki, and all parents provided written informed consent for their child to participate. The experimental procedures were approved by the Vaudois Cantonal Ethics Committee (Commission cantonale d'éthique de la recherche sur l'être humain).
Tasks and procedure
All experiments took place within Swiss French-speaking schools, and a separate room was set up for testing of individual children. Two different examiners collected the data, and task order was randomized. For computerized tasks, children were seated in front of a 20”-screen laptop. The auditory stimuli were presented over headphones (model: CASIO LK-260), and the volume was adjusted to a comfortable level (~60 dB, as measured with the Decibel meter from the laptop)42. Both tasks were presented and controlled electronically using the E-Prime 2.0 Professional software (Psychology Software Tools, Pittsburgh, PA), and the behavioural data were recorded through the laptop’s keyboard.
Simple detection task
Children were presented with either visual (V), auditory (A) or audiovisual (AV) stimuli. Each child was presented with a total of 60 trials with a pseudo-randomised presentation, and equal distribution of the V, A, and AV conditions (i.e. 20 per condition). The visual stimuli were white drawings (cloud or star) presented on a black background, and the auditory stimuli were two different tones (44100 Hz digitisation; 16 bit stereo) that differed in their spectral composition to create two “opposite” types of sounds (the first one ranged from 20 Hz to 21000 Hz and the second one - from 18700 Hz to 19600 Hz). Stimuli were intermixed within blocks to maintain a high level of attention and unpredictability (in terms of which specific sensory modality would be stimulated). The audiovisual (AV) stimuli were the simultaneous and synchronous presentation of a visual and auditory stimulus. This type of detection paradigm is highly similar to that used by Fort and colleagues43 in their seminal work in adults. Stimulus duration was 500 ms and was followed by a randomised inter-stimulus interval (ISI) ranging from 1500 to 1900 ms, during which time a central, white fixation cross was presented. Children were asked to press a button (the keyboard spacebar) as fast as possible when they perceived any type of stimulus. Both accuracy and reaction time were recorded.
Continuous recognition task
Children performed a continuous recognition task, adapted from Thelen et al.25 The task was a 2-alterative forced choice that required the discrimination of initial (i.e., ‘first’) from repeated (i.e., ‘second’) instances of line drawings of common objects presented in a series of trials within a block (i.e., an “old/new” task) by pressing one of two buttons. The visual objects were black drawings presented centrally on a white background. The sounds were also selected from Thelen et al. (16 bit stereo; 44100 Hz digitization; 10 ms rise/fall to avoid clicks, they differed in their spectral composition, ranging from 100 Hz to 4700 Hz, and sometimes were modulated in terms of amplitude envelopes and/or waveform types). Trials were pseudo-randomised within a block of 60 trials (30 different drawings). On each trial a single image (selected from the original study) was presented alone (V) or with a congruent (AVc) or meaningless (AVm) sound (equal distribution of the three conditions; 10 trials per condition). Images were controlled to equate spatial frequency spectra and luminance between image groups (AV vs. V), according to the original task. Stimuli were presented for 500 ms, followed by a randomised inter-stimulus interval (IS) ranging from 900 to 1500 ms, where a fixation cross was shown. The mean number of trials between the initial and the repeated presentation was 5 ± 1 pictures for both V and AV conditions. Altogether, children performed four different blocks with new drawings each time (only two presentations of each drawing over all the experiment). The second presentation being always unisensory (V). Emphasis was put on both speed and accuracy. Supplementary Figure 1 illustrates the paradigm. Stimulus timing and synchrony across sensory modalities for both the simple detection task and the continuous recognition task were tested and verified using the EEG system in our laboratory as an “oscilloscope”. Visual signals were converted to voltage with a photodiode, and auditory signals were directly taken from the output of the sound card. Simultaneous stimulus presentation has been reported to be perceived as synchronous both by adults and children (e.g.44).
Children performed the Ascending Span task from the WISC-IV45 to investigate the relationship between elementary multisensory processes and more complex cognitive abilities such as working memory46. The child was asked to listen and memorise a string of numbers spoken out loud by the experimenter and to repeat the string in an ascending order. The assessment started with a two digits string, and if the child successfully performed two trials in a row, an additional digit was added to the string. If the child missed a trial, a digit was removed from the string. If they missed three trials in a row the evaluation stopped. A final score was computed for the ascending digit task, based on the maximal number of correctly memorized and properly re-ordered digits, with a maximum of 7. No time limit was set for the answer; only accuracy was emphasized. These scores were then age-standardised based on mean span per year of age based on ref. 47.
Fluid intelligence (g factor)
Children performed the black and white version48 of Raven’s Coloured Progressive Matrices28 to assess abstract reasoning and non-verbal intelligence. It is a multiple-choice test composed of 36 items. For each item, an incomplete matrix was presented, and the child was asked to identify the missing element to complete the matrix. Participants had 15 minutes to complete as many matrices as possible. This test was conducted collectively (per small groups of maximally 5 children). Raw scores were based on the number of correct items (max. 36). The raw scores were then age-standardised using the calibration scale based on a sample of 1064 French schoolchildren following a traditional pedagogy (ECPA Pearson)49.
As mentioned above, participants who missed more than 30% of the trials at the Simple Detection Task (3 children; mean age = 6.53, SD = 2.15), or with missing data due to technical issues (6 children; mean age = 10.28, SD = 1.10) were excluded from the analyses. Computerized data were pre-processed using Excel; correct trials with a valid RT (subject’ smean RT ± 3SD) were considered in analyses. Statistical analyses were run with Jamovi open-access software (retrieved from https://www.jamovi.org) as well as SPSS version 26 (IBM Corporation). Statistical significance criterion was set at p ≤ 0.05. For all tests, the effect size is reported (either partial eta squared or Cohen’s d). The full correlation matrix of the measures used in this study are provided in Supplemental Table 1.
First, to confirm multisensory benefits on a simple detection task, a repeated-measures analysis of covariance variance (ANCOVA) on mean RTs was performed with the within-subjects factor Condition (A, V, AV), and Age as the co-variate. We also performed this ANCOVA on detection rates. We also ran a repeated-measures ANCOVA on the accuracy rate [%] with the repetition conditions only from the continuous recognition task. The within subjects factor was Condition (V−, V + c, V + m) and Age was the co-variate. While previous results in adults has repeatedly indicated that RTs are not significantly modulated across conditions in this task50, we nonetheless also analysed RTs from the continuous recognition task in a similar ANCOVA design as described above.
Second, in order to investigate how low-level multisensory gain (simple detection task) was related to high-level (continuous recognition task) multisensory gain as well as to both working memory and fluid intelligence scores, a relative multisensory gain was derived from the detection task for each subject as:
In addition, a relative multisensory memory gain was computed from the continuous recognition task for congruent AV recall condition as:
In this study, we specifically addressed the relationship between low-level multisensory processes and higher-order cognitive abilities. First, the relative multisensory gain value of each subject was related to the relative multisensory memory gain from the continuous recognition task using a stepwise linear regression with the relative multisensory memory gain as the dependent variable and relative multisensory gain and age as the independent variables. Next, we related the relative multisensory gain and age-standardised working memory scores using a logistic regression model (given the fact that the working memory scores are discrete rather than continuous). Finally, we related the relative multisensory gain with age-standardised fluid intelligence scores using a stepwise linear regression with the fluid intelligence scores as the dependent variable and relative multisensory gain and age as the independent variables. For completion and despite our specific research questions regarding the relationship of relative multisensory gain to various global cognition measures, we also include a complete correlation table across all the measures in this study.
In addition, to control for the specificity of multisensory versus unisensory processes, we also computed the relative unisensory gain from the detection task, as:
We identified the slower and the faster sensory modality for each participant, separately. In 65 of the children, the visual modality was faster. In the remaining 3 children, the auditory modality was faster. This measure of relative unisensory gain was then related to (i) the relative multisensory memory gain from the continuous recognition task, (ii) age-standardised working memory scores, and age-standardised fluid intelligence scores in an analogous manner to what is described above.
Simple detection task
The children performed the simple detection task with near-ceiling performance. Mean detection rates were 93.1%, 95.1%, and 96.5% for the visual, auditory, and multisensory conditions, respectively. These data were submitted to a one-way repeated-measures ANCOVA, with Condition as the within-subjects factor and Age as the co-variate (Greenhouse-Geisser corrected degrees of freedom are reported in cases of violation of assumptions of sphericity). There was a main effect of Condition (F(1.813,119.639) = 4.769, p = 0.012, ηp2 = 0.07) and a general increase in accuracy with age (i.e., significant covariation; F(1,66)=14.11, p < 0.001, ηp2 = 0.18). However, this co-variation did not reliably differ across conditions (F(1.813,119.639) = 2.22, p = 0.12, ηp2 = 0.03). Detection rates for visual stimuli were significantly lower than those for multisensory stimuli (pbonferroni = 0.005, d = 0.36). No other contrasts were statistically significant (p’s > 0.17). Thus, and despite RTs being overall slower for A than V conditions (see below), there was no evidence that this slowing was matched by impaired detection rates.
Mean RTs were computed for each condition (AV, V, A) and subject (see Table 1 for group averages). Results of the one-way repeated-measures ANCOVA, with Condition as the within-subjects factor and Age as the co-variate, yielded a main effect of Condition (F(2,132) = 23.53, p < 0.001, ηp2 = 0.26) and a general decrease of RT with age (i.e., significant covariation; F(1,66) = 40.50, p < 0.001, ηp2 = 0.38). However, this co-variation did not reliably differ across conditions (F(2,132) = 2.27, p = 0.11, ηp2 = 0.00) (Fig. 1A). Post-hoc paired t-tests with a false-rate discovery (FDR) p-value correction at q = 0.05, showed participants had faster RTs on trials with AV stimuli than those with A stimuli (t(67) = 12.95, pFDR = 0.002, Cohen’s d = 1.57) as well as those with V stimuli (t(67) = 2.14, pFDR = 0.036; Cohen’s d = 0.26), and faster RTs for V than A condition (t(67) = 11.58, pFDR = 0.002, Cohen’s d = 1.40).
Across participants, the average relative multisensory gain was 3.19%, SD = 10.2%. The average absolute multisensory gain in milliseconds was 17.78 ms, SD = 75.95 ms. These metrics were highly positively correlated, even when controlling for age (partial r(65) = 0.975; p < 0.001). Across participants, the average relative unisensory gain was 14.98%, SD = 6.98%. The relative multisensory and unisensory gains (in percentages) were negatively correlated, when controlling for age (partial r(65) = −0.343; p = 0.004).
It is important to mention that our paradigm, which entailed 2 visual stimuli, 2 auditory stimuli and their 4 multisensory combinations. It could be argued that one of the visual stimuli or auditory stimuli was more challenging to process, despite the task requirement of simple detection and the high performance rates of the participants. To assess this possibility, we compared mean RTs for the 2 visual stimuli, and there was no significant difference (635 vs. 626 ms; p = 0.43). We also compared mean RTs for the 2 auditory stimuli, and there was no significant difference (735 vs. 753 ms; p = 0.20). It could also be argued that participants established an implicit association between a given visual and auditory stimulus; a notion referred to as crossmodal correspondence51. While the fact that all multisensory combinations were equally probable provides one level of argument against this possibility, we also assessed this empirically by comparing mean RTs from what could arguably be labelled as the congruent and incongruent combinations51. There was no significant difference (548 vs. 562 ms; p = 0.24).
Continuous recognition task
Accuracy rates [%] were computed for each repetition condition per subject; initially visual [V−] (mean = 68.8%, SD = 19.5%), initially paired with a meaningless sound [V + m] (mean = 69.4%, SD = 17.3%), and initially paired with a semantically congruent sound [V + c] (mean = 70.2%, SD = 17.3%). A repeated-measures ANCOVA, with Condition as the within-subjects factor and Age as the co-variate, yielded a significant covariation between accuracy and age (F(1,132) = 23.0, p < 0.001, ηp2 = 0.26). Neither the main effect of Condition (F(2,132) = 1.39, p = 0.25, ηp2 = 0.20), nor the interaction term of age co-varying differently across Condition (F(2,132) = 2.03, p = 0.14, ηp2 = 0.03) were reliable. Across participants, the average relative multisensory memory gain was 1.40%, SD = 13.0%, range −30% to 45%. The ANCOVA using RTs as the dependent measure did not yield a reliable main effect of Condition (F(2,132) < 1) or any reliable covariation with age (F(2,132) < 1).
Predictive value of gains in simple detection for memory and global cognitive functions
We first conducted a stepwise linear regression, using the relative multisensory memory gain as the dependent, outcome variable and relative multisensory gain on the detection task as well as age as independent variables. The regression model was statistically significant (R = 0.316; F(1,66) = 7.296, p = 0.009). Only the relative multisensory gain on the detection task was identified as a significant predictor of relative multisensory memory gain, accounting for 10% of the unique variance (part r = 0.316). Age did not significantly increase the performance of the model, p = 0.456). Figure 1B shows a scatterplot relating the relative multisensory gain on the detection task with that on the continuous recognition memory task.
Next, we conducted a multinomial logistic regression, using the age-standardised working memory scores as the dependent, outcome variable and relative multisensory gain on the detection task as well as age as covariates. Addition of the relative multisensory gain on the detection task and age to a model that contained only the intercept significantly improved the fit between the model and data, χ2(8, N = 68) = 30.90, Nagelkerke R2 = 0.392, p < 0.001. Significant unique contributions were made by both the relative multisensory gain on the detection task [χ2(4, N = 68) = 11.381; p = 0.023] and age [χ2(4, N = 68) = 19.724; p = 0.001]. Goodness of fit was explored by using the Pearson chi-square statistic, which was not statistically significant (p = 0.98). Figure 1C shows a scatterplot relating the relative multisensory gain on the detection task with age-standardised working memory scores.
Finally, we conducted a stepwise linear regression, using the age-standardised fluid intelligence scores as the dependent, outcome variable and relative multisensory gain on the detection task as well as age as independent variables. This regression model was statistically significant (R = 0.258; F(1,66) = 4.718, p = 0.033). Only the relative multisensory gain on the detection task was identified as a significant predictor of the age-standardised fluid intelligence scores, accounting for 6.7% of the unique variance (part r = 0.258). Age did not significantly increase the performance of the model, p = 0.392). Figure 1D shows a scatterplot relating the relative multisensory gain on the detection task with age-standardised fluid intelligence scores scores.
To assess the specificity of the relative multisensory gain on the detection task as a predictor of global cognitive functions, we performed the abovementioned regressions with the relative unisensory gain on the detection task. In the case of relative multisensory memory gain, the model including age and unisensory gain as predictors did not result in a significant improvement (R = 0.175; F(2,65) = 1.31; p = 0.362). In the case of age-standardised working memory scores, addition of the unisensory gain and age to a model that contained only the intercept significantly improved the fit between the model and data, χ2(8, N = 68) = 20.537, Nagelkerke R2 = 0.280, p = 0.008. Significant unique contributions were made only by age [χ2(4, N = 68) = 18.966; p = 0.001], but not by the unisensory gain [χ2(4, N = 68) = 1.016; p = 0.907]. Goodness of fit was explored by using the Pearson chi-square statistic, which was not significant (p = 0.94). In the case of age-standardised fluid intelligence scores, the model including age and unisensory gain as predictors did not result in a significant improvement (R = 0.221; F(2,65) = 1.67; p = 0.196).
In this study, we investigated the relationship between multisensory gain in a simple detection task and global cognitive measures such as memory, working memory and fluid intelligence. Our principal finding is the statistically significant and selective link between low-level multisensory processes and multiple measures of higher-order cognitive performance in schoolchildren. These links were observed not only with laboratory-based tasks, for which the contribution of age was controlled, but also with age-standardized clinical evaluation tools that index working memory and fluid intelligence. Such links did not generalize to unisensory processes, suggestive of a certain degree of specificity of the studied constructs. These collective findings reinforce the hypothesis that multisensory perceptual processes provide a crucial scaffolding for cognition throughout the lifespan1.
A long history of research has reported links between unisensory as well as multisensory stimulus processing and measures of cognition and intelligence in infants and pre-school children5,10,16,17. However, the majority of these studies involved tasks that required matching information (e.g. shape or temporal pattern) across the senses rather than simple detection of the stimuli19,20, obfuscating the ability to claim that it is specifically the low-level stimulus processing mediating such links, rather than a common higher-level cognitive function contributing to both tasks. In fact, we do know that children and adults allocate attention differently to unisensory and multisensory stimuli52,53. In a series of studies in schoolchildren, Barutchu and colleagues did not observe a linear correlation between indices of low-level multisensory processing and intelligence scores. Specifically, in their 2009 study, they found no evidence for associations between simple reaction times to multisensory stimuli (or absolute measures of multisensory facilitation as derived from mean RTs between multisensory condition and the better unisensory condition) and non-verbal IQ (as measured with Raven’s progressive matrices) or reading abilities (as measured with the Neal Analysis of Reading Ability). Likewise, no correlation was observed in a subsequent 2011 paper that used the WISC-IV as a measure of IQ. The 2009 study focused on age-related differences in the extent to which simple reaction times were facilitated under multisensory conditions beyond what could be explained by probability summation, using Miller’s race model inequality54. The 2011 study revealed IQ differences between sub-groups of children according to whether or not the child’s multisensory facilitation of RTs exceeded predictions of probability summation. The present results and those of Barutchu et al. are consistent to the extent that they both indicate that the degree to which a child benefits from multisensory stimuli is related to their global cognition. Here, in our view, age-related differences in violations of Miller’s race model inequality, even if repeatedly reported, ought to be considered with some caution. For one, there are examples in adults where such violation have not been systematically observed55, raising the possibility that this metric is not fully indicative of the maturity of multisensory processes. Second, biases in the use of Miller’s inequality can be observed when numbers of trials are low56,57, which is often the case in studies of children. Third, non-linear neural response interactions can be decoupled from violations in Miller’s inequality58,59. Additional research is clearly required to fully ascertain which multisensory or integrative processes are indexed by violations of Miller’s inequality. Such notwithstanding, evidence from multiple laboratories would indeed indicate that some forms of multisensory integration remain immature in children as old as 10–11 years old. Our findings thus provide an important extension beyond what has been previously reported by Barutchu and colleagues. By showing a correlation between the percentage of multisensory facilitation of simple reaction-time and age-standardised measures of IQ, we provide evidence that multisensory processes are sufficiently mature already in school-aged children to be informative of their global cognitive abilities. Our findings also extend the studies showing that cross-modal matching is linked with cognitive functions19,20. We show that multisensory gains during simple detection, which arguably rely on more rudimentary processes than those involved in temporal matching, can reliably predict several measures of global cognitive functions.
It is also worth mentioning that our data are consistent with a rich literature characterising links between unisensory processing speed (as measured on either simple or choice reaction time tasks) and intelligence measures (e.g.11,60; reviewed in61). This can be gleaned from the correlation matrix (Supplementary Table 1), which generally shows a negative correlation between unisensory reaction times and working memory as well as fluid intelligence scores, though not with performance on the continuous recognition task. However, multiple regression analyses that included both RTs and age did not lead to a consistent pattern of results. While unisensory RTs reliably predicted age-standardised fluid intelligence scores, they did not predict either age-standardised working memory scores or relative multisensory memory gains on the continuous recognition task (Supplementary Table 2). Such notwithstanding, the claim in these prior studies is that simple reaction times, in general, reflect processing speed of basic cognitive operations. One prominent hypothesis focuses on the notion of neural efficiency (reviewed in62). More efficient processing, as in the case of individuals with higher intelligence or cognitive abilities, is paralleled by faster reaction times. It has also be shown that multisensory conditions result in less variable reaction times63, which may be a further contributing factor as to why multisensory processes may be particularly informative of cognitive functions. Here, our findings only reinforce the idea that low-level stimulus processing – and specifically the ability to garner benefits from multisensory contexts during low-level stimulus processing – are tightly related to higher-level cognitive processes (i.e., memory) and intellectual abilities in schoolchildren. Nonetheless, additional research will be required to not only determine to what extent multisensory processes are innate and/or experience-dependent (see10 for discussion), but also to what extent genetic factors contribute to multisensory stimulus processing, particularly given some evidence for genetic contributions to speed of information processing and its link to IQ (e.g.60). Regarding the former aspect, an ongoing clinical trial by our group is investigating multisensory processes in prematurely born infants and children as well as their predictive value for cognitive functions and scholastic achievement64. It is likewise important to consider the extent to which our findings are indicative of links between multisensory processes and a common (and perhaps general) cognitive construct or multiple such constructs. One access point to this issue is the pattern of relationships across the continuous recognition task, working memory task, and fluid intelligence. There was no reliable association between the relative multisensory memory gain from the continuous recognition task and age-standardised working memory scores (η = 0.291; p = 0.225) nor a correlation between such and age-standardised fluid intelligence scores (r(66) = −0.027; p = 0.828). By contrast, and unsurprisingly (see65), age-standardised working memory and fluid intelligence measures were reliably associated (η = 0.482; p < 0.001) (see Supplementary Table 1). This overall pattern would suggest that the measures of working memory and fluid intelligence may be indexing a common cognitive construct. By contrast, the continuous recognition task is likely gauging a distinct construct. As such, it would thus appear that a child’s ability to garner multisensory benefits on a simple detection task provides an indicator of the integrity of at least two distinct cognitive systems.
There is a particularly straightforward and exciting implication of our findings; it suggests that multisensory learning, which is arguably more reflective of the sensory environment a child confronts and acts upon from birth onwards, could potentially empower cognitive development66. Linked to the above idea is the question of the extent to which multisensory processing abilities can be trained. This is a burgeoning field of empiric research. On the one hand, there are data showing that the so-called temporal binding window over which multisensory signals are perceptually bound is flexible and subject to learning67,68. This is important as the temporal binding window has been reported to be altered in a number of neurodevelopmental disorders69, as well as in aging70, and also to scale across tasks from simple detection to speech processing71. On the other hand, there are data showing that multisensory contexts are particularly effective for recognition memory not only in adults (reviewed in41), but also in schoolchildren21,22,23,24. That is, if multisensory processes can be trained67, preschool and school years may be ideal to facilitate early learning and basic perceptual skills through multisensory programs (discussed in72). This was already acknowledged by some educational approaches, such as Montessori Education where children work mainly through the manipulation of sensory materials73. Interestingly, the few quantitative existing studies in this area do report scholastic advantages for schoolchildren following a Montessori versus traditional system74,75,76. An additional, exploratory analysis of the current dataset was thus run on the children according to their schooling background and their multisensory gain on the detection task. The results revealed that Montessori scholars (half of the participants) were more likely to exhibit a multisensory gain than their peers in traditional schooling (χ2(1) = 5.7, p = 0.02) (Fig. S2). These preliminary results call for further investigation of the topic of pedagogical tools, but already support increased efforts in multisensory enrichment targeting learning and memory processes. More generally, our results are in line with the cognitive cascade model as proposed by Rose et al.16 that focuses on the relationship between low-level multisensory processes and higher-order cognitive skills. While our sample size was modest, there was the added value of readily controlling for many demographic factors in a setting such as Switzerland. However, replication and multi-cultural studies will be required to establish the potential utility of multisensory tasks as a screening tool and multisensory enrichment as a learning aide.
It is important to mention some limitations of the present study. First, our study included a wider range of ages than other comparable studies27,30,31. The fact that we included younger children may be one contributing factor to the smaller average relative multisensory gain on the detection task that we observed here versus that observed in works by Barutchu and colleagues. When we considered an age range restricted to the 6–11 year-old range in Barutchu et al.27, the average relative multisensory gain on the detection task was 5%. That said, it is perhaps important to note that relative multisensory gains in studies of adults exhibit considerable inter-individual as well as between-study variability55,77,78,79,80. More importantly, our results provide no evidence that age was significantly contributing to any of the models using relative multisensory gain on the detection task as a predictor. Second, our study made no effort to calibrate the stimuli used in the simple detection task; RTs to auditory stimuli were slower than those to visual stimuli. Other similar work in children has used stimuli that resulted in equivalent mean RTs to both unisensory conditions27,81. Larger multisensory gains are obtained when the distributions to the unisensory conditions are closer to each other63,82. Interestingly and consistently with our prior work in adults26, we observed a strong negative correlation between relative multisensory gains and relative unisensory gains. While there was no evidence here that relative unisensory gains were reliable predictors of cognitive abilities, it may be that a combined metric of relative multisensory and unisensory processes may prove particularly effective should a multisensory detection task be used as a screening tool for neurodevelopmental disorders (cf.26 for a similar tactic in the case of mild cognitive impairment). Third, our study used a detection task that was somewhat different from that used in prior works. While prior studies used a single visual stimulus, a single auditory stimulus and their multisensory combination, the present study used 2 visual stimuli, 2 auditory stimuli, and all 4 multisensory combinations thereof. Prior studies in adults have used a detection task with multiple stimuli and did not observe differences between specific items43. Similarly, we found no such differences here nor any evidence for implicit crossmodal correspondences (at least with the stimulus set we used). Nonetheless, it would be informative for future research to determine what might constitute an optimal detection task design both in terms of predictive value for cognitive (dys)function and in terms of ease-of-use in schoolchildren, but also in preschoolers and infants.
The present findings of reliable links between multisensory processes and higher-level cognition cannot directly speak to their causality. Nonetheless, our results would indeed suggest that low-level multisensory processes may constitute an effective access point for the assessment of children and their cognitive development. They reinforce the possible applicability of multisensory processes to public health screening in schoolchildren. In fact, our group has already demonstrated such in the case of screening for mild cognitive impairment in the elderly based on a similar multisensory simple detection task26. In that study, a combined measure of sensory dominance and multisensory gain on performance reliably classified healthy elderly from those with mild cognitive impairment at level comparable with a standard clinical tool (i.e. the Hopkins Verbal Learning Task). It would be particularly promising to apply the present results in screening of (pre)school children, particularly given that multisensory processing has been shown to be selectively impaired in dyslexia (e.g.33,83) as well as autism (e.g.84, reviewed in10). Moreover, the detection task per se circumvents some of the major limitations of current screening batteries (e.g. parental report, socio-economic bias, requirement of literacy/numeracy skills). Combined with a prompt administration time, a simple detection task makes an attractive potential screening tool for pre-schoolers or pre-linguistic children.
Murray, M. M., Lewkowicz, D. J., Amedi, A. & Wallace, M. T. Multisensory Processes: A Balancing Act across the Lifespan. Trends Neurosci. 39, 567–579 (2016).
Amso, D. & Scerif, G. The attentive brain: insights from developmental cognitive neuroscience. Nat. Rev. 16, 606–619 (2015).
Stein, B. E. & Meredith, M. A. The Merging of the Senses. (MIT Press, 1993).
Murray, M. M. & Wallace, M. T. The Neural Bases of Multisensory Processes. (CRC Press, 2012).
Bahrick, L. E. & Lickliter, R. Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Dev. Psychol. 36, 190–201 (2000).
Bahrick, L. E. & Lickliter, R. The role of intersensory redundancy in early perceptual, cognitive, and social development. In Multisensory development (eds. Bremner, A. J., Lewkowicz, D. J. & Spence, C.) 183–206 (Oxford University Press, 2012).
Lewkowicz, D. J. & King, A. J. The developmental and evolutionary emergence of multisensory processing: From single cells to behavior. In The New Handbook of Multisensory Processing (ed. Stein, B. E.) (MIT Press, 2012).
Lewkowicz, D. J. Early experience and multisensory perceptual narrowing. Dev. Psychobiol. 56, 292–315 (2014).
Matusz, P. J., Dikker, S., Huth, A. G. & Perrodin, C. Are We Ready for Real-world Neuroscience? J. Cogn. Neurosci. 1–13. https://doi.org/10.1162/jocn_e_01276 (2018).
Dionne-Dostie, E., Paquette, N., Lassonde, M. & Gallagher, A. Multisensory Integration and Child Neurodevelopment. Brain Sci. 5, 32–57 (2015).
Baker, L. A., Vernon, P. A. & Ho, H. Z. The genetic correlation between intelligence and speed of information processing. Behav. Genet. 21, 351–367 (1991).
Vernon, P. A., Nador, S. & Kantor, L. Reaction times and speed-of-processing: Their relationship to timed and untimed measures of intelligence. Intelligence 9, 357–374 (1985).
Sheppard, L. D. & Vernon, P. A. Intelligence and speed of information-processing: A review of 50 years of research. Pers. Individ. Dif. 44, 535–551 (2008).
Vernon, P. A. & Weese, S. E. Predicting intelligence with multiple speed of information-processing tests. Pers. Individ. Dif. 14, 413–419 (1993).
Park, J., Mainela-Arnold, E. & Miller, C. A. Information processing speed as a predictor of IQ in children with and without specific language impairment in grades 3 and 8. J. Commun. Disord. 53, 57–69 (2015).
Rose, S. A. et al. Cognitive Cascade in Infancy: Pathways from Prematurity to Later Mental Development. Intelligence 36, 367–378 (2008).
Birch, H. G., Belmont, L., Ph, D., Belmont, L. & Ph, D. Auditory visual integration in normal and retarded readers. Bull. Ort. Soc. 15, 48–96 (1965).
Birch, H. G. & Belmont, L. Auditory-Visual Integration, Intelligence and Reading Ability in School Children. Percept. Mot. Skills 20, 295–305 (1965).
Birch, H. G. & Belmont, L. Auditory visual integration in normal and retarded readers. Bull. Ort. Soc. 15, 48–96 (1965).
Rose, S. A., Feldman, J. F., Jankowski, J. J. & Futterweit, L. R. Visual and auditory temporal processing, cross-modal transfer, and reading. J. Learn. Disabil. 32, 256–66.
Heikkilä, J. & Tiippana, K. School-aged children can benefit from audiovisual semantic congruency during memory encoding. Exp. brain Res. 62, 123–130 (2015).
Heikkilä, J., Alho, K., Hyvönen, H. & Tiippana, K. Audiovisual semantic congruency during encoding enhances memory performance. Exp. Psychol. 62, 123–30 (2015).
Broadbent, H. J., White, H., Mareschal, D. & Kirkham, N. Z. Incidental learning in a multisensory environment across childhood. Dev. Sci. 21, (2018).
Broadbent, H. J., Osborne, T., Mareschal, D. & Kirkham, N. Z. Withstanding the test of time: Multisensory cues improve the delayed retention of incidental learning. Dev. Sci. 1–7. https://doi.org/10.1111/desc.12726 (2018).
Thelen, A., Matusz, P. J. P. J. & Murray, M. M. M. Multisensory context portends object memory. Curr. Biol. 24, R734–R735 (2014).
Murray, M. M. et al. Sensory dominance and multisensory integration as screening tools in aging. Sci. Rep. 8, 8901 (2018).
Barutchu, A., Crewther, D. P. & Crewther, S. G. The race that precedes coactivation: Development of multisensory facilitation in children. Dev. Sci. 12, 464–473 (2009).
John & Raven, J. Raven Progressive Matrices. in Handbook of Nonverbal Assessment 223–237 (Springer US, 2003). https://doi.org/10.1007/978-1-4615-0153-4_11.
Neale, M. D., McKay, M. F. & Childs, G. H. The Neale Analysis of Reading Ability - Revised. Br. J. Educ. Psychol. 56, 346–356 (1986).
Barutchu, A. et al. The relationship between multisensory integration and IQ in children. Dev. Psychol. 47, 877–885 (2011).
Barutchu, A., Fifer, J. M., Shivdasani, M. N., Crewther, S. G. & Paolini, A. G. The Interplay Between Multisensory Associative Learning and IQ in Children. Child Dev., https://doi.org/10.1111/cdev.13210 (2019).
Barutchu, A., Sahu, A., Humphreys, G. W. & Spence, C. Multisensory processing in event-based prospective memory. Acta Psychol. (Amst). 192, 23–30 (2019).
Harrar, V. et al. Multisensory integration and attention in developmental dyslexia. Curr. Biol. 24, 531–535 (2014).
Wallace, M. T. & Stevenson, R. A. The construct of the multisensory temporal binding window and its Dysregulation in developmental Disabilities. Neuropsychologia. https://doi.org/10.1016/j.neuropsychologia.2014.08.005 (2014).
Murray, M. M. et al. Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21, 125–135 (2004).
Lehmann, S. & Murray, M. M. The role of multisensory memories in unisensory object discrimination. Cogn. Brain Res. 24, (2005).
Murray, M. M., Foxe, J. J. & Wylie, G. R. The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage 27, 473–8 (2005).
Thelen, A., Cappe, C. & Murray, M. M. Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62, 1478–1488 (2012).
Matusz, P. J. P. J. et al. The role of auditory cortices in the retrieval of single-trial auditory-visual object memories. Eur. J. Neurosci. 41, 699–708 (2015).
Thelen, A., Talsma, D. & Murray, M. M. Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138, (2015).
Matusz, P. J., Wallace, M. T. & Murray, M. M. A multisensory perspective on object memory. Neuropsychologia 105, (2017).
Huth, M. E., Popelka, G. R. & Blevins, N. H. Comprehensive measures of sound exposures in cinemas using smart phones. Ear Hear. 35, 680–6.
Fort, A., Delpuech, C., Pernier, J. & Giard, M.-H. Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Cereb. Cortex 12, 1031–9 (2002).
Hillock, A. R., Powers, A. R. & Wallace, M. T. Binding of sights and sounds: Age-related changes in multisensory temporal processing. Neuropsychologia 49, 461–467 (2011).
Wechsler, D. Wechsler Intelligence Scale for Children and Adolescents. (2003).
Rose, S. A., Feldman, J. F. & Jankowski, J. J. The building blocks of cognition. J. Pediatr. 143, S54–61 (2003).
Fournier, M. & Albaret, J.-M. Étalonnage des blocs de Corsi sur une population d’enfants scolarisés du CP à la 6e. Développements 16–17, 76 (2013).
Tuddenham, R. D., Davis, L., Davison, L. & Schindler, R. An experimental group version for school children of the progressive matrices. J. Consult. Psychol. 22, 30 (1958).
Raven, J., Raven, J. & Court, H. Manual for Raven’s Progressive Matrices. (Harcourt Assessment, 2003).
Murray, M. M. & Thelen, A. The Efficacy of Single-Trial Multisensory Memories. Multisens. Res. 26, 483–502 (2013).
Spence, C. & Deroy, O. How automatic are crossmodal correspondences? Conscious. Cogn. 22, 245–260 (2013).
Matusz, P. J. et al. Multi-modal distraction: Insights from children’s limited attention. Cognition 136, 156–165 (2015).
Matusz, P. J., Merkley, R., Faure, M. & Scerif, G. Expert attention: Attentional allocation depends on the differential development of multisensory number representations. Cognition 186, 171–177 (2019).
Miller, J. Divided attention: evidence for coactivation with redundant signals. Cogn. Psychol. 14, 247–79 (1982).
Hughes, H. C., Reuter-Lorenz, P. A., Nozawa, G. & Fendrich, R. Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J. Exp. Psychol. Hum. Percept. Perform. 20, 131–53 (1994).
Kiesel, A., Miller, J. & Ulrich, R. Systematic biases and Type i error accumulation in tests of the race model inequality. Behav. Res. Methods 39, 539–551 (2007).
Gondan, M. & Minakata, K. A tutorial on testing the race model inequality. Attention, Perception, Psychophys. 78, 723–735 (2016).
Sperdin, H. F., Cappe, C., Foxe, J. J. & Murray, M. M. Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front. Integr. Neurosci. 3, 2 (2009).
Murray, M. M., Foxe, J. J., Higgins, B. A., Javitt, D. C. & Schroeder, C. E. Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: a high-density electrical mapping study. Neuropsychologia 39, 828–44 (2001).
Rijsdijk, F. V., Vernon, P. A. & Boomsma, D. I. The genetic basis of the relation between speed-of-information- processing and IQ. Behav. Brain Res. 95, 77–84 (1998).
Vernon, P. A. Speed of Information Processing and Intelligence. (Ablex Publishing, 1987).
Jensen, A. R. Reaction Time and Psychometric g. In A Model for Intelligence 93–132 (Springer Berlin Heidelberg, 1982). https://doi.org/10.1007/978-3-642-68664-1_4.
Murray, M. M., Thelen, A., Ionta, S. & Wallace, M. T. Contributions of intra- and inter- individual differences to multisensory processes. J. Cogn. Neurosci. (2018).
Neel, M. L. et al. Randomized controlled trial protocol to improve multisensory neural processing, language and motor outcomes in preterm infants. BMC Pediatr. 19, 81 (2019).
Fry, A. F. & Hale, S. Processing speed, working memory, and fluid intelligence. Psychol. Sci. 7, 237–241 (1996).
Shams, L. & Seitz, A. R. Benefits of multisensory learning. Trends Cogn. Sci. 12, 411–7 (2008).
Powers, A. R., Hillock, A. R. & Wallace, M. T. Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29, 12265–74 (2009).
Powers, A. R., Hillock-Dunn, A. & Wallace, M. T. Generalization of multisensory perceptual learning. Sci. Rep. 6, 1–9 (2016).
Stevenson, R. A. et al. Multisensory temporal integration in autism spectrum disorders. J Neurosci 34, 691–697 (2014).
Poliakoff, E., Shore, D. I., Lowe, C. & Spence, C. Visuotactile temporal order judgments in ageing. Neurosci. Lett. 396, 207–11 (2006).
Stevenson, R. A., Wallace, M. T. & Altieri, N. The interaction between stimulus factors and cognitive factors during multisensory integration of audiovisual speech. Front. Psychol. 5, 352 (2014).
Murray, M. M., Matusz, P. J. & Amedi, A. Neuroplasticity: Unexpected Consequences of Early Blindness. Curr. Biol. 25, (2015).
Montessori, M. The Secret of Childhood. (Ballantine Books, 1982).
Lillard, A. S. et al. Montessori preschool elevates and equalizes child outcomes: A longitudinal study. Front. Psychol. 8, 1–19 (2017).
Lillard, A. THE EARLY YEARS: Evaluating Montessori Education. Science (80-.). 313, 1893–1894 (2006).
Denervaud, S., Knebel, J.F., Hagmann, P. & Gentaz, E. Beyond executive functions, creativity skills benefit academic outcomes: Insights from Montessori education. PLoS One 14, e0225319 (2019).
Hecht, D., Reiner, M. & Karni, A. Multisensory enhancement: gains in choice and in simple response times. Exp. Brain Res. 189, 133–43 (2008).
Romei, V., Murray, M. M., Merabet, L. B. & Thut, G. Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J. Neurosci. 27, 11465–72 (2007).
Martuzzi, R. et al. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb. Cortex 17, 1672–9 (2007).
Molholm, S. et al. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res. Cogn. Brain Res. 14, 115–28 (2002).
Brandwein, A. B. et al. The Development of Multisensory Integration in High-Functioning Autism: High-Density Electrical Mapping and Psychophysical Measures Reveal Impairments in the Processing of Audiovisual Inputs. Cereb. Cortex. https://doi.org/10.1093/cercor/bhs109 (2012).
Otto, T. U., Dassy, B. & Mamassian, P. Principles of multisensory behavior. J. Neurosci. 33, 7463–74 (2013).
Hahn, N., Foxe, J. J. & Molholm, S. Impairments of multisensory integration and cross-sensory learning as pathways to dyslexia. Neurosci. Biobehav. Rev. 47, 384–392 (2014).
Stevenson, R. A. et al. The cascading influence of multisensory processing on speech perception in autism. Autism 22, 609–624 (2018).
Snodgrass, J. G. & Vanderwart, M. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. J. Exp. Psychol. Hum. Learn. 6, 174–215 (1980).
This study was supported by the Swiss National Sciences Foundation (Grants 320030-149982 and 320030-169206 to MMM, and grant PZ00P1_174150 to PJM) as well as by the Fondation Asile des aveugles (232933 to MMM), the Pierre Mercier Foundation (RSVS30349 to PJM) and a grantor advised by Carigest SA (232930 to MMM). We thank all the children and parents for their participation. We are likewise grateful to Professor David Sander for his kind support of S.D.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Denervaud, S., Gentaz, E., Matusz, P.J. et al. Multisensory Gains in Simple Detection Predict Global Cognition in Schoolchildren. Sci Rep 10, 1394 (2020). https://doi.org/10.1038/s41598-020-58329-4
Neonatal Multisensory Processing in Preterm and Term Infants Predicts Sensory Reactivity and Internalizing Tendencies in Early Childhood
Brain Topography (2020)
Multisensory Perception, Verbal, Visuo-spatial and Motor Working Memory Modulation After a Single Open- or Closed-Skill Exercise Session in Children
Journal of Cognitive Enhancement (2020)