To study a core component of human intelligence—our ability to combine the meaning of words—neuroscientists have looked to linguistics. However, linguistic theories are insufficient to account for all brain responses reflecting linguistic composition. In contrast, we adopt a data-driven approach to study the composed meaning of words beyond their individual meaning, which we term ‘supra-word meaning’. We construct a computational representation for supra-word meaning and study its brain basis through brain recordings from two complementary imaging modalities. Using functional magnetic resonance imaging, we reveal that hubs that are thought to process lexical meaning also maintain supra-word meaning, suggesting a common substrate for lexical and combinatorial semantics. Surprisingly, we cannot detect supra-word meaning in magnetoencephalography, which suggests that composed meaning might be maintained through a different neural mechanism than the synchronized firing of pyramidal cells. This sensitivity difference has implications for past neuroimaging results and future wearable neurotechnology.
This is a preview of subscription content, access via your institution
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Two of the three data sets analysed during this study can be found at http://www.cs.cmu.edu/~fmri/plosone/ for the fMRI data set and at https://kilthub.cmu.edu/articles/dataset/RSVP_reading_of_book_chapter_in_MEG/20465898 for the MEG data set. The remaining data set is available from the Courtois Neuromod group at https://docs.cneuromod.ca/en/latest/ACCESS.html. Source data for Figs. 2–4 is available with this manuscript.
Pylkkänen, L. Neural basis of basic composition: what we have learned from the red-boat studies and their extensions. Philos. Trans. R. Soc. B 375, 20190299 (2020).
Pylkkänen, L. & McElree, B. An MEG study of silent meaning. J. Cogn. Neurosci. 19, 1905–1921 (2007).
Baggio, G., Choma, T., Van Lambalgen, M. & Hagoort, P. Coercion and compositionality. J. Cogn. Neurosci. 22, 2131–2140 (2010).
Bemis, D. K. & Pylkkänen, L. Simple composition: a magnetoencephalography investigation into the comprehension of minimal linguistic phrases. J. Neurosci. 31, 2801–2814 (2011).
Brooks, T. L. & de Garcia, D. C. Evidence for morphological composition in compound words using MEG. Front. Hum. Neurosci. 9, 215 (2015).
Kim, S. & Pylkkänen, L. Composition of event concepts: evidence for distinct roles for the left and right anterior temporal lobes. Brain Lang. 188, 18–27 (2019).
Peters M. E. et al. Deep contextualized word representations. In Proceedings of NAACL-HLT, pp 2227–2237 (2018).
Devlin, J., Chang M.-W., Lee, K., & Toutanova, K.. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186 (2019).
Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
Wehbe, L., Vaswani, A., Knight, K., & Mitchell, T.. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014).
Jain, S. & Huth, A. Incorporating context into language encoding models for fmri. In Advances in Neural Information Processing Systems, pp 6628–6637 (2018).
Toneva, M. & Wehbe, L. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems, pp 14928–14938 (2019).
Schrimpf, M. et al. The neural architecture of language: integrative modeling converges on predictive processing. Proc. Natl Acad. Sci. USA 118, 45 (2021).
Caucheteux, C. & King, J.-R. Brains and algorithms partially converge in natural language processing. Commun. Biol. 5, 1–10 (2022).
Goldstein, A. et al. Shared computational principles for language processing in humans and deep language models. Nat. Neurosci. 25, 369–380 (2022).
Levesque, H., Davis, E. & Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer (2012).
Marvin, R. & Linzen, T. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 1192–1202 (Association for Computational Linguistics, 2018).
Baroni, M. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. Preprint at https://arxiv.org/abs/2106.08694 (2021).
Hagoort, P. The meaning-making mechanism(s) behind the eyes and between the ears. Phil. Trans. R. Soc. B 375, 20190301 (2020).
Hickok, G. & Poeppel, D. The cortical organization of speech processing. Nat. Rev. Neurosci. 8, 393–402 (2007).
Hall, E. L., Robson, S. E., Morris, P. G. & Brookes, M. J. The relationship between MEG and fMRI. NeuroImage 102, 80–91 (2014).
Muthukumaraswamy, S. D. & Singh, K. D. Spatiotemporal frequency tuning of bold and gamma band MEG responses compared in primary visual cortex. NeuroImage 40, 1552–1560 (2008).
Muthukumaraswamy, S. D. & Singh, K. D. Functional decoupling of BOLD and gamma-band amplitudes in human primary visual cortex. Hum. Brain Mapp. 30, 2000–2007 (2009).
Swettenham, J. B., Muthukumaraswamy, S. D. & Singh, K. D. BOLD responses in human primary visual cortex are insensitive to substantial changes in neural activity. Front. Hum. Neurosci. 7, 76 (2013).
Leonardelli, E. & Fairhall, S. L. Similarity-based fMRI–MEG fusion reveals hierarchical organisation within the brainas semantic system. NeuroImage 259, 119405 (2022).
Halgren, E. et al. N400-like magnetoencephalography responses modulated by semantic context, word frequency, and lexical class in sentences. NeuroImage 17, 1101–1116 (2002).
Lyu, B. et al. Neural dynamics of semantic composition. Proc. Natl Acad. Sci. USA 116, 21318–21327 (2019).
Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
Fedorenko, E., Hsieh, P.-J., Nieto-Castanon, A., Whitfield-Gabrieli, S. & Kanwisher, N. New method for fMRI investigations of language: defining ROIs functionally in individual subjects. J. Neurophysiol. 104, 1177–1194 (2010).
Fedorenko, E. & Thompson-Schill, S. L. Reworking the language network. Trends Cogn. Sci. 18, 120–126 (2014).
Binder, J. R., Desai, R. H., Graves, W. W. & Conant, L. L. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex 19, 2767–2796 (2009).
Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).
Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979).
King, J.-R. & Dehaene, S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci. 18, 203–210 (2014).
Huth, A. G. et al. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).
Visser, M., Jefferies, E. & Ralph, M. A. L. Semantic processing in the anterior temporal lobes: a meta-analysis of the functional neuroimaging literature. J. Cogn. Neurosci. 22, 1083–1094 (2010).
Pallier, C., Devauchelle, A.-D. & Dehaene, S. Cortical representation of the constituent structure of sentences. Proc. Natl Acad. Sci. 108, 2522–2527 (2011).
Friederici, A. D. The brain basis of language processing: from structure to function. Physiol. Rev. 91, 1357–1392 (2011).
Frankland, S. M. & Greene, J. D. An architecture for encoding sentence meaning in left mid-superior temporal cortex. Proc. Natl Acad. Sci. 112, 11732–11737 (2015).
Skeide, M. A. & Friederici, A. D. The ontogeny of the cortical language network. Nat. Rev. Neurosci. 17, 323–332 (2016).
Kutas, M. & Federmeier, K. D. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Annu. Ref. Psychol. 62, 621–647 (2011).
Kuperberg, G. R. et al. Distinct patterns of neural modulation during the processing of conceptual and syntactic anomalies. J. Cogn. Neurosci. 15, 272–293 (2003).
Kuperberg, G. R. Neural mechanisms of language comprehension: challenges to syntax. Brain Res. 1146, 23–49 (2007).
Rabovsky, M., Hansen, S. S. & McClelland, J. L. Modelling the N400 brain potential as change in a probabilistic representation of meaning. Nat. Hum. Behav. 2, 693–705 (2018).
Goldman-Rakic, P. S. Regional and cellular fractionation of working memory. Proc. Natl Acad. Sci. 93, 13473–13480 (1996).
Luck, S. J., Vogel, E. K. & Shapiro, K. L. Word meanings can be accessed but not reported during the attentional blink. Nature 383, 616–618 (1996).
Courtney, S. M., Ungerleider, L. G., Keil, K. & Haxby, J. V. Transient and sustained activity in a distributed neural system for human working memory. Nature 386, 608–611 (1997).
Chen, G. et al. Hyperbolic trade-off: the importance of balancing trial and subject sample sizes in neuroimaging. NeuroImage 247, 118786 (2022).
Fedorenko, E. et al. Neural correlate of the construction of sentence meaning. Proc. Natl Acad. Sci. 113, E6256–E6262 (2016).
Hultén, A., Schoffelen, J.-M., Uddén, J., Lam, N. H. L. & Hagoort, P. How the brain makes sense beyond the processing of single words-an meg study. NeuroImage 186, 586–594 (2019).
Toneva, M., Williams, J., Bollu, A., Dann, C. & Wehbe, L. Same cause; different effects in the brain. In First Conference on Causal Learning and Reasoning (2021).
Tenney, I. et al. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations (2019).
Makin, J. G., Moses, D. A. & Chang, E. F. Machine Translation of Cortical Activity to Text with an Encoder–Decoder Framework (Nature Publishing Group, 2020).
Wehbe, L. et al. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PloS ONE 9, e112575 (2014).
Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352 (2008).
Rowling, J. K. Harry Potter and the Sorcerer’s Stone (Pottermore Limited, 2012).
Fischl, B. Freesurfer. NeuroImage 62, 774–781 (2012).
Gao, J. S., Huth, A. G., Lescroart, M. D. & Gallant, J. L. Pycortex: an interactive surface visualizer for fMRI. Front. Neuroinform. 9, 23 (2015).
Esteban, O. et al. fmriprep software. Zenodo https://doi.org/10.5281/zenodo.852659 (2018).
Taulu, S., Kajola, M. & Simola, J. Suppression of interference and artifacts by the signal space separation method. Brain Topogr. 16, 269–275 (2004).
Taulu, S. & Simola, J. Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements. Phys. Med. Biol. 51, 1759–1768 (2006).
Gardner, M. et al. Allennlp: a deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pp 1–6 (2018).
Chelba, C. et al. One billion word benchmark for measuring progress in statistical language modeling. Preprint at https://arxiv.org/abs/1312.3005 (2013).
Khandelwal, U., He, H., Qi, P. & Jurafsky, D. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 284–294 (2018).
Sudre, G. et al. Tracking neural coding of perceptual and semantic features of concrete nouns. NeuroImage 62, 451–463 (2012).
Nishimoto, S. et al. Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1641–1646 (2011).
Wehbe, L., Ramdas, A., Steorts, R. C. & Shalizi, C. R. Regularized brain reading with shrinkage and smoothing. Ann. Appl. Stat. 9, 1997–2022 (2015).
Mitchell, T. M. et al. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195 (2008).
Çukur, T., Nishimoto, S., Huth, A. G. & Gallant, J. L. Attention during natural vision warps semantic representation across the human brain. Nat. Neurosci. 16, 763–770 (2013).
Deniz, F., Nunez-Elizalde, A. O., Huth, A. G. & Gallant, J. L. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. J. Neurosci. 39, 7722–7736 (2019).
Sheppard, K. et al. Xavier RENE-CORAIL, and syncoding. bashtage/arch: release 4.15, June 2020. Zenodo https://doi.org/10.5281/zenodo.3906869 (2020).
Efron, B. & Tibshirani, R. J. An Introduction to the Bootstrap (CRC, 1994).
Miller, J. A warning about median reaction time. J. Exp. Psychol. Hum. Percept. Perform. 14, 539–543 (1988).
Toneva, M. & Wehbe, L. brainml/supraword: version 1, October 2022. Zenodo https://doi.org/10.5281/zenodo.7178795 (2022).
We thank E. Laing and D. Howarth for help with data collection and preprocessing, and M.J. Tarr for helpful feedback on the manuscript. Research reported in this publication was partially supported by the National Institute on Deafness and other Communication Disorders of the National Institutes of Health under award no. R01DC020088. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This research was also supported in part by a Google Faculty Research Award and the Air Force Office of Scientific Research through research grants FA95501710218 and FA95502010118.
The authors declare no competing interests.
Peer review information
Nature Computational Science thanks Katrin Erk, Milena Rabovsky, Chengqing Zong and Willem Zuidema for their contribution to the peer review of this work. Handling editor: Kaitlin McCardle, in collaboration with the Nature Computational Science team. Peer reviewer reports are available.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Toneva, M., Mitchell, T.M. & Wehbe, L. Combining computational controls with natural text reveals aspects of meaning composition. Nat Comput Sci 2, 745–757 (2022). https://doi.org/10.1038/s43588-022-00354-6
This article is cited by
Nature Computational Science (2022)