Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Combining computational controls with natural text reveals aspects of meaning composition

Nature Computational Science volume 2, pages 745–757 (2022)Cite this article

Subjects

A preprint version of the article is available at bioRxiv.

Abstract

To study a core component of human intelligence—our ability to combine the meaning of words—neuroscientists have looked to linguistics. However, linguistic theories are insufficient to account for all brain responses reflecting linguistic composition. In contrast, we adopt a data-driven approach to study the composed meaning of words beyond their individual meaning, which we term ‘supra-word meaning’. We construct a computational representation for supra-word meaning and study its brain basis through brain recordings from two complementary imaging modalities. Using functional magnetic resonance imaging, we reveal that hubs that are thought to process lexical meaning also maintain supra-word meaning, suggesting a common substrate for lexical and combinatorial semantics. Surprisingly, we cannot detect supra-word meaning in magnetoencephalography, which suggests that composed meaning might be maintained through a different neural mechanism than the synchronized firing of pyramidal cells. This sensitivity difference has implications for past neuroimaging results and future wearable neurotechnology.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Approach.
Fig. 2: fMRI results.
Fig. 3: MEG prediction results at different spatial granularity.
Fig. 4: Direct comparisons of prediction performance of different meaning embeddings.

Similar content being viewed by others

Data availability

Two of the three data sets analysed during this study can be found at http://www.cs.cmu.edu/~fmri/plosone/ for the fMRI data set and at https://kilthub.cmu.edu/articles/dataset/RSVP_reading_of_book_chapter_in_MEG/20465898 for the MEG data set. The remaining data set is available from the Courtois Neuromod group at https://docs.cneuromod.ca/en/latest/ACCESS.html. Source data for Figs. 2–4 is available with this manuscript.

Code availability

All custom scripts are available without restrictions at https://github.com/brainML/supraword74.

References

  1. Pylkkänen, L. Neural basis of basic composition: what we have learned from the red-boat studies and their extensions. Philos. Trans. R. Soc. B 375, 20190299 (2020).

    Article  Google Scholar 

  2. Pylkkänen, L. & McElree, B. An MEG study of silent meaning. J. Cogn. Neurosci. 19, 1905–1921 (2007).

    Article  Google Scholar 

  3. Baggio, G., Choma, T., Van Lambalgen, M. & Hagoort, P. Coercion and compositionality. J. Cogn. Neurosci. 22, 2131–2140 (2010).

    Article  Google Scholar 

  4. Bemis, D. K. & Pylkkänen, L. Simple composition: a magnetoencephalography investigation into the comprehension of minimal linguistic phrases. J. Neurosci. 31, 2801–2814 (2011).

    Article  Google Scholar 

  5. Brooks, T. L. & de Garcia, D. C. Evidence for morphological composition in compound words using MEG. Front. Hum. Neurosci. 9, 215 (2015).

    Article  Google Scholar 

  6. Kim, S. & Pylkkänen, L. Composition of event concepts: evidence for distinct roles for the left and right anterior temporal lobes. Brain Lang. 188, 18–27 (2019).

    Article  Google Scholar 

  7. Peters M. E. et al. Deep contextualized word representations. In Proceedings of NAACL-HLT, pp 2227–2237 (2018).

  8. Devlin, J., Chang M.-W., Lee, K., & Toutanova, K.. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186 (2019).

  9. Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).

    Google Scholar 

  10. Wehbe, L., Vaswani, A., Knight, K., & Mitchell, T.. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014).

  11. Jain, S. & Huth, A. Incorporating context into language encoding models for fmri. In Advances in Neural Information Processing Systems, pp 6628–6637 (2018).

  12. Toneva, M. & Wehbe, L. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems, pp 14928–14938 (2019).

  13. Schrimpf, M. et al. The neural architecture of language: integrative modeling converges on predictive processing. Proc. Natl Acad. Sci. USA 118, 45 (2021).

    Article  Google Scholar 

  14. Caucheteux, C. & King, J.-R. Brains and algorithms partially converge in natural language processing. Commun. Biol. 5, 1–10 (2022).

    Article  Google Scholar 

  15. Goldstein, A. et al. Shared computational principles for language processing in humans and deep language models. Nat. Neurosci. 25, 369–380 (2022).

    Article  Google Scholar 

  16. Levesque, H., Davis, E. & Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer (2012).

  17. Marvin, R. & Linzen, T. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 1192–1202 (Association for Computational Linguistics, 2018).

  18. Baroni, M. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. Preprint at https://arxiv.org/abs/2106.08694 (2021).

  19. Hagoort, P. The meaning-making mechanism(s) behind the eyes and between the ears. Phil. Trans. R. Soc. B 375, 20190301 (2020).

    Article  Google Scholar 

  20. Hickok, G. & Poeppel, D. The cortical organization of speech processing. Nat. Rev. Neurosci. 8, 393–402 (2007).

    Article  Google Scholar 

  21. Hall, E. L., Robson, S. E., Morris, P. G. & Brookes, M. J. The relationship between MEG and fMRI. NeuroImage 102, 80–91 (2014).

    Article  Google Scholar 

  22. Muthukumaraswamy, S. D. & Singh, K. D. Spatiotemporal frequency tuning of bold and gamma band MEG responses compared in primary visual cortex. NeuroImage 40, 1552–1560 (2008).

    Article  Google Scholar 

  23. Muthukumaraswamy, S. D. & Singh, K. D. Functional decoupling of BOLD and gamma-band amplitudes in human primary visual cortex. Hum. Brain Mapp. 30, 2000–2007 (2009).

    Article  Google Scholar 

  24. Swettenham, J. B., Muthukumaraswamy, S. D. & Singh, K. D. BOLD responses in human primary visual cortex are insensitive to substantial changes in neural activity. Front. Hum. Neurosci. 7, 76 (2013).

    Article  Google Scholar 

  25. Leonardelli, E. & Fairhall, S. L. Similarity-based fMRI–MEG fusion reveals hierarchical organisation within the brainas semantic system. NeuroImage 259, 119405 (2022).

    Article  Google Scholar 

  26. Halgren, E. et al. N400-like magnetoencephalography responses modulated by semantic context, word frequency, and lexical class in sentences. NeuroImage 17, 1101–1116 (2002).

    Article  Google Scholar 

  27. Lyu, B. et al. Neural dynamics of semantic composition. Proc. Natl Acad. Sci. USA 116, 21318–21327 (2019).

    Article  Google Scholar 

  28. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).

    Google Scholar 

  29. Fedorenko, E., Hsieh, P.-J., Nieto-Castanon, A., Whitfield-Gabrieli, S. & Kanwisher, N. New method for fMRI investigations of language: defining ROIs functionally in individual subjects. J. Neurophysiol. 104, 1177–1194 (2010).

    Article  Google Scholar 

  30. Fedorenko, E. & Thompson-Schill, S. L. Reworking the language network. Trends Cogn. Sci. 18, 120–126 (2014).

    Article  Google Scholar 

  31. Binder, J. R., Desai, R. H., Graves, W. W. & Conant, L. L. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex 19, 2767–2796 (2009).

    Article  Google Scholar 

  32. Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).

    MathSciNet  MATH  Google Scholar 

  33. Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979).

    MathSciNet  MATH  Google Scholar 

  34. King, J.-R. & Dehaene, S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci. 18, 203–210 (2014).

    Article  Google Scholar 

  35. Huth, A. G. et al. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).

    Article  Google Scholar 

  36. Visser, M., Jefferies, E. & Ralph, M. A. L. Semantic processing in the anterior temporal lobes: a meta-analysis of the functional neuroimaging literature. J. Cogn. Neurosci. 22, 1083–1094 (2010).

    Article  Google Scholar 

  37. Pallier, C., Devauchelle, A.-D. & Dehaene, S. Cortical representation of the constituent structure of sentences. Proc. Natl Acad. Sci. 108, 2522–2527 (2011).

    Article  Google Scholar 

  38. Friederici, A. D. The brain basis of language processing: from structure to function. Physiol. Rev. 91, 1357–1392 (2011).

    Article  Google Scholar 

  39. Frankland, S. M. & Greene, J. D. An architecture for encoding sentence meaning in left mid-superior temporal cortex. Proc. Natl Acad. Sci. 112, 11732–11737 (2015).

    Article  Google Scholar 

  40. Skeide, M. A. & Friederici, A. D. The ontogeny of the cortical language network. Nat. Rev. Neurosci. 17, 323–332 (2016).

    Article  Google Scholar 

  41. Kutas, M. & Federmeier, K. D. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Annu. Ref. Psychol. 62, 621–647 (2011).

    Article  Google Scholar 

  42. Kuperberg, G. R. et al. Distinct patterns of neural modulation during the processing of conceptual and syntactic anomalies. J. Cogn. Neurosci. 15, 272–293 (2003).

    Article  Google Scholar 

  43. Kuperberg, G. R. Neural mechanisms of language comprehension: challenges to syntax. Brain Res. 1146, 23–49 (2007).

    Article  Google Scholar 

  44. Rabovsky, M., Hansen, S. S. & McClelland, J. L. Modelling the N400 brain potential as change in a probabilistic representation of meaning. Nat. Hum. Behav. 2, 693–705 (2018).

    Article  Google Scholar 

  45. Goldman-Rakic, P. S. Regional and cellular fractionation of working memory. Proc. Natl Acad. Sci. 93, 13473–13480 (1996).

    Article  Google Scholar 

  46. Luck, S. J., Vogel, E. K. & Shapiro, K. L. Word meanings can be accessed but not reported during the attentional blink. Nature 383, 616–618 (1996).

    Article  Google Scholar 

  47. Courtney, S. M., Ungerleider, L. G., Keil, K. & Haxby, J. V. Transient and sustained activity in a distributed neural system for human working memory. Nature 386, 608–611 (1997).

    Article  Google Scholar 

  48. Chen, G. et al. Hyperbolic trade-off: the importance of balancing trial and subject sample sizes in neuroimaging. NeuroImage 247, 118786 (2022).

    Article  Google Scholar 

  49. Fedorenko, E. et al. Neural correlate of the construction of sentence meaning. Proc. Natl Acad. Sci. 113, E6256–E6262 (2016).

    Article  Google Scholar 

  50. Hultén, A., Schoffelen, J.-M., Uddén, J., Lam, N. H. L. & Hagoort, P. How the brain makes sense beyond the processing of single words-an meg study. NeuroImage 186, 586–594 (2019).

    Article  Google Scholar 

  51. Toneva, M., Williams, J., Bollu, A., Dann, C. & Wehbe, L. Same cause; different effects in the brain. In First Conference on Causal Learning and Reasoning (2021).

  52. Tenney, I. et al. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations (2019).

  53. Makin, J. G., Moses, D. A. & Chang, E. F. Machine Translation of Cortical Activity to Text with an Encoder–Decoder Framework (Nature Publishing Group, 2020).

  54. Wehbe, L. et al. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PloS ONE 9, e112575 (2014).

    Article  Google Scholar 

  55. Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352 (2008).

    Article  Google Scholar 

  56. Rowling, J. K. Harry Potter and the Sorcerer’s Stone (Pottermore Limited, 2012).

  57. Fischl, B. Freesurfer. NeuroImage 62, 774–781 (2012).

    Article  Google Scholar 

  58. Gao, J. S., Huth, A. G., Lescroart, M. D. & Gallant, J. L. Pycortex: an interactive surface visualizer for fMRI. Front. Neuroinform. 9, 23 (2015).

    Article  Google Scholar 

  59. Esteban, O. et al. fmriprep software. Zenodo https://doi.org/10.5281/zenodo.852659 (2018).

  60. Taulu, S., Kajola, M. & Simola, J. Suppression of interference and artifacts by the signal space separation method. Brain Topogr. 16, 269–275 (2004).

    Article  MATH  Google Scholar 

  61. Taulu, S. & Simola, J. Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements. Phys. Med. Biol. 51, 1759–1768 (2006).

    Article  Google Scholar 

  62. Gardner, M. et al. Allennlp: a deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pp 1–6 (2018).

  63. Chelba, C. et al. One billion word benchmark for measuring progress in statistical language modeling. Preprint at https://arxiv.org/abs/1312.3005 (2013).

  64. Khandelwal, U., He, H., Qi, P. & Jurafsky, D. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 284–294 (2018).

  65. Sudre, G. et al. Tracking neural coding of perceptual and semantic features of concrete nouns. NeuroImage 62, 451–463 (2012).

    Article  Google Scholar 

  66. Nishimoto, S. et al. Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1641–1646 (2011).

    Article  Google Scholar 

  67. Wehbe, L., Ramdas, A., Steorts, R. C. & Shalizi, C. R. Regularized brain reading with shrinkage and smoothing. Ann. Appl. Stat. 9, 1997–2022 (2015).

    Article  MathSciNet  MATH  Google Scholar 

  68. Mitchell, T. M. et al. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195 (2008).

    Article  Google Scholar 

  69. Çukur, T., Nishimoto, S., Huth, A. G. & Gallant, J. L. Attention during natural vision warps semantic representation across the human brain. Nat. Neurosci. 16, 763–770 (2013).

    Article  Google Scholar 

  70. Deniz, F., Nunez-Elizalde, A. O., Huth, A. G. & Gallant, J. L. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. J. Neurosci. 39, 7722–7736 (2019).

    Article  Google Scholar 

  71. Sheppard, K. et al. Xavier RENE-CORAIL, and syncoding. bashtage/arch: release 4.15, June 2020. Zenodo https://doi.org/10.5281/zenodo.3906869 (2020).

  72. Efron, B. & Tibshirani, R. J. An Introduction to the Bootstrap (CRC, 1994).

  73. Miller, J. A warning about median reaction time. J. Exp. Psychol. Hum. Percept. Perform. 14, 539–543 (1988).

    Article  Google Scholar 

  74. Toneva, M. & Wehbe, L. brainml/supraword: version 1, October 2022. Zenodo https://doi.org/10.5281/zenodo.7178795 (2022).

Download references

Acknowledgements

We thank E. Laing and D. Howarth for help with data collection and preprocessing, and M.J. Tarr for helpful feedback on the manuscript. Research reported in this publication was partially supported by the National Institute on Deafness and other Communication Disorders of the National Institutes of Health under award no. R01DC020088. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This research was also supported in part by a Google Faculty Research Award and the Air Force Office of Scientific Research through research grants FA95501710218 and FA95502010118.

Author information

Authors and Affiliations

  1. Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA

    Mariya Toneva, Tom M. Mitchell & Leila Wehbe

  2. Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA

    Mariya Toneva, Tom M. Mitchell & Leila Wehbe

  3. Max Planck Institute for Software Systems, Saarbrücken, Germany

    Mariya Toneva

  4. Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, USA

    Leila Wehbe

Authors
  1. Mariya Toneva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tom M. Mitchell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leila Wehbe
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

L.W. and T.M.M. selected the experimental stimuli. L.W. collected the fMRI and MEG data. All authors helped conceive and design the experimental analyses and analysed the data. M.T. developed the technique to remove shared information in neural network embeddings and conducted subsequent analyses. M.T. and L.W. wrote the original draft of the manuscript. All authors contributed to the review and editing.

Corresponding author

Correspondence to Leila Wehbe.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Computational Science thanks Katrin Erk, Milena Rabovsky, Chengqing Zong and Willem Zuidema for their contribution to the peer review of this work. Handling editor: Kaitlin McCardle, in collaboration with the Nature Computational Science team. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Source data

Source Data Fig. 2

Statistical source data for Fig. 2b and the ATL and PTL sides of Fig. 2c.

Source Data Fig. 3

Statistical source data for Fig. 3.

Source Data Fig. 4

Statistical source data for Fig. 4.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Toneva, M., Mitchell, T.M. & Wehbe, L. Combining computational controls with natural text reveals aspects of meaning composition. Nat Comput Sci 2, 745–757 (2022). https://doi.org/10.1038/s43588-022-00354-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43588-022-00354-6

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics