Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Reverse-engineering the cortical architecture for controlled semantic cognition

Nature Human Behaviour volume 5, pages 774–786 (2021)Cite this article

Subjects

Abstract

We employ a reverse-engineering approach to illuminate the neurocomputational building blocks that combine to support controlled semantic cognition: the storage and context-appropriate use of conceptual knowledge. By systematically varying the structure of a computational model and assessing the functional consequences, we identified the architectural properties that best promote some core functions of the semantic system. Semantic cognition presents a challenging test case, as the brain must achieve two seemingly contradictory functions: abstracting context-invariant conceptual representations across time and modalities, while producing specific context-sensitive behaviours appropriate for the immediate task. These functions were best achieved in models possessing a single, deep multimodal hub with sparse connections from modality-specific regions, and control systems acting on peripheral rather than deep network layers. The reverse-engineered model provides a unifying account of core findings in the cognitive neuroscience of controlled semantic cognition, including evidence from anatomy, neuropsychology and functional brain imaging.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The seven different architectures.
Fig. 2: The model environment.
Fig. 3: Comparing the conceptual abstraction across the architectures without (in Phase 1) and with (in Phase 2) the additional demand of context-appropriate output.
Fig. 4: Comparing the training time across the architectures without (in Phase 1) and with (in Phase 2) the additional demand of context-appropriate output.
Fig. 5: Consequences of the location of the connection to control.
Fig. 6: Simulating different error patterns in SD versus SA.
Fig. 7: Simulating dynamic changes in univariate activation and functional connectivity across contexts.

Similar content being viewed by others

Data availability

The data are available upon request or can be generated using the code provided.

Code availability

The code for replicating all the simulations is available in the Supplementary Information and online at https://github.com/JacksonBecky/reverse-engineered-semantics. The code for further analysis is available online.

References

  1. Lambon Ralph, M. A., Jefferies, E., Patterson, K. & Rogers, T. T. The neural and computational bases of semantic cognition. Nat. Rev. Neurosci. 18, 42–55 (2017).

    Article  Google Scholar 

  2. Jefferies, E. The neural basis of semantic cognition: converging evidence from neuropsychology, neuroimaging and TMS. Cortex 49, 611–625 (2013).

    Article  PubMed  Google Scholar 

  3. Abel, T. J. et al. Direct physiologic evidence of a heteromodal convergence region for proper naming in human left anterior temporal lobe. J. Neurosci. 35, 1513–1520 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Wittgenstein, L. Philosophical Investigations (Blackwell, 1953).

  5. Lambon Ralph, M. A., Sage, K., Jones, R. W. & Mayberry, E. J. Coherent concepts are computed in the anterior temporal lobes. Proc. Natl Acad. Sci. USA 107, 2717–2722 (2010).

    Article  CAS  PubMed  Google Scholar 

  6. Rogers, T. T. & McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach (MIT Press, 2004).

  7. Saffran, E. M. The organization of semantic memory: in support of a distributed model. Brain Lang. 71, 204–212 (2000).

    Article  CAS  PubMed  Google Scholar 

  8. Thompson-Schill, S. L., D’Esposito, M., Aguirre, G. K. & Farah, M. J. Role of left inferior prefrontal cortex in retrieval of semantic knowledge: a reevaluation. Proc. Natl Acad. Sci. USA 94, 14792–14797 (1997).

    Article  CAS  PubMed  Google Scholar 

  9. Eggert, G. H. Wernicke’s Works on Aphasia: A Sourcebook and Review (Mouton, 1977).

  10. Patterson, K., Nestor, P. J. & Rogers, T. T. Where do you know what you know? The representation of semantic knowledge in the human brain. Nat. Rev. Neurosci. 8, 976–987 (2007).

    Article  CAS  PubMed  Google Scholar 

  11. Acosta-Cabronero, J. et al. Atrophy, hypometabolism and white matter abnormalities in semantic dementia tell a coherent story. Brain 134, 2025–2035 (2011).

    Article  PubMed  Google Scholar 

  12. Warrington, E. K. Selective impairment of semantic memory. Q. J. Exp. Psychol. 27, 635–657 (1975).

    Article  CAS  PubMed  Google Scholar 

  13. Jefferies, E. & Lambon Ralph, M. A. Semantic impairment in stroke aphasia versus semantic dementia: a case-series comparison. Brain 129, 2132–2147 (2006).

    Article  PubMed  Google Scholar 

  14. Martin, A., Haxby, J. V., Lalonde, F. M., Wiggs, C. L. & Ungerleider, L. G. Discrete cortical regions associated with knowledge of color and knowledge of action. Science 270, 102–105 (1995).

    Article  CAS  PubMed  Google Scholar 

  15. Wang, Y. et al. Dynamic neural architecture for social knowledge retrieval. Proc. Natl Acad. Sci. USA 114, E3305–E3314 (2017).

    CAS  PubMed  Google Scholar 

  16. Rosch, E., Mervis, C. B., Gray, W., Johnson, D. & Boyes-Braem, P. Basic objects in natural categories. Cogn. Psychol. 8, 382–439 (1976).

    Article  Google Scholar 

  17. Murphy, G. L. & Medin, D. L. The role of theories in conceptual coherence. Psychol. Rev. 92, 289–316 (1985).

    Article  CAS  PubMed  Google Scholar 

  18. Keil, F. C. in The Epigenesis of Mind: Essays on Biology and Cognition (eds Carey, S. & Gelman, R.) 237–256 (Lawrence Erlbaum Associates, 1991).

  19. Barsalou, L. W. Perceptual symbol systems. Behav. Brain Sci. 22, 577–660 (1999).

    Article  CAS  PubMed  Google Scholar 

  20. Gelman, S. A., Leslie, S. J., Was, A. M. & Koch, C. M. Children’s interpretations of general quantifiers, specific quantifiers and generics. Lang. Cogn. Neurosci. 30, 448–461 (2015).

    Article  PubMed  Google Scholar 

  21. Martin, A. & Chao, L. L. Semantic memory and the brain: structure and processes. Curr. Opin. Neurobiol. 11, 194–201 (2001).

    Article  CAS  PubMed  Google Scholar 

  22. Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  23. McCrae, K., de Sa, V. R. & Seidenberg, M. S. On the nature and scope of featural representations of word meaning. J. Exp. Psychol. Gen. 126, 99–130 (1997).

    Article  Google Scholar 

  24. Lambon Ralph, M. A., McClelland, J. L., Patterson, K., Galton, C. J. & Hodges, J. R. No right to speak? The relationship between object naming and semantic impairment: neuropsychological abstract evidence and a computational model. J. Cogn. Neurosci. 13, 341–356 (2001).

    Article  CAS  PubMed  Google Scholar 

  25. Farah, M. J. & McClelland, J. L. A computational model of semantic memory impairment: modality specificity and emergent category specificity. J. Exp. Psychol. Gen. 120, 339–357 (1991).

    Article  CAS  PubMed  Google Scholar 

  26. Devereux, B. J., Clarke, A. & Tyler, L. K. Integrated deep visual and semantic attractor neural networks predict fMRI pattern-information along the ventral object processing pathway. Sci. Rep. 8, 10636 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Binder, J. R. & Desai, R. H. The neurobiology of semantic memory. Trends Cogn. Sci. 15, 527–536 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  28. Damasio, H., Grabowski, T. J., Tranel, D., Hichwa, R. D. & Damasio, A. R. A neural basis for lexical retrieval. Nature 380, 499–505 (1996).

    Article  CAS  PubMed  Google Scholar 

  29. Damasio, A. R. & Damasio, H. in Computational Neuroscience: Large-Scale Neuronal Theories of the Brain (eds Koch, C. & Davis, J. L.) 61–74 (MIT Press, 1994).

  30. Mahon, B. Z. & Caramazza, A. A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. J. Physiol. Paris 102, 59–70 (2008).

    Article  PubMed  Google Scholar 

  31. Rogers, T. T. et al. Structure and deterioration of semantic memory: a neuropsychological and computational investigation. Psychol. Rev. 111, 205–235 (2004).

    Article  PubMed  Google Scholar 

  32. Lambon Ralph, M. A., Lowe, C. & Rogers, T. T. Neural basis of category-specific semantic deficits for living things: evidence from semantic dementia, HSVE and a neural network model. Brain 130, 1127–1137 (2007).

    Article  PubMed  Google Scholar 

  33. Binney, R. J., Embleton, K. V., Jefferies, E., Parker, G. J. M. & Lambon Ralph, M. A. The ventral and inferolateral aspects of the anterior temporal lobe are crucial in semantic memory: evidence from a novel direct comparison of distortion-corrected fMRI, rTMS, and semantic dementia. Cereb. Cortex 20, 2728–2738 (2010).

    Article  PubMed  Google Scholar 

  34. Visser, M., Jefferies, E., Embleton, K. V. & Lambon Ralph, M. A. Both the middle temporal gyrus and the ventral anterior temporal area are crucial for multimodal semantic processing: distortion-corrected fMRI evidence for a double gradient of information convergence in the temporal lobes. J. Cogn. Neurosci. 24, 1766–1778 (2012).

    Article  PubMed  Google Scholar 

  35. Shimotake, A. et al. Direct exploration of the ventral anterior temporal lobe in semantic memory: cortical stimulation and local field potential evidence from subdural grid electrodes. Cereb. Cortex 25, 3802–3817 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  36. Matsumoto, R. et al. Functional connectivity in the human language system: a cortico-cortical evoked potential study. Brain 127, 2316–2330 (2004).

    Article  PubMed  Google Scholar 

  37. Pobric, G., Jefferies, E. & Lambon Ralph, M. A. Anterior temporal lobes mediate semantic representation: mimicking semantic dementia by using rTMS in normal participants. Proc. Natl Acad. Sci. USA 104, 20137–20141 (2007).

    Article  CAS  PubMed  Google Scholar 

  38. Pobric, G., Jefferies, E. & Lambon Ralph, M. A. Amodal semantic representations depend on both anterior temporal lobes: evidence from repetitive transcranial magnetic stimulation. Neuropsychologia 48, 1336–1342 (2010).

    Article  PubMed  Google Scholar 

  39. Krizhevsky, A., Sutskever, I. & Hinton, G. E. in Advances in Neural Information Processing Systems 1097–1105 (2012).

  40. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Preprint at arXiv https://arxiv.org/abs/1512.03385 (2015).

  41. Chen, L., Lambon Ralph, M. A. & Rogers, T. T. A unified model of human semantic knowledge and its disorders. Nat. Hum. Behav. 1, 0039 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).

    Article  PubMed  Google Scholar 

  43. Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V. & McDermott, J. H. A task-optimized neural network replicates human auditory behaviour, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98, 630–644 (2018).

    Article  CAS  PubMed  Google Scholar 

  44. Plaut, D. C. Graded modality-specific specialisation in semantics: a computational account of optic aphasia. Cogn. Neuropsychol. 19, 603–639 (2002).

    Article  PubMed  Google Scholar 

  45. Nelson, M. E. & Bower, J. M. Brain maps and parallel computers. Trends Neurosci. 13, 403–408 (1990).

    Article  CAS  PubMed  Google Scholar 

  46. McNorgan, C., Reid, J. & McRae, K. Integrating conceptual knowledge within and across representational modalities. Cognition 118, 211–233 (2011).

    Article  PubMed  Google Scholar 

  47. Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

    PubMed  PubMed Central  Google Scholar 

  48. Dilkina, K. & Lambon Ralph, M. A. Conceptual structure within and between modalities. Front. Hum. Neurosci. 31, 333 (2013).

    Google Scholar 

  49. Cohen, J. D., Dunbar, K. & McClelland, J. L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990).

    Article  CAS  PubMed  Google Scholar 

  50. Visser, M., Embleton, K. V., Jefferies, E., Parker, G. J. & Lambon Ralph, M. A. The inferior, anterior temporal lobes and semantic memory clarified: novel evidence from distortion-corrected fMRI. Neuropsychologia 48, 1689–1696 (2010).

    Article  CAS  PubMed  Google Scholar 

  51. Rice, G. E., Hoffman, P. & Lambon Ralph, M. A. Graded specialization within and between the anterior temporal lobes. Ann. N. Y. Acad. Sci. 1359, 84–97 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Halai, A., Welbourne, S., Embleton, K. V. & Parkes, L. A comparison of dual-echo and spin-echo fMRI of the inferior temporal lobe. Hum. Brain Mapp. 35, 4118–4128 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  53. Chen, Y. et al. The ‘when’ and ‘where’ of semantic coding in the anterior temporal lobe: temporal representational similarity analysis of electrocorticogram data. Cortex 79, 1–13 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Marinkovic, K. et al. Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron 38, 487–497 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Herbet, G., Zemmoura, I. & Duffau, H. Functional anatomy of the inferior longitudinal fasciculus: from historical reports to current hypotheses. Front. Neuroanat. https://doi.org/10.3389/fnana.2018.00077 (2018).

  56. Catani, M., Jones, D. K., Donato, R. & Ffytche, D. H. Occipito-temporal connections in the human brain. Brain 126, 2093–2107 (2003).

    Article  PubMed  Google Scholar 

  57. Bajada, C. J., Banks, B. A., Lambon Ralph, M. A. & Cloutman, L. L. Reconnecting with Joseph and Augusta Dejerine: 100 years on. Brain 140, 2752–2759 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  58. Bouhali, F. et al. Anatomical connections of the visual word form area. J. Neurosci. 34, 15402–15414 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Binney, R. J., Parker, G. J. M. & Lambon Ralph, M. A. Convergent connectivity and graded specialization in the rostral human temporal lobe as revealed by diffusion-weighted imaging probabilistic tractography. J. Cogn. Neurosci. 24, 1998–2014 (2012).

    Article  PubMed  Google Scholar 

  60. Jung, J., Cloutman, L., Binney, R. J. & Lambon Ralph, M. A. The structural connectivity of higher order association cortices reflects human functional brain networks. Cortex 97, 221–239 (2016).

    Article  PubMed  Google Scholar 

  61. Morton, J. & Patterson, K. in Deep Dyslexia (eds Patterson, K. et al.) 91–118 (Routledge and Kegan Paul, 1980).

  62. Bozeat, S., Lambon Ralph, M. A., Patterson, K., Garrard, P. & Hodges, J. R. Non-verbal semantic impairment in semantic dementia. Neuropsychologia 38, 1207–1215 (2000).

    Article  CAS  PubMed  Google Scholar 

  63. Rogers, T. T., Patterson, K., Jefferies, E. & Lambon Ralph, M. A. Disorders of representation and control in semantic cognition: effects of familiarity, typicality, and specificity. Neuropsychologia 76, 220–239 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  64. Kuhnke, P., Kiefer, M. & Hartwigsen, G. Task-dependent recruitment of modality-specific and multimodal regions during conceptual processing. Cereb. Cortex 30, 3938–3959 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  65. Chiou, R., Humphreys, G. F., Jung, J. & Lambon Ralph, M. A. Controlled semantic cognition relies upon dynamic and flexible interactions between the executive ‘semantic control’ and hub-and-spoke ‘semantic representation’ systems. Cortex 103, 100–116 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Martin, A. GRAPES—grounding representations in action, perception, and emotion systems: how object properties and categories are represented in the human brain. Psychon. Bull. Rev. 23, 979–990 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  67. Bengio, Y. & Delalleau, O. in International Conference on Algorithmic Learning Theory (eds Kivinen, J. et al.) 18–36 (Springer, 2011).

  68. Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl. Based Syst. 06, 107–116 (1998).

    Article  Google Scholar 

  69. Saxe, A. M., McClelland, J. L. & Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (2014).

  70. Bar, M. A cortical mechanism for triggering top-down facilitation in visual object recognition. J. Cogn. Neurosci. 15, 600–609 (2003).

    Article  PubMed  Google Scholar 

  71. Bar, M. et al. Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA 103, 449–454 (2006).

    Article  CAS  PubMed  Google Scholar 

  72. Noonan, K. A., Jefferies, E., Visser, M. & Lambon Ralph, M. A. Going beyond inferior prefrontal involvement in semantic control: evidence for the additional contribution of dorsal angular gyrus and posterior middle temporal cortex. J. Cogn. Neurosci. 25, 1824–1850 (2013).

    Article  PubMed  Google Scholar 

  73. McKee, J. L., Riesenhuber, M., Miller, E. K. & Freedman, D. J. Task dependence of visual and category representations in prefrontal and inferior temporal cortices. J. Neurosci. 34, 16065–16075 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  74. Jackson, R. L., Cloutman, L. & Lambon Ralph, M. A. Exploring distinct default mode and semantic networks using a systematic ICA approach. Cortex 113, 279–297 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  75. Davey, J. et al. Exploring the role of the posterior middle temporal gyrus in semantic cognition: integration of anterior temporal lobe with executive processes. NeuroImage 137, 165–177 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  76. Rohde, D. L. T. LENS: The Light, Efficient Network Simulator Technical Report No. CMU-CS-99-164 (Carnegie Mellon University, Department of Computer Science, 1999).

  77. SPSS Statistics for Windows v.25.0 (IBM, 2017).

  78. Cloutman, L. L., Binney, R. J., Drakesmith, M., Parker, G. J. M. & Lambon Ralph, M. A. The variation of function across the human insula mirrors its pattern of structural connectivity: evidence from in vivo probabilistic tractography. NeuroImage 59, 3514–3521 (2012).

    Article  PubMed  Google Scholar 

  79. McIntosh, A. R. Mapping cognition to the brain through neural interactions. Memory 7, 523–548 (1999).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by a British Academy Postdoctoral Fellowship awarded to R.L.J. (no. pf170068), a programme grant to M.A.L.R. and T.T.R. from the Medical Research Council (grant no. MR/R023883/1), an Advanced Grant from the European Research Council to M.A.L.R. (GAP: 670428) and Medical Research Council intramural funding (no. MC_UU_00005/18). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. MRC Cognition & Brain Sciences Unit, University of Cambridge, Cambridge, UK

    Rebecca L. Jackson & Matthew A. Lambon Ralph

  2. Department of Psychology, University of Wisconsin–Madison, Madison, WI, USA

    Timothy T. Rogers

Authors
  1. Rebecca L. Jackson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timothy T. Rogers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew A. Lambon Ralph
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

R.L.J., T.T.R. and M.A.L.R. made substantial contributions to the conception and design of the work, the interpretation of the data and the manuscript revisions. R.L.J. acquired the results and drafted the manuscript.

Corresponding authors

Correspondence to Rebecca L. Jackson or Matthew A. Lambon Ralph.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Human Behaviour thanks Peter Hagoort, Andrew Saxe and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Tables 1–7, Notes 1–10 (including Figs. 1–12) and Methods 1 and 2.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jackson, R.L., Rogers, T.T. & Lambon Ralph, M.A. Reverse-engineering the cortical architecture for controlled semantic cognition. Nat Hum Behav 5, 774–786 (2021). https://doi.org/10.1038/s41562-020-01034-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-020-01034-z

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing