We employ a reverse-engineering approach to illuminate the neurocomputational building blocks that combine to support controlled semantic cognition: the storage and context-appropriate use of conceptual knowledge. By systematically varying the structure of a computational model and assessing the functional consequences, we identified the architectural properties that best promote some core functions of the semantic system. Semantic cognition presents a challenging test case, as the brain must achieve two seemingly contradictory functions: abstracting context-invariant conceptual representations across time and modalities, while producing specific context-sensitive behaviours appropriate for the immediate task. These functions were best achieved in models possessing a single, deep multimodal hub with sparse connections from modality-specific regions, and control systems acting on peripheral rather than deep network layers. The reverse-engineered model provides a unifying account of core findings in the cognitive neuroscience of controlled semantic cognition, including evidence from anatomy, neuropsychology and functional brain imaging.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 per month
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Get just this article for as long as you need it
Prices may be subject to local taxes which are calculated during checkout
The data are available upon request or can be generated using the code provided.
The code for replicating all the simulations is available in the Supplementary Information and online at https://github.com/JacksonBecky/reverse-engineered-semantics. The code for further analysis is available online.
Lambon Ralph, M. A., Jefferies, E., Patterson, K. & Rogers, T. T. The neural and computational bases of semantic cognition. Nat. Rev. Neurosci. 18, 42–55 (2017).
Jefferies, E. The neural basis of semantic cognition: converging evidence from neuropsychology, neuroimaging and TMS. Cortex 49, 611–625 (2013).
Abel, T. J. et al. Direct physiologic evidence of a heteromodal convergence region for proper naming in human left anterior temporal lobe. J. Neurosci. 35, 1513–1520 (2015).
Wittgenstein, L. Philosophical Investigations (Blackwell, 1953).
Lambon Ralph, M. A., Sage, K., Jones, R. W. & Mayberry, E. J. Coherent concepts are computed in the anterior temporal lobes. Proc. Natl Acad. Sci. USA 107, 2717–2722 (2010).
Rogers, T. T. & McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach (MIT Press, 2004).
Saffran, E. M. The organization of semantic memory: in support of a distributed model. Brain Lang. 71, 204–212 (2000).
Thompson-Schill, S. L., D’Esposito, M., Aguirre, G. K. & Farah, M. J. Role of left inferior prefrontal cortex in retrieval of semantic knowledge: a reevaluation. Proc. Natl Acad. Sci. USA 94, 14792–14797 (1997).
Eggert, G. H. Wernicke’s Works on Aphasia: A Sourcebook and Review (Mouton, 1977).
Patterson, K., Nestor, P. J. & Rogers, T. T. Where do you know what you know? The representation of semantic knowledge in the human brain. Nat. Rev. Neurosci. 8, 976–987 (2007).
Acosta-Cabronero, J. et al. Atrophy, hypometabolism and white matter abnormalities in semantic dementia tell a coherent story. Brain 134, 2025–2035 (2011).
Warrington, E. K. Selective impairment of semantic memory. Q. J. Exp. Psychol. 27, 635–657 (1975).
Jefferies, E. & Lambon Ralph, M. A. Semantic impairment in stroke aphasia versus semantic dementia: a case-series comparison. Brain 129, 2132–2147 (2006).
Martin, A., Haxby, J. V., Lalonde, F. M., Wiggs, C. L. & Ungerleider, L. G. Discrete cortical regions associated with knowledge of color and knowledge of action. Science 270, 102–105 (1995).
Wang, Y. et al. Dynamic neural architecture for social knowledge retrieval. Proc. Natl Acad. Sci. USA 114, E3305–E3314 (2017).
Rosch, E., Mervis, C. B., Gray, W., Johnson, D. & Boyes-Braem, P. Basic objects in natural categories. Cogn. Psychol. 8, 382–439 (1976).
Murphy, G. L. & Medin, D. L. The role of theories in conceptual coherence. Psychol. Rev. 92, 289–316 (1985).
Keil, F. C. in The Epigenesis of Mind: Essays on Biology and Cognition (eds Carey, S. & Gelman, R.) 237–256 (Lawrence Erlbaum Associates, 1991).
Barsalou, L. W. Perceptual symbol systems. Behav. Brain Sci. 22, 577–660 (1999).
Gelman, S. A., Leslie, S. J., Was, A. M. & Koch, C. M. Children’s interpretations of general quantifiers, specific quantifiers and generics. Lang. Cogn. Neurosci. 30, 448–461 (2015).
Martin, A. & Chao, L. L. Semantic memory and the brain: structure and processes. Curr. Opin. Neurobiol. 11, 194–201 (2001).
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).
McCrae, K., de Sa, V. R. & Seidenberg, M. S. On the nature and scope of featural representations of word meaning. J. Exp. Psychol. Gen. 126, 99–130 (1997).
Lambon Ralph, M. A., McClelland, J. L., Patterson, K., Galton, C. J. & Hodges, J. R. No right to speak? The relationship between object naming and semantic impairment: neuropsychological abstract evidence and a computational model. J. Cogn. Neurosci. 13, 341–356 (2001).
Farah, M. J. & McClelland, J. L. A computational model of semantic memory impairment: modality specificity and emergent category specificity. J. Exp. Psychol. Gen. 120, 339–357 (1991).
Devereux, B. J., Clarke, A. & Tyler, L. K. Integrated deep visual and semantic attractor neural networks predict fMRI pattern-information along the ventral object processing pathway. Sci. Rep. 8, 10636 (2018).
Binder, J. R. & Desai, R. H. The neurobiology of semantic memory. Trends Cogn. Sci. 15, 527–536 (2011).
Damasio, H., Grabowski, T. J., Tranel, D., Hichwa, R. D. & Damasio, A. R. A neural basis for lexical retrieval. Nature 380, 499–505 (1996).
Damasio, A. R. & Damasio, H. in Computational Neuroscience: Large-Scale Neuronal Theories of the Brain (eds Koch, C. & Davis, J. L.) 61–74 (MIT Press, 1994).
Mahon, B. Z. & Caramazza, A. A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. J. Physiol. Paris 102, 59–70 (2008).
Rogers, T. T. et al. Structure and deterioration of semantic memory: a neuropsychological and computational investigation. Psychol. Rev. 111, 205–235 (2004).
Lambon Ralph, M. A., Lowe, C. & Rogers, T. T. Neural basis of category-specific semantic deficits for living things: evidence from semantic dementia, HSVE and a neural network model. Brain 130, 1127–1137 (2007).
Binney, R. J., Embleton, K. V., Jefferies, E., Parker, G. J. M. & Lambon Ralph, M. A. The ventral and inferolateral aspects of the anterior temporal lobe are crucial in semantic memory: evidence from a novel direct comparison of distortion-corrected fMRI, rTMS, and semantic dementia. Cereb. Cortex 20, 2728–2738 (2010).
Visser, M., Jefferies, E., Embleton, K. V. & Lambon Ralph, M. A. Both the middle temporal gyrus and the ventral anterior temporal area are crucial for multimodal semantic processing: distortion-corrected fMRI evidence for a double gradient of information convergence in the temporal lobes. J. Cogn. Neurosci. 24, 1766–1778 (2012).
Shimotake, A. et al. Direct exploration of the ventral anterior temporal lobe in semantic memory: cortical stimulation and local field potential evidence from subdural grid electrodes. Cereb. Cortex 25, 3802–3817 (2014).
Matsumoto, R. et al. Functional connectivity in the human language system: a cortico-cortical evoked potential study. Brain 127, 2316–2330 (2004).
Pobric, G., Jefferies, E. & Lambon Ralph, M. A. Anterior temporal lobes mediate semantic representation: mimicking semantic dementia by using rTMS in normal participants. Proc. Natl Acad. Sci. USA 104, 20137–20141 (2007).
Pobric, G., Jefferies, E. & Lambon Ralph, M. A. Amodal semantic representations depend on both anterior temporal lobes: evidence from repetitive transcranial magnetic stimulation. Neuropsychologia 48, 1336–1342 (2010).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. in Advances in Neural Information Processing Systems 1097–1105 (2012).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Preprint at arXiv https://arxiv.org/abs/1512.03385 (2015).
Chen, L., Lambon Ralph, M. A. & Rogers, T. T. A unified model of human semantic knowledge and its disorders. Nat. Hum. Behav. 1, 0039 (2017).
Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V. & McDermott, J. H. A task-optimized neural network replicates human auditory behaviour, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98, 630–644 (2018).
Plaut, D. C. Graded modality-specific specialisation in semantics: a computational account of optic aphasia. Cogn. Neuropsychol. 19, 603–639 (2002).
Nelson, M. E. & Bower, J. M. Brain maps and parallel computers. Trends Neurosci. 13, 403–408 (1990).
McNorgan, C., Reid, J. & McRae, K. Integrating conceptual knowledge within and across representational modalities. Cognition 118, 211–233 (2011).
Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).
Dilkina, K. & Lambon Ralph, M. A. Conceptual structure within and between modalities. Front. Hum. Neurosci. 31, 333 (2013).
Cohen, J. D., Dunbar, K. & McClelland, J. L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990).
Visser, M., Embleton, K. V., Jefferies, E., Parker, G. J. & Lambon Ralph, M. A. The inferior, anterior temporal lobes and semantic memory clarified: novel evidence from distortion-corrected fMRI. Neuropsychologia 48, 1689–1696 (2010).
Rice, G. E., Hoffman, P. & Lambon Ralph, M. A. Graded specialization within and between the anterior temporal lobes. Ann. N. Y. Acad. Sci. 1359, 84–97 (2015).
Halai, A., Welbourne, S., Embleton, K. V. & Parkes, L. A comparison of dual-echo and spin-echo fMRI of the inferior temporal lobe. Hum. Brain Mapp. 35, 4118–4128 (2014).
Chen, Y. et al. The ‘when’ and ‘where’ of semantic coding in the anterior temporal lobe: temporal representational similarity analysis of electrocorticogram data. Cortex 79, 1–13 (2016).
Marinkovic, K. et al. Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron 38, 487–497 (2003).
Herbet, G., Zemmoura, I. & Duffau, H. Functional anatomy of the inferior longitudinal fasciculus: from historical reports to current hypotheses. Front. Neuroanat. https://doi.org/10.3389/fnana.2018.00077 (2018).
Catani, M., Jones, D. K., Donato, R. & Ffytche, D. H. Occipito-temporal connections in the human brain. Brain 126, 2093–2107 (2003).
Bajada, C. J., Banks, B. A., Lambon Ralph, M. A. & Cloutman, L. L. Reconnecting with Joseph and Augusta Dejerine: 100 years on. Brain 140, 2752–2759 (2017).
Bouhali, F. et al. Anatomical connections of the visual word form area. J. Neurosci. 34, 15402–15414 (2014).
Binney, R. J., Parker, G. J. M. & Lambon Ralph, M. A. Convergent connectivity and graded specialization in the rostral human temporal lobe as revealed by diffusion-weighted imaging probabilistic tractography. J. Cogn. Neurosci. 24, 1998–2014 (2012).
Jung, J., Cloutman, L., Binney, R. J. & Lambon Ralph, M. A. The structural connectivity of higher order association cortices reflects human functional brain networks. Cortex 97, 221–239 (2016).
Morton, J. & Patterson, K. in Deep Dyslexia (eds Patterson, K. et al.) 91–118 (Routledge and Kegan Paul, 1980).
Bozeat, S., Lambon Ralph, M. A., Patterson, K., Garrard, P. & Hodges, J. R. Non-verbal semantic impairment in semantic dementia. Neuropsychologia 38, 1207–1215 (2000).
Rogers, T. T., Patterson, K., Jefferies, E. & Lambon Ralph, M. A. Disorders of representation and control in semantic cognition: effects of familiarity, typicality, and specificity. Neuropsychologia 76, 220–239 (2015).
Kuhnke, P., Kiefer, M. & Hartwigsen, G. Task-dependent recruitment of modality-specific and multimodal regions during conceptual processing. Cereb. Cortex 30, 3938–3959 (2020).
Chiou, R., Humphreys, G. F., Jung, J. & Lambon Ralph, M. A. Controlled semantic cognition relies upon dynamic and flexible interactions between the executive ‘semantic control’ and hub-and-spoke ‘semantic representation’ systems. Cortex 103, 100–116 (2018).
Martin, A. GRAPES—grounding representations in action, perception, and emotion systems: how object properties and categories are represented in the human brain. Psychon. Bull. Rev. 23, 979–990 (2016).
Bengio, Y. & Delalleau, O. in International Conference on Algorithmic Learning Theory (eds Kivinen, J. et al.) 18–36 (Springer, 2011).
Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl. Based Syst. 06, 107–116 (1998).
Saxe, A. M., McClelland, J. L. & Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (2014).
Bar, M. A cortical mechanism for triggering top-down facilitation in visual object recognition. J. Cogn. Neurosci. 15, 600–609 (2003).
Bar, M. et al. Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA 103, 449–454 (2006).
Noonan, K. A., Jefferies, E., Visser, M. & Lambon Ralph, M. A. Going beyond inferior prefrontal involvement in semantic control: evidence for the additional contribution of dorsal angular gyrus and posterior middle temporal cortex. J. Cogn. Neurosci. 25, 1824–1850 (2013).
McKee, J. L., Riesenhuber, M., Miller, E. K. & Freedman, D. J. Task dependence of visual and category representations in prefrontal and inferior temporal cortices. J. Neurosci. 34, 16065–16075 (2014).
Jackson, R. L., Cloutman, L. & Lambon Ralph, M. A. Exploring distinct default mode and semantic networks using a systematic ICA approach. Cortex 113, 279–297 (2019).
Davey, J. et al. Exploring the role of the posterior middle temporal gyrus in semantic cognition: integration of anterior temporal lobe with executive processes. NeuroImage 137, 165–177 (2016).
Rohde, D. L. T. LENS: The Light, Efficient Network Simulator Technical Report No. CMU-CS-99-164 (Carnegie Mellon University, Department of Computer Science, 1999).
SPSS Statistics for Windows v.25.0 (IBM, 2017).
Cloutman, L. L., Binney, R. J., Drakesmith, M., Parker, G. J. M. & Lambon Ralph, M. A. The variation of function across the human insula mirrors its pattern of structural connectivity: evidence from in vivo probabilistic tractography. NeuroImage 59, 3514–3521 (2012).
McIntosh, A. R. Mapping cognition to the brain through neural interactions. Memory 7, 523–548 (1999).
This work was supported by a British Academy Postdoctoral Fellowship awarded to R.L.J. (no. pf170068), a programme grant to M.A.L.R. and T.T.R. from the Medical Research Council (grant no. MR/R023883/1), an Advanced Grant from the European Research Council to M.A.L.R. (GAP: 670428) and Medical Research Council intramural funding (no. MC_UU_00005/18). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing interests.
Peer review information Nature Human Behaviour thanks Peter Hagoort, Andrew Saxe and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Marike Schiffer.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Tables 1–7, Notes 1–10 (including Figs. 1–12) and Methods 1 and 2.
Rights and permissions
About this article
Cite this article
Jackson, R.L., Rogers, T.T. & Lambon Ralph, M.A. Reverse-engineering the cortical architecture for controlled semantic cognition. Nat Hum Behav 5, 774–786 (2021). https://doi.org/10.1038/s41562-020-01034-z
This article is cited by
A topic-based multi-channel attention model under hybrid mode for image caption
Neural Computing and Applications (2022)