Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problem-solving

Nature Neuroscience volume 26, pages 879–890 (2023)Cite this article

Subjects

Abstract

Learning-to-learn, a progressive speedup of learning while solving a series of similar problems, represents a core process of knowledge acquisition that draws attention in both neuroscience and artificial intelligence. To investigate its underlying brain mechanism, we trained a recurrent neural network model on arbitrary sensorimotor mappings known to depend on the prefrontal cortex. The network displayed an exponential time course of accelerated learning. The neural substrate of a schema emerges within a low-dimensional subspace of population activity; its reuse in new problems facilitates learning by limiting connection weight changes. Our work highlights the weight-driven modifications of the vector field, which determines the population trajectory of a recurrent network and behavior. Such plasticity is especially important for preserving and reusing the learned schema in spite of undesirable changes of the vector field due to the transition to learning a new problem; the accumulated changes across problems account for the learning-to-learn dynamics.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: RNNs trained on delayed sensorimotor association problems exhibit learning-to-learn.
Fig. 2: Neural representations of decision and choice are shared across problems.
Fig. 3: Manifold perturbations reveal that reusing the schematic decision manifold facilitates learning.
Fig. 4: Learned trajectories emerge from VFCs.
Fig. 5: Weight-driven and state-driven VFCs differentially contribute to population activity change.
Fig. 6: The magnitude of recurrent weight changes explains both the magnitude of the weight-driven VFC and the number of trials to learn a problem.
Fig. 7: Accumulation of weight changes progressively improves invariance of existing representations to learning.
Fig. 8: Learning-to-learn is a process with three time scales.

Similar content being viewed by others

Data availability

Data files, including pre-trained networks, are available for further analyses on GitHub (https://github.com/xjwanglab/learning-2-learn) in Python and MATLAB readable formats.

Code availability

All training and analysis codes are available on GitHub (https://github.com/xjwanglab/learning-2-learn).

References

  1. Piaget, J. The Language and Thought of the Child (Harcourt Brace, 1926).

  2. Bartlett, F.C. Remembering: A Study in Experimental and Social Psychology (Cambridge University Press, 1932).

  3. Rumelhart, D. E. Schemata: the building blocks of cognition. in Theoretical Issues in Reading Comprehension 33–58 (Erlbaum Associates, 1980).

  4. Gilboa, A. & Marlatte, H. Neurobiology of schemas and schema-mediated memory. Trends Cogn. Sci. 21, 618–631 (2017).

    Article  PubMed  Google Scholar 

  5. Chi, M. T., Glaser, R. & Rees, E. Expertise in problem solving. https://www.public.asu.edu/~mtchi/papers/ChiGlaserRees.pdf (1982).

  6. Harlow, H. F. The formation of learning sets. Psychological Review 56, 51–65 (1949).

    Article  CAS  PubMed  Google Scholar 

  7. Lewis, P. A. & Durrant, S. J. Overlapping memory replay during sleep builds cognitive schemata. Trends Cogn. Sci. 15, 343–351 (2011).

    Article  PubMed  Google Scholar 

  8. Behrens, T. E. J. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).

    Article  CAS  PubMed  Google Scholar 

  9. Preston, A. R. & Eichenbaum, H. Interplay of hippocampus and prefrontal cortex in memory. Curr. Biol. 23, R764–R773. (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Wang, S.-H. & Morris, R. G. Hippocampal–neocortical interactions in memory formation, consolidation, and reconsolidation. Annu. Rev. Psychol. 61, 49–79 (2010).

    Article  PubMed  Google Scholar 

  11. McKenzie, S. et al. Hippocampal representation of related and opposing memories develop within distinct, hierarchically organized neural schemas. Neuron 83, 202–215 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Bernardi, S. et al. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell 183, 954–967 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Braun, D. A., Mehring, C. & Wolpert, D. M. Structure learning in action. Behav. Brain Res. 206, 157–165 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Finn, C., Abbeel, P. & Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning 1126–1135 (PMLR, 2017).

  15. Wang, J. X. et al. Prefrontal cortex as a meta-reinforcement learning system. Nat. Neurosci. 21, 860–868 (2018).

    Article  CAS  PubMed  Google Scholar 

  16. Passingham, R. The Frontal Lobes and Voluntary Action (Oxford University Press, 1995).

  17. Asaad, W. F., Rainer, G. & Miller, E. K. Neural activity in the primate prefrontal cortex during associative learning. Neuron 21, 1399–1407 (1998).

    Article  CAS  PubMed  Google Scholar 

  18. Fusi, S., Asaad, W. F., Miller, E. K. & Wang, X.-J. A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron 54, 319–333 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Cromer, J. A., Machon, M. & Miller, E. K. Rapid association learning in the primate prefrontal cortex in the absence of behavioral reversals. J. Cogn. Neurosci. 23, 1823–1828 (2011).

    Article  PubMed  Google Scholar 

  20. Bussey, T. J., Wise, S. P. & Murray, E. A. Interaction of ventral and orbital prefrontal cortex with inferotemporal cortex in conditional visuomotor learning. Behav. Neurosci. 116, 703–715 (2002).

    Article  PubMed  Google Scholar 

  21. Petrides, M. Deficits on conditional associative-learning tasks after frontal- and temporal-lobe lesions in man. Neuropsychologia 23, 601–614 (1985).

    Article  CAS  PubMed  Google Scholar 

  22. Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364, 255 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  23. Zhou, J. et al. Evolving schema representations in orbitofrontal ensembles during learning. Nature 590, 606–611 (2021).

  24. Sadtler, P. T. et al. Neural constraints on learning. Nature 512, 423–426 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Bao, P., She, L., McGill, M. & Tsao, D. Y. A map of object space in primate inferotemporal cortex. Nature 583, 103–108 (2020).

  26. Eacott, M. & Gaffan, D. Inferotemporal–frontal disconnection: the uncinate fascicle and visual associative learning in monkeys. Eur. J. Neurosci. 4, 1320–1332 (1992).

    Article  PubMed  Google Scholar 

  27. Kobak, D. et al. Demixed principal component analysis of neural population data. eLife 5, e10989 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  28. Anderson, R. C., Spiro, R. J. & Anderson, M. C. Schemata as scaffolding for the representation of information in connected discourse. Am. Educ. Res. J. 15, 433–440 (1978).

    Article  Google Scholar 

  29. Rumelhart, D. E. & Norman, D. A. Accretion, tuning and restructuring: three modes of learning. https://www.dsoergel.com/UBLIS571DS-06.1a-1Reading10RumelhartAccretionTuningAndRestructuring.pdf (1978).

  30. Thorndyke, P. W. & Hayes-Roth, B. The use of schemata in the acquisition and transfer of knowledge. Cogn. Psychol. 11, 82–106 (1979).

    Article  Google Scholar 

  31. Kaufman, M. T., Churchland, M. M., Ryu, S. I. & Shenoy, K. V. Cortical activity in the null space: permitting preparation without movement. Nat. Neurosci. 17, 440–448 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Strogatz, S. H. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering 2nd edn (Taylor & Francis, 2016).

  33. Vyas, S., Golub, M. D., Sussillo, D. & Shenoy, K. V. Computation through neural population dynamics. Annu. Rev. Neurosci. 43, 249–275 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Long, P. M. & Sedghi, H. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations (ICLR, 2020).

  35. Gouk, H., Hospedales, T. M. & Pontil, M. Distance-based regularisation of deep networks for fine-tuning. In International Conference on Learning Representations (ICLR, 2021).

  36. Kaufman, M. T. et al. The largest response component in the motor cortex reflects movement timing but not movement type. eNeuro 3, ENEURO.0085-16.2016 (2016).

  37. Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a mind: statistics, structure, and abstraction. Science 331, 1279–1285 (2011).

    Article  CAS  PubMed  Google Scholar 

  38. Sussillo, D. & Barak, O. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25, 626–649 (2013).

    Article  PubMed  Google Scholar 

  39. Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F. & Ostojic, S. The role of population structure in computations through neural dynamics. Nat. Neurosci. 25, 783–794 (2022).

  40. Wang, S.-H., Tse, D. & Morris, R. G. Anterior cingulate cortex in schema assimilation and expression. Learn. Mem. 19, 315–318 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  41. Sehgal, M., Song, C., Ehlers, V. L. & Moyer, J. R. Jr. Learning to learn—intrinsic plasticity as a metaplasticity mechanism for memory formation. Neurobiol. Learn. Mem. 105, 186–199 (2013).

  42. Bittner, K. C., Milstein, A. D., Grienberger, C., Romani, S. & Magee, J. C. Behavioral time scale synaptic plasticity underlies CA1 place fields. Science 357, 1033–1036 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Wang, X.-J. Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36, 955–968 (2002).

    Article  CAS  PubMed  Google Scholar 

  44. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR, 2015).

  45. Abadi, M. et al. TensorFlow: a system for large-scale machine learning. USENIX Symposium on Operating Systems Design and Implementation 16, 265–283 (2016).

  46. Stewart, G. W. The efficient generation of random orthogonal matrices with an application to condition estimators. SIAM J. Numer. Anal. 17, 403–409 (1980).

    Article  Google Scholar 

  47. Krogh, A. & Hertz, J. A. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems 950–957 (NeurIPS, 1991).

  48. Merity, S., McCann, B. & Socher, R. Revisiting activation regularization for language RNNs. In International Conference on Machine Learning’s Workshop on Learning to Generate Natural Language (ICML, 2017).

  49. Gao, P. et al. A theory of multineuronal dimensionality, dynamics and measurement. Preprint at https://www.biorxiv.org/content/10.1101/214262v2 (2017).

Download references

Acknowledgements

We thank A. L. Fairhall, I. Skelin, J. J. Lin, B. Doiron, G. R. Yang, N. Y. Masse, U. P. Obilinovic, L. Y. Tian, D. V. Buonomano, J. Jaramillo, J. E. Fitzgerald and H. Sompolinksy for fruitful discussions and Y. Liu, K. Berlemont, A. Battista and P. Theodoni for critical comments on the manuscript. This work was supported by National Institute of Health U-19 program grant 5U19NS107609-03 and Office of Naval Research grant N00014-23-1-2040.

Author information

Authors and Affiliations

  1. Center for Neural Science, New York University, New York, NY, USA

    Vishwa Goudar & Xiao-Jing Wang

  2. Department of Neurobiology, University of Chicago, Chicago, IL, USA

    Barbara Peysakhovich & David J. Freedman

  3. Department of Physiology and Biophysics, University of Washington School of Medicine, Seattle, WA, USA

    Elizabeth A. Buffalo

  4. Washington National Primate Research Center, Seattle, WA, USA

    Elizabeth A. Buffalo

Authors
  1. Vishwa Goudar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barbara Peysakhovich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David J. Freedman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elizabeth A. Buffalo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiao-Jing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

B.P., D.J.F., E.A.B. and X.-J.W. designed the study. V.G. performed the research. V.G. and X.-J.W. wrote the manuscript.

Corresponding author

Correspondence to Xiao-Jing Wang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Neuroscience thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes, Supplementary Methods, Supplementary Discussion and Supplementary Figs. 1–10.

Reporting Summary

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goudar, V., Peysakhovich, B., Freedman, D.J. et al. Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problem-solving. Nat Neurosci 26, 879–890 (2023). https://doi.org/10.1038/s41593-023-01293-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-023-01293-9

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing