Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

If deep learning is the answer, what is the question?

Nature Reviews Neuroscience volume 22, pages 55–67 (2021)Cite this article

Subjects

Abstract

Neuroscience research is undergoing a minor revolution. Recent advances in machine learning and artificial intelligence research have opened up new ways of thinking about neural computation. Many researchers are excited by the possibility that deep neural networks may offer theories of perception, cognition and action for biological brains. This approach has the potential to radically reshape our approach to understanding neural systems, because the computations performed by deep networks are learned from experience, and not endowed by the researcher. If so, how can neuroscientists use deep networks to model and understand biological brains? What is the outlook for neuroscientists who seek to characterize computations or neural codes, or who wish to understand perception, attention, memory and executive functions? In this Perspective, our goal is to offer a road map for systems neuroscience research in the age of deep learning. We discuss the conceptual and methodological challenges of comparing behaviour, learning dynamics and neural representations in artificial and biological systems, and we highlight new research questions that have emerged for neuroscience as a direct consequence of recent advances in machine learning.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Representational equivalence between neural networks and the primate brain.
Fig. 2: Emerging methods for comparing deep learning and the brain.
Fig. 3: Testing principles of learning using perceptual learning paradigms.
Fig. 4: Understanding deep networks using idealized models.
Fig. 5: Developmental trajectories in deep linear neural networks.

Similar content being viewed by others

References

  1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    CAS  PubMed  Google Scholar 

  2. Krizhevsky, A., Hinton, G. E. & Sutskever, I. ImageNet classification with deep convolutional neural networks. Adv. Neural Inform. Process. Syst. 25, 1106–1114 (2012).

    Google Scholar 

  3. Eslami, S. M. A. et al. Neural scene representation and rendering. Science 360, 1204–1210 (2018).

    CAS  PubMed  Google Scholar 

  4. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

    CAS  PubMed  Google Scholar 

  5. Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).

    CAS  PubMed  Google Scholar 

  6. Hassabis, D., Kumaran, D., Summerfield, C. & Botvinick, M. Neuroscience-inspired artificial intelligence. Neuron 95, 245–258 (2017).

    CAS  PubMed  Google Scholar 

  7. Golan, T., Raju, P. C. & Kriegeskorte, N. Controversial stimuli: pitting neural networks against each other as models of human recognition. Preprint at arXiv https://arxiv.org/abs/1911.09288 (2020).

  8. Flesch, T., Balaguer, J., Dekker, R., Nili, H. & Summerfield, C. Comparing continual task learning in minds and machines. Proc. Natl Acad. Sci. USA 115, E10313–E10322 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  9. Geirhos, R. et al. Generalisation in humans and deep neural networks. NeurIPS Proc. (2018).

  10. Zhou, Z. & Firestone, C. Humans can decipher adversarial images. Nat. Commun. 10, 1334 (2019).

    PubMed  PubMed Central  Google Scholar 

  11. Rajalingham, R. et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 38, 7255–7269 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  12. Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).

    CAS  PubMed  Google Scholar 

  13. Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619–8624 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  14. Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).

    PubMed  PubMed Central  Google Scholar 

  15. Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V. & McDermott, J. H. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98, 630–644.e16 (2018).

    CAS  PubMed  Google Scholar 

  16. Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A. & Oliva, A. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Sci. Rep. 6, 27755 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  17. Kar, K., Kubilius, J., Schmidt, K., Issa, E. B. & DiCarlo, J. J. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nat. Neurosci. 22, 974–983 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  18. Kietzmann, T. C. et al. Recurrence is required to capture the representational dynamics of the human visual system. Proc. Natl Acad. Sci. USA 116, 21854–21863 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  19. Guclu, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).

    PubMed  PubMed Central  Google Scholar 

  20. Elsayed, G. F. et al. Adversarial examples that fool both computer vision and time-limited humans. NeurIPS Proc. (2018).

  21. Ullman, S., Assif, L., Fetaya, E. & Harari, D. Atoms of recognition in human and computer vision. Proc. Natl Acad. Sci. USA 113, 2744–2749 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  22. Sinz, F. H., Pitkow, X., Reimer, J., Bethge, M. & Tolias, A. S. Engineering a less artificial intelligence. Neuron 103, 967–979 (2019).

    CAS  PubMed  Google Scholar 

  23. Marblestone, A. H., Wayne, G. & Kording, K. P. Toward an integration of deep learning and neuroscience. Front. Comput. Neurosci. 10, 1–61 (2016).

    Google Scholar 

  24. Kell, A. J. & McDermott, J. H. Deep neural network models of sensory systems: windows onto the role of task constraints. Curr. Opin. Neurobiol. 55, 121–132 (2019).

    CAS  PubMed  Google Scholar 

  25. Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1, 417–446 (2015).

    PubMed  Google Scholar 

  26. Bowers, J. S. Parallel distributed processing theory in the age of deep networks. Trends Cogn. Sci. 21, 950–961 (2017).

    PubMed  Google Scholar 

  27. Cichy, R. M. & Kaiser, D. Deep neural networks as scientific models. Trends Cogn. Sci. 23, 305–317 (2019).

    PubMed  Google Scholar 

  28. Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017).

    PubMed  Google Scholar 

  29. Lindsay, G. W. Convolutional neural networks as a model of the visual system: past, present, and future. J. Cogn. Neurosci. https://doi.org/10.1162/jocn_a_01544 (2020).

  30. Zador, A. M. A critique of pure learning and what artificial neural networks can learn from animal brains. Nat. Commun. 10, 3770 (2019).

    PubMed  PubMed Central  Google Scholar 

  31. Rogers, T. T. & Mcclelland, J. L. Parallel distributed processing at 25: further explorations in the microstructure of cognition. Cogn. Sci. 38, 1024–1077 (2014).

    PubMed  Google Scholar 

  32. Hasson, U., Nastase, S. A. & Goldstein, A. Direct fit to nature: an evolutionary perspective on biological and artificial neural networks. Neuron 105, 416–434 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  33. Richards, B. A. et al. A deep learning framework for neuroscience. Nat. Neurosci. 22, 1761–1770 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  34. Lillicrap, T. P. & Kording, K. P. What does it mean to understand a neural network? Preprint at arXiv https://arxiv.org/abs/1907.06374 (2019).

  35. Saxe, A., Bhand, M., Mudur, R., Suresh, B. & Ng, A. Y. Unsupervised learning models of primary cortical receptive fields and receptive field plasticity. Adv. Neural Inform. Process. Syst. 25, 1971–1979 (2011).

    Google Scholar 

  36. Stevenson, I. H. & Kording, K. P. How advances in neural recording affect data analysis. Nat. Neurosci. 14, 139–142 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Saxena, S. & Cunningham, J. P. Towards the neural population doctrine. Curr. Opin. Neurobiol. 55, 103–111 (2019).

    CAS  PubMed  Google Scholar 

  38. Fusi, S., Miller, E. K. & Rigotti, M. Why neurons mix: high dimensionality for higher cognition. Curr. Opin. Neurobiol. 37, 66–74 (2016).

    CAS  PubMed  Google Scholar 

  39. Rigotti, M. et al. The importance of mixed selectivity in complex cognitive tasks. Nature 497, 585–590 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  40. Johnston, W. J., Palmer, S. E. & Freedman, D. J. Nonlinear mixed selectivity supports reliable neural computation. PLoS Comput. Biol. 16, e1007544 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  41. Raposo, D., Kaufman, M. T. & Churchland, A. K. A category-free neural population supports evolving demands during decision-making. Nat. Neurosci. 17, 1784–1792 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  42. Higgins, I. et al. Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons. Preprint at arXiv https://arxiv.org/abs/2006.14304 (2020).

  43. Park, I. M., Meister, M. L. R., Huk, A. C. & Pillow, J. W. Encoding and decoding in parietal cortex during sensorimotor decision-making. Nat. Neurosci. 17, 1395–1403 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  44. Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, W. T. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503, 78–84 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Chaisangmongkon, W., Swaminathan, S. K., Freedman, D. J. & Wang, X.-J. Computing by robust transience: how the fronto-parietal network performs sequential, category-based decisions. Neuron 93, 1504–1517.e4 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  46. Engel, T. A., Chaisangmongkon, W., Freedman, D. J. & Wang, X. J. Choice-correlated activity fluctuations underlie learning of neuronal category representation. Nat. Commun. 6, 6454 (2015).

    CAS  PubMed  Google Scholar 

  47. Remington, E. D., Egger, S. W., Narain, D., Wang, J. & Jazayeri, M. A dynamical systems perspective on flexible motor timing. Trends Cogn. Sci. 22, 938–952 (2018).

    PubMed  PubMed Central  Google Scholar 

  48. Remington, E. D., Narain, D., Hosseini, E. A. & Jazayeri, M. Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron 98, 1005–1019.e5 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  49. Orhan, A. E. & Ma, W. J. A diverse range of factors affect the nature of neural representations underlying short-term memory. Nat. Neurosci. 22, 275–283 (2019).

    CAS  PubMed  Google Scholar 

  50. Masse, N. Y., Rosen, M. C. & Freedman, D. J. Reevaluating the role of persistent neural activity in short-term memory. Trends Cogn. Sci. 24, 242–258 (2020).

    PubMed  PubMed Central  Google Scholar 

  51. Masse, N. Y., Yang, G. R., Song, H. F., Wang, X.-J. & Freedman, D. J. Circuit mechanisms for the maintenance and manipulation of information in working memory. Nat. Neurosci. 22, 1159–1167 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  52. Lindsey, J., Ocko, S. A., Ganguli, S. & Deny, S. A unified theory of early visual representations from retina to cortex through anatomically constrained deep CNNs. Preprint at bioRxiv https://doi.org/10.1101/511535 (2019).

  53. Rahwan, I. et al. Machine behaviour. Nature 568, 477–486 (2019).

    CAS  PubMed  Google Scholar 

  54. Thompson, J. A. F., Bengio, Y., Formisano, E. & Schönwiesner, M. How can deep learning advance computational modeling of sensory information processing? Preprint at arXiv https://arxiv.org/abs/1810.08651 (2018).

  55. Schrimpf, M. et al. Brain-score: which artificial neural network for object recognition is most brain-like? Preprint at bioRxiv https://doi.org/10.1101/407007 (2018).

    Article  Google Scholar 

  56. Kriegeskorte, N. Representational similarity analysis – connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

    PubMed  PubMed Central  Google Scholar 

  57. Bashivan, P., Kar, K. & DiCarlo, J. J. Neural population control via deep image synthesis. Science 364, eaav9436 (2019).

    CAS  PubMed  Google Scholar 

  58. Ponce, C. R. et al. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell 177, 999–1009.e10 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  59. Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. Neuroscience needs behavior: correcting a reductionist bias. Neuron 93, 480–490 (2017).

    CAS  PubMed  Google Scholar 

  60. Gomez-Marin, A. & Ghazanfar, A. A. The life of behavior. Neuron 104, 25–36 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  61. Rich, A. S. & Gureckis, T. M. Lessons for artificial intelligence from the study of natural stupidity. Nat. Mach. Intell. 1, 174–180 (2019).

    Google Scholar 

  62. Shenhav, A., Botvinick, M. M. & Cohen, J. D. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79, 217–240 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  63. Deen, B. et al. Organization of high-level visual cortex in human infants. Nat. Commun. 8, 13995 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  64. Op de Beeck, H. P., Pillet, I. & Ritchie, J. B. Factors determining where category-selective areas emerge in visual cortex. Trends Cogn. Sci. 23, 784–797 (2019).

    PubMed  Google Scholar 

  65. Arcaro, M. J., Schade, P. F., Vincent, J. L., Ponce, C. R. & Livingstone, M. S. Seeing faces is necessary for face-domain formation. Nat. Neurosci. 20, 1404–1412 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  66. Olshausen, B. A. & Field, D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996).

    CAS  PubMed  Google Scholar 

  67. Simoncelli, E. P. & Olshausen, B. A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001).

    CAS  PubMed  Google Scholar 

  68. Friston, K. J. The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138 (2010).

    CAS  PubMed  Google Scholar 

  69. Kingma, D. P. & Welling, M. Auto-encoding variational bayes. Preprint at arXiv https://arxiv.org/abs/1312.6114 (2014).

  70. Burgess, C. P. et al. MONet: unsupervised scene decomposition and representation. Preprint at arXiv https://arxiv.org/abs/1901.11390 (2019).

  71. Lillicrap, T. P., Cownden, D., Tweed, D. B. & Akerman, C. J. Random synaptic feedback weights support error backpropagation for deep learning. Nat. Commun. 7, 13276 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  72. Detorakis, G., Bartley, T. & Neftci, E. Contrastive Hebbian learning with random feedback weights. Neural Netw. https://doi.org/10.1016/j.neunet.2019.01.008 (2019).

  73. Saxe, A. Deep Linear Networks: A Theory of Learning in the Brain and Mind. Thesis, Stanford Univ. (2015).

  74. Wenliang, L. K. & Seitz, A. R. Deep neural networks for modeling visual perceptual learning. J. Neurosci. 38, 6028–6044 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  75. Whittington, J. C. R. & Bogacz, R. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Comput. 29, 1229–1262 (2017).

    PubMed  PubMed Central  Google Scholar 

  76. Murphy, G. L. The Big Book of Concepts (MIT Press, 2002).

  77. Graves, A. et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016).

    PubMed  Google Scholar 

  78. Wayne, G. et al. Unsupervised predictive memory in a goal-directed agent. Preprint at arXiv https://arxiv.org/abs/1803.10760 (2018).

  79. Moser, E. I., Kropff, E. & Moser, M.-B. Place cells, grid cells, and the brain’s spatial representation system. Annu. Rev. Neurosci. 31, 69–89 (2008).

    CAS  PubMed  Google Scholar 

  80. Fee, M. S., Kozhevnikov, A. A. & Hahnloser, R. H. R. Neural mechanisms of vocal sequence generation in the songbird. Ann. N. Y. Acad. Sci. 1016, 153–170 (2004).

    PubMed  Google Scholar 

  81. Hanes, D. P. & Schall, J. D. Neural control of voluntary movement initiation. Science 274, 427–430 (1996).

    CAS  PubMed  Google Scholar 

  82. Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a mind: statistics, structure, and abstraction. Science 331, 1279–1285 (2011).

    CAS  PubMed  Google Scholar 

  83. Tervo, D. G. R., Tenenbaum, J. B. & Gershman, S. J. Toward the neural implementation of structure learning. Curr. Opin. Neurobiol. 37, 99–105 (2016).

    CAS  PubMed  Google Scholar 

  84. Behrens, T. E. J. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).

    CAS  PubMed  Google Scholar 

  85. Penn, D. C., Holyoak, K. J. & Povinelli, D. J. Darwin’s mistake: explaining the discontinuity between human and nonhuman minds. Behav. Brain Sci. 31, 109–130 (2008).

    PubMed  Google Scholar 

  86. Schuck, N. W. & Niv, Y. Sequential replay of nonspatial task states in the human hippocampus. Science 364, eaaw5181 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  87. Kurth-Nelson, Z., Economides, M., Dolan, R. J. & Dayan, P. Fast sequences of non-spatial state representations in humans. Neuron 91, 194–204 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  88. Liu, Y., Dolan, R. J., Kurth-Nelson, Z. & Behrens, T. E. J. Human replay spontaneously reorganizes experience. Cell 178, 640–652.e14 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  89. Barron, H. C. et al. Unmasking latent inhibitory connections in human cortex to reveal dormant cortical memories. Neuron 90, 191–203 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  90. Koolschijn, R. S. et al. The hippocampus and neocortical inhibitory engrams protect against memory interference. Neuron 101, 528–541.e6 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  91. Doeller, C. F., Barry, C. & Burgess, N. Evidence for grid cells in a human memory network. Nature 463, 657–661 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  92. Constantinescu, A. O., OReilly, J. X. & Behrens, T. E. J. Organizing conceptual knowledge in humans with a gridlike code. Science 352, 1464–1468 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  93. Tsividis, P. A., Pouncy, T., Xu, J., Tenenbaum, J. B. & Gershman, S. J. Human learning in Atari (AAAI, 2017).

  94. Schapiro, A. C., Rogers, T. T., Cordova, N. I., Turk-Browne, N. B. & Botvinick, M. M. Neural representations of events arise from temporal community structure. Nat. Neurosci. 16, 486–492 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  95. O’Keefe, J. & Dostrovsky, J. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34, 171–175 (1971).

    PubMed  Google Scholar 

  96. Baraduc, P., Duhamel, J.-R. & Wirth, S. Schema cells in the macaque hippocampus. Science 363, 635–639 (2019).

    CAS  PubMed  Google Scholar 

  97. Miyashita, Y. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature 335, 817–820 (1988).

    CAS  PubMed  Google Scholar 

  98. Garvert, M. M., Dolan, R. J. & Behrens, T. E. A map of abstract relational knowledge in the human hippocampal-entorhinal cortex. eLife 6, e17086 (2017).

    PubMed  PubMed Central  Google Scholar 

  99. Schapiro, A. C., Kustner, L. V. & Turk-Browne, N. B. Shaping of object representations in the human medial temporal lobe based on temporal regularities. Curr. Biol. 22, 1622–1627 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  100. Schapiro, A. C., Turk-Browne, N. B., Norman, K. A. & Botvinick, M. M. Statistical learning of temporal community structure in the hippocampus:. Hippocampus 26, 3–8 (2016).

    PubMed  Google Scholar 

  101. Schlichting, M. L., Mumford, J. A. & Preston, A. R. Learning-related representational changes reveal dissociable integration and separation signatures in the hippocampus and prefrontal cortex. Nat. Commun. 6, 8151 (2015).

    CAS  PubMed  Google Scholar 

  102. Zeithamova, D., Dominick, A. L. & Preston, A. R. Hippocampal and ventral medial prefrontal activation during retrieval-mediated learning supports novel inference. Neuron 75, 168–179 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  103. Park, S. A., Miller, D. S., Nili, H., Ranganath, C. & Boorman, E. D. Map making: constructing, combining, and inferring on abstract cognitive maps. Neuron 107, 1226–1238.e8 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  104. Kumaran, D., Banino, A., Blundell, C., Hassabis, D. & Dayan, P. Computations underlying social hierarchy learning: distinct neural mechanisms for updating and representing self-relevant information. Neuron 92, 1135–1147 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  105. Baram, A. B., Muller, T. H., Nili, H., Garvert, M. & Behrens, T. E. J. Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems. Preprint at bioRxiv https://doi.org/10.1101/827253 (2019).

  106. Dolan, R. J. & Dayan, P. Goals and habits in the brain. Neuron 80, 312–325 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  107. Higgins, I. et al. Early visual concept learning with unsupervised deep learning. Preprint at arXiv https://arxiv.org/abs/1606.05579 (2016).

  108. Higgins, I. et al. SCAN: learning hierarchical compositional visual concepts. Preprint at arXiv https://arxiv.org/abs/1707.03389 (2018).

  109. Hessel, M. et al. Rainbow: combining improvements in deep reinforcement learning (AAAI, 2018).

  110. Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. Design principles of the hippocampal cognitive map. Int. Conf. Neural Inform. Process. Syst. 2, 2528–2536 (2014).

    Google Scholar 

  111. Whittington, J. C. et al. The Tolman-Eichenbaum machine: unifying space and relational memory through generalisation in the hippocampal formation. Preprint at bioRxiv https://doi.org/10.1101/770495 (2019).

  112. Bellmund, J. L. S., Gärdenfors, P., Moser, E. I. & Doeller, C. F. Navigating cognition: spatial codes for human thinking. Science 362, eaat6766 (2018).

    PubMed  Google Scholar 

  113. Cueva, C. J. & Wei, X.-X. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. Preprint at arXiv https://arxiv.org/abs/1803.07770 (2018).

  114. Banino, A. et al. Vector-based navigation using grid-like representations in artificial agents. Nature 557, 429–433 (2018).

    CAS  PubMed  Google Scholar 

  115. Parisi, G. I., Kemker, R., Part, J. L., Kanan, C. & Wermter, S. Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019).

    PubMed  Google Scholar 

  116. Schapiro, A. C., Turk-Browne, N. B., Botvinick, M. M. & Norman, K. A. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Phil. Trans. R. Soc. B 372, 20160049 (2017).

    PubMed  PubMed Central  Google Scholar 

  117. French, R. Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3, 128–135 (1999).

    CAS  PubMed  Google Scholar 

  118. McCloskey, M. & Cohen, N. J. Catastrophic interference in connectionist networks: the sequential learning problem. in Psychology of Learning and Motivation Vol. 24 109–165 (Academic, 1989).

  119. McClelland, J. L., McNaughton, B. L. & O’Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102, 419–457 (1995).

    PubMed  Google Scholar 

  120. O’Reilly, R. C., Bhattacharyya, R., Howard, M. D. & Ketz, N. Complementary learning systems. Cogn. Sci. 38, 1229–1248 (2014).

    PubMed  Google Scholar 

  121. Tulving, E. Episodic memory: from mind to brain. Annu. Rev. Psychol. 53, 1–25 (2002).

    PubMed  Google Scholar 

  122. Carr, M. F., Jadhav, S. P. & Frank, L. M. Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nat. Neurosci. 14, 147–153 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  123. Zola-Morgan, S. & Squire, L. The primate hippocampal formation: evidence for a time-limited role in memory storage. Science 250, 288–290 (1990).

    CAS  PubMed  Google Scholar 

  124. Yonelinas, A. P. The nature of recollection and familiarity: a review of 30 years of research. J. Mem. Lang. 46, 441–517 (2002).

    Google Scholar 

  125. van de Ven, G. M. & Tolias, A. S. Generative replay with feedback connections as a general strategy for continual learning. Preprint at arXiv https://arxiv.org/abs/1809.10635 (2019).

  126. Kumaran, D., Hassabis, D. & McClelland, J. L. What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends Cogn. Sci. 20, 512–534 (2016).

    PubMed  Google Scholar 

  127. Qian, T. & Aslin, R. N. Learning bundles of stimuli renders stimulus order as a cue, not a confound. Proc. Natl Acad. Sci. USA 111, 14400–14405 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  128. Collins, A. G. E. & Frank, M. J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol. Rev. 120, 190–229 (2013).

    PubMed  PubMed Central  Google Scholar 

  129. Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl Acad. Sci. USA 114, 3521–3526 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  130. Zenke, F., Poole, B. & Ganguli, S. Continual learning through synaptic intelligence. Proc. Mach. Learn. Res. 70, 3987–3995 (2017).

    PubMed  PubMed Central  Google Scholar 

  131. Masse, N. Y., Grant, G. D. & Freedman, D. J. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proc. Natl Acad. Sci. USA 115, E10467–E10475 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  132. Zeng, G., Chen, Y., Cui, B. & Yu, S. Continual learning of context-dependent processing in neural networks. Nat. Mach. Intell. 1, 364–372 (2019).

    Google Scholar 

  133. Bouchacourt, F., Palminteri, S., Koechlin, E. & Ostojic, S. Temporal chunking as a mechanism for unsupervised learning of task-sets. eLife 9, e50469 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  134. Harvey, C. D., Coen, P. & Tank, D. W. Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature 484, 62–68 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  135. Rule, M. E., O’Leary, T. & Harvey, C. D. Causes and consequences of representational drift. Curr. Opin. Neurobiol. 58, 141–147 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  136. Musslick, S. et al. Multitasking capability versus learning efficiency in neural network architectures. in Annual Meeting of the Cognitive Science Society 829–834 (Cognitive Science Society, 2017).

  137. Bahri, Y. et al. Statistical mechanics of deep learning. Annu. Rev. Condens. Matter Phys. 11, 501–528 (2020).

    Google Scholar 

  138. Saxe, A. M., McClelland, J. L. & Ganguli, S. A mathematical theory of semantic development in deep neural networks. Proc. Natl Acad. Sci. USA 116, 11537–11546 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  139. Saxe, A. M., McClelland, J. L. & Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. Preprint at arXiv https://arxiv.org/abs/1312.6120 (2014).

  140. Seung, H. S., Sompolinsky, H. & Tishby, N. Statistical mechanics of learning from examples. Phys. Rev. A 45, 6056–6091 (1992).

    CAS  PubMed  Google Scholar 

  141. Goldt, S., Advani, M. S., Saxe, A. M., Krzakala, F. & Zdeborová, L. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. NeurIPS Proc. (2019).

  142. Jacot, A., Gabriel, F. & Hongler, C. Neural tangent kernel: convergence and generalization in neural networks. NeurIPS Proc. (2018).

  143. Lee, J. et al. Wide neural networks of any depth evolve as linear models under gradient descent. NeurIPS Proc. (2019).

  144. Advani, M. S. & Saxe, A. M. High-dimensional dynamics of generalization error in neural networks. Preprint at arXiv https://arxiv.org/abs/1710.03667 (2017).

  145. Krogh, A. & Hertz, J. A. Generalization in a linear perceptron in the presence of noise. J. Phys. Math. Gen. 25, 1135–1147 (1992).

    Google Scholar 

  146. Dauphin, Y. et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NeurIPS Proc. (2014).

  147. Carey, S. Précis of ‘The Origin of Concepts’. Behav. Brain Sci. 34, 113–124 (2011).

    PubMed  PubMed Central  Google Scholar 

  148. Rogers, T. T. & McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach (MIT Press, 2004).

  149. Walker, E. Y. et al. Inception loops discover what excites neurons most using deep predictive models. Nat. Neurosci. 22, 2060–2065 (2019).

    CAS  PubMed  Google Scholar 

  150. Schoups, A., Vogels, R., Qian, N. & Orban, G. Practising orientation identification improves orientation coding in V1 neurons. Nature 412, 549–553 (2001).

    CAS  PubMed  Google Scholar 

  151. Belkin, M., Hsu, D., Ma, S. & Mandal, S. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proc. Natl Acad. Sci. USA 116, 15849–15854 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work was supported by generous funding from the European Research Council (ERC Consolidator award to C.S. and Special Grant Agreement 3 of the Human Brain Project) and a Sir Henry Dale Fellowship to A.S. from the Wellcome Trust and Royal Society (grant number 216386/Z/19/Z). A.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains programme.

Author information

Authors and Affiliations

  1. Department of Experimental Psychology, University of Oxford, Oxford, UK

    Andrew Saxe, Stephanie Nelli & Christopher Summerfield

Authors
  1. Andrew Saxe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephanie Nelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher Summerfield
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to all aspects of the article.

Corresponding authors

Correspondence to Andrew Saxe, Stephanie Nelli or Christopher Summerfield.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Neuroscience thanks M. Jazayeri and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saxe, A., Nelli, S. & Summerfield, C. If deep learning is the answer, what is the question?. Nat Rev Neurosci 22, 55–67 (2021). https://doi.org/10.1038/s41583-020-00395-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41583-020-00395-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing