The hippocampus as a predictive map

Abstract

A cognitive map has long been the dominant metaphor for hippocampal function, embracing the idea that place cells encode a geometric representation of space. However, evidence for predictive coding, reward sensitivity and policy dependence in place cells suggests that the representation is not purely spatial. We approach this puzzle from a reinforcement learning perspective: what kind of spatial representation is most useful for maximizing future reward? We show that the answer takes the form of a predictive representation. This representation captures many aspects of place cell responses that fall outside the traditional view of a cognitive map. Furthermore, we argue that entorhinal grid cells encode a low-dimensionality basis set for the predictive representation, useful for suppressing noise in predictions and extracting multiscale structure for hierarchical planning.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Updating value with the SR following change in reward.
Figure 2: SR illustration and model comparison.
Figure 3: Behaviorally dependent changes in place fields.
Figure 4: Hippocampal representations in nonspatial task.
Figure 5: Hippocampal representations in spatiotemporal task.
Figure 6: Grid fields in geometric environments.
Figure 7: Grid fragmentation in compartmentalized maze.
Figure 8: Grid fields in a multicompartment environment.

Change history

  • 25 April 2018

    In the version of this article initially published, equation (7) read It should have read The error has been corrected in the HTML and PDF versions of the article.

References

  1. 1

    Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. 2

    Tolman, E.C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. 3

    Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. 4

    Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624 (1993).

    Article  Google Scholar 

  5. 5

    O'Keefe, J. & Nadel, L. The Hippocampus as a Cognitive Map (Clarendon Press, 1978).

  6. 6

    Muller, R.U., Stead, M. & Pach, J. The hippocampus as a cognitive graph. J. Gen. Physiol. 107, 663–694 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. 7

    Penny, W.D., Zeidman, P. & Burgess, N. Forward and backward inference in spatial cognition. PLOS Comput. Biol. 9, e1003383 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. 8

    Foster, D.J., Morris, R.G. & Dayan, P. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10, 1–16 (2000).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. 9

    Gustafson, N.J. & Daw, N.D. Grid cells, place cells, and geodesic generalization for spatial reinforcement learning. PLOS Comput. Biol. 7, e1002235 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. 10

    Hafting, T., Fyhn, M., Molden, S., Moser, M.B. & Moser, E.I. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801–806 (2005).

    CAS  Article  Google Scholar 

  11. 11

    Gershman, S.J., Moore, C.D., Todd, M.T., Norman, K.A. & Sederberg, P.B. The successor representation and temporal context. Neural Comput. 24, 1553–1568 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12

    Russek, E.M., Momennejad, I., Botvinick, M.M., Gershman, S.J. & Daw, N.D. Predictive representations can link model-based reinforcement learning to model-free mechanisms. Preprint at https://doi.org/10.1101/083857 (2017).

  13. 13

    Schapiro, A.C., Turk-Browne, N.B., Norman, K.A. & Botvinick, M.M. Statistical learning of temporal community structure in the hippocampus. Hippocampus 26, 3–8 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14

    Dordek, Y., Meir, R. & Derdikman, D. Extracting grid characteristics from spatially distributed place cell inputs using non-negative PCA. Preprint at https://arxiv.org/abs/1505.03711 (2015).

  15. 15

    Mehta, M.R., Quirk, M.C. & Wilson, M.A. Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron 25, 707–715 (2000).

    Article  CAS  Google Scholar 

  16. 16

    Muller, R.U. & Kubie, J.L. The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. J. Neurosci. 7, 1951–1968 (1987).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. 17

    Skaggs, W.E. & McNaughton, B.L. Spatial firing properties of hippocampal CA1 populations in an environment containing two visually identical regions. J. Neurosci. 18, 8455–8466 (1998).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. 18

    Alvernhe, A., Save, E. & Poucet, B. Local remapping of place cell firing in the Tolman detour task. Eur. J. Neurosci. 33, 1696–1705 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  19. 19

    Hollup, S.A., Molden, S., Donnett, J.G., Moser, M.B. & Moser, E.I. Accumulation of hippocampal place fields at the goal location in an annular water maze task. J. Neurosci. 21, 1635–1644 (2001).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. 20

    Strange, B.A., Witter, M.P., Lein, E.S. & Moser, E.I. Functional organization of the hippocampal longitudinal axis. Nat. Rev. Neurosci. 15, 655–669 (2014).

    Article  CAS  Google Scholar 

  21. 21

    Garvert, M.M., Dolan, R.J. & Behrens, T.E. A map of abstract relational knowledge in the human hippocampal-entorhinal cortex. eLife 6, 17086 (2017).

    Article  Google Scholar 

  22. 22

    Deuker, L., Bellmund, J.L., Navarro Schröder, T. & Doeller, C.F. An event map of memory space in the hippocampus. eLife 5, e16534 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23

    Krupic, J., Bauza, M., Burton, S., Barry, C. & O'Keefe, J. Grid cell symmetry is shaped by environmental geometry. Nature 518, 232–235 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. 24

    Derdikman, D. et al. Fragmentation of grid cell maps in a multicompartment environment. Nat. Neurosci. 12, 1325–1332 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. 25

    Carpenter, F., Manson, D., Jeffery, K., Burgess, N. & Barry, C. Grid cells form a global representation of connected environments. Curr. Biol. 25, 1176–1182 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. 26

    Mazumder, R., Hastie, T. & Tibshirani, R. Spectral regularization algorithms for learning large incomplete matrices. J. Mach. Learn. Res. 11, 2287–2322 (2010).

    PubMed  PubMed Central  Google Scholar 

  27. 27

    Mahadevan, S. & Maggioni, M. Proto-value functions: a Laplacian framework for learning representation and control in markov decision processes. J. Mach. Learn. Res. 8, 2169–2231 (2007).

    Google Scholar 

  28. 28

    Bonnevie, T. et al. Grid cells require excitatory drive from the hippocampus. Nat. Neurosci. 16, 309–317 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. 29

    Hales, J.B. et al. Medial entorhinal cortex lesions only partially disrupt hippocampal place cells and hippocampus-dependent place memory. Cell Rep. 9, 893–901 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. 30

    Solway, A. et al. Optimal behavioral hierarchy. PLoS Comput. Biol. 10, e1003779 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. 31

    Ribas-Fernandes, J.J. et al. A neural signature of hierarchical reinforcement learning. Neuron 71, 370–379 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. 32

    Schlesiger, M.I. et al. The medial entorhinal cortex is necessary for temporal organization of hippocampal neuronal activity. Nat. Neurosci. 18, 1123–1132 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. 33

    Blum, K.I. & Abbott, L.F. A model of spatial map formation in the hippocampus of the rat. Neural Comput. 8, 85–93 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. 34

    Levy, W.B., Hocking, A.B. & Wu, X. Interpreting hippocampal function as recoding and forecasting. Neural Netw. 18, 1242–1264 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  35. 35

    Hassabis, D. & Maguire, E.A. The construction system of the brain. Phil. Trans. R. Soc. Lond. B 364, 1263–1271 (2009).

    Article  Google Scholar 

  36. 36

    Buckner, R.L. The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol. 61, 27–48, C1–C8 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  37. 37

    Barry, C., Hayman, R., Burgess, N. & Jeffery, K.J. Experience-dependent rescaling of entorhinal grids. Nat. Neurosci. 10, 682–684 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. 38

    Fanselow, M.S. From contextual fear to a dynamic view of memory systems. Trends Cogn. Sci. 14, 7–15 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  39. 39

    Wiltgen, B.J., Sanders, M.J., Anagnostaras, S.G., Sage, J.R. & Fanselow, M.S. Context fear learning in the absence of the hippocampus. J. Neurosci. 26, 5484–5491 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. 40

    Maurer, A.P. & McNaughton, B.L. Network and intrinsic cellular mechanisms underlying theta phase precession of hippocampal neurons. Trends Neurosci. 30, 325–333 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. 41

    Johnson, A. & Redish, A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. 42

    Pezzulo, G., van der Meer, M.A., Lansink, C.S. & Pennartz, C.M. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci. 18, 647–657 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  43. 43

    Hasselmo, M.E. & Stern, C.E. Theta rhythm and the encoding and retrieval of space and time. Neuroimage 85, 656–666 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  44. 44

    Ekstrom, A.D., Meltzer, J., McNaughton, B.L. & Barnes, C.A. NMDA receptor antagonism blocks experience-dependent expansion of hippocampal “place fields”. Neuron 31, 631–638 (2001).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. 45

    Hafting, T., Fyhn, M., Bonnevie, T., Moser, M.-B. & Moser, E.I. Hippocampus-independent phase precession in entorhinal grid cells. Nature 453, 1248–1252 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. 46

    Sutton, R.S. DYNA, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin 2, 160–163 (1991).

    Article  Google Scholar 

  47. 47

    Zhang, J., Springenberg, J.T., Boedecker, J. & Burgard, W. Deep reinforcement learning with successor features for navigation across similar environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (2017).

  48. 48

    Momennejad, I. et al. The successor representation in human reinforcement learning. Preprint at https://doi.org/10.1101/083824 (2017).

  49. 49

    Howard, M.W., Fotedar, M.S., Datey, A.V. & Hasselmo, M.E. The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. Psychol. Rev. 112, 75–116 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  50. 50

    Krupic, J., Burgess, N. & O'Keefe, J. Neural representations of location composed of spatially periodic bands. Science 337, 853–857 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. 51

    Sutton, R. & Barto, A. Reinforcement Learning: an Introduction (MIT Press, 1998).

  52. 52

    Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. 53

    Bialek, W. Biophysics: Searching for Principles (Princeton University Press, 2012).

  54. 54

    Weng, J., Zhang, Y. & Hwang, W. Candid covariance-free incremental principal component analysis. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1034–1040 (2003).

    Article  Google Scholar 

  55. 55

    Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 888–905 (2000).

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to T. Behrens, I. Mommenejad, and K. Miller for helpful discussions, and to A. Mathis, H. Sanders, M. Chadwick, and D. Kumaran for comments on an earlier draft of the paper. This research was supported by the NSF Collaborative Research in Computational Neuroscience (CRCNS) Program Grant IIS-120 7833 and The John Templeton Foundation. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the funding agencies.

Author information

Affiliations

Authors

Contributions

All authors conceived the model and wrote the manuscript. Simulations were carried out by K.S.

Corresponding author

Correspondence to Kimberly L Stachenfeld.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Note and Supplementary Figures 1–19 (PDF 13297 kb)

Life Sciences Reporting Summary (PDF 395 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Stachenfeld, K., Botvinick, M. & Gershman, S. The hippocampus as a predictive map. Nat Neurosci 20, 1643–1653 (2017). https://doi.org/10.1038/nn.4650

Download citation

Further reading

Search

Sign up for the Nature Briefing newsletter for a daily update on COVID-19 science.
Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing