A cognitive map has long been the dominant metaphor for hippocampal function, embracing the idea that place cells encode a geometric representation of space. However, evidence for predictive coding, reward sensitivity and policy dependence in place cells suggests that the representation is not purely spatial. We approach this puzzle from a reinforcement learning perspective: what kind of spatial representation is most useful for maximizing future reward? We show that the answer takes the form of a predictive representation. This representation captures many aspects of place cell responses that fall outside the traditional view of a cognitive map. Furthermore, we argue that entorhinal grid cells encode a low-dimensionality basis set for the predictive representation, useful for suppressing noise in predictions and extracting multiscale structure for hierarchical planning.
Subscribe to Journal
Get full journal access for 1 year
only $17.42 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).
Tolman, E.C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).
Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624 (1993).
O'Keefe, J. & Nadel, L. The Hippocampus as a Cognitive Map (Clarendon Press, 1978).
Muller, R.U., Stead, M. & Pach, J. The hippocampus as a cognitive graph. J. Gen. Physiol. 107, 663–694 (1996).
Penny, W.D., Zeidman, P. & Burgess, N. Forward and backward inference in spatial cognition. PLOS Comput. Biol. 9, e1003383 (2013).
Foster, D.J., Morris, R.G. & Dayan, P. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10, 1–16 (2000).
Gustafson, N.J. & Daw, N.D. Grid cells, place cells, and geodesic generalization for spatial reinforcement learning. PLOS Comput. Biol. 7, e1002235 (2011).
Hafting, T., Fyhn, M., Molden, S., Moser, M.B. & Moser, E.I. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801–806 (2005).
Gershman, S.J., Moore, C.D., Todd, M.T., Norman, K.A. & Sederberg, P.B. The successor representation and temporal context. Neural Comput. 24, 1553–1568 (2012).
Russek, E.M., Momennejad, I., Botvinick, M.M., Gershman, S.J. & Daw, N.D. Predictive representations can link model-based reinforcement learning to model-free mechanisms. Preprint at https://doi.org/10.1101/083857 (2017).
Schapiro, A.C., Turk-Browne, N.B., Norman, K.A. & Botvinick, M.M. Statistical learning of temporal community structure in the hippocampus. Hippocampus 26, 3–8 (2016).
Dordek, Y., Meir, R. & Derdikman, D. Extracting grid characteristics from spatially distributed place cell inputs using non-negative PCA. Preprint at https://arxiv.org/abs/1505.03711 (2015).
Mehta, M.R., Quirk, M.C. & Wilson, M.A. Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron 25, 707–715 (2000).
Muller, R.U. & Kubie, J.L. The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. J. Neurosci. 7, 1951–1968 (1987).
Skaggs, W.E. & McNaughton, B.L. Spatial firing properties of hippocampal CA1 populations in an environment containing two visually identical regions. J. Neurosci. 18, 8455–8466 (1998).
Alvernhe, A., Save, E. & Poucet, B. Local remapping of place cell firing in the Tolman detour task. Eur. J. Neurosci. 33, 1696–1705 (2011).
Hollup, S.A., Molden, S., Donnett, J.G., Moser, M.B. & Moser, E.I. Accumulation of hippocampal place fields at the goal location in an annular water maze task. J. Neurosci. 21, 1635–1644 (2001).
Strange, B.A., Witter, M.P., Lein, E.S. & Moser, E.I. Functional organization of the hippocampal longitudinal axis. Nat. Rev. Neurosci. 15, 655–669 (2014).
Garvert, M.M., Dolan, R.J. & Behrens, T.E. A map of abstract relational knowledge in the human hippocampal-entorhinal cortex. eLife 6, 17086 (2017).
Deuker, L., Bellmund, J.L., Navarro Schröder, T. & Doeller, C.F. An event map of memory space in the hippocampus. eLife 5, e16534 (2016).
Krupic, J., Bauza, M., Burton, S., Barry, C. & O'Keefe, J. Grid cell symmetry is shaped by environmental geometry. Nature 518, 232–235 (2015).
Derdikman, D. et al. Fragmentation of grid cell maps in a multicompartment environment. Nat. Neurosci. 12, 1325–1332 (2009).
Carpenter, F., Manson, D., Jeffery, K., Burgess, N. & Barry, C. Grid cells form a global representation of connected environments. Curr. Biol. 25, 1176–1182 (2015).
Mazumder, R., Hastie, T. & Tibshirani, R. Spectral regularization algorithms for learning large incomplete matrices. J. Mach. Learn. Res. 11, 2287–2322 (2010).
Mahadevan, S. & Maggioni, M. Proto-value functions: a Laplacian framework for learning representation and control in markov decision processes. J. Mach. Learn. Res. 8, 2169–2231 (2007).
Bonnevie, T. et al. Grid cells require excitatory drive from the hippocampus. Nat. Neurosci. 16, 309–317 (2013).
Hales, J.B. et al. Medial entorhinal cortex lesions only partially disrupt hippocampal place cells and hippocampus-dependent place memory. Cell Rep. 9, 893–901 (2014).
Solway, A. et al. Optimal behavioral hierarchy. PLoS Comput. Biol. 10, e1003779 (2014).
Ribas-Fernandes, J.J. et al. A neural signature of hierarchical reinforcement learning. Neuron 71, 370–379 (2011).
Schlesiger, M.I. et al. The medial entorhinal cortex is necessary for temporal organization of hippocampal neuronal activity. Nat. Neurosci. 18, 1123–1132 (2015).
Blum, K.I. & Abbott, L.F. A model of spatial map formation in the hippocampus of the rat. Neural Comput. 8, 85–93 (1996).
Levy, W.B., Hocking, A.B. & Wu, X. Interpreting hippocampal function as recoding and forecasting. Neural Netw. 18, 1242–1264 (2005).
Hassabis, D. & Maguire, E.A. The construction system of the brain. Phil. Trans. R. Soc. Lond. B 364, 1263–1271 (2009).
Buckner, R.L. The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol. 61, 27–48, C1–C8 (2010).
Barry, C., Hayman, R., Burgess, N. & Jeffery, K.J. Experience-dependent rescaling of entorhinal grids. Nat. Neurosci. 10, 682–684 (2007).
Fanselow, M.S. From contextual fear to a dynamic view of memory systems. Trends Cogn. Sci. 14, 7–15 (2010).
Wiltgen, B.J., Sanders, M.J., Anagnostaras, S.G., Sage, J.R. & Fanselow, M.S. Context fear learning in the absence of the hippocampus. J. Neurosci. 26, 5484–5491 (2006).
Maurer, A.P. & McNaughton, B.L. Network and intrinsic cellular mechanisms underlying theta phase precession of hippocampal neurons. Trends Neurosci. 30, 325–333 (2007).
Johnson, A. & Redish, A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).
Pezzulo, G., van der Meer, M.A., Lansink, C.S. & Pennartz, C.M. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci. 18, 647–657 (2014).
Hasselmo, M.E. & Stern, C.E. Theta rhythm and the encoding and retrieval of space and time. Neuroimage 85, 656–666 (2014).
Ekstrom, A.D., Meltzer, J., McNaughton, B.L. & Barnes, C.A. NMDA receptor antagonism blocks experience-dependent expansion of hippocampal “place fields”. Neuron 31, 631–638 (2001).
Hafting, T., Fyhn, M., Bonnevie, T., Moser, M.-B. & Moser, E.I. Hippocampus-independent phase precession in entorhinal grid cells. Nature 453, 1248–1252 (2008).
Sutton, R.S. DYNA, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin 2, 160–163 (1991).
Zhang, J., Springenberg, J.T., Boedecker, J. & Burgard, W. Deep reinforcement learning with successor features for navigation across similar environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (2017).
Momennejad, I. et al. The successor representation in human reinforcement learning. Preprint at https://doi.org/10.1101/083824 (2017).
Howard, M.W., Fotedar, M.S., Datey, A.V. & Hasselmo, M.E. The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. Psychol. Rev. 112, 75–116 (2005).
Krupic, J., Burgess, N. & O'Keefe, J. Neural representations of location composed of spatially periodic bands. Science 337, 853–857 (2012).
Sutton, R. & Barto, A. Reinforcement Learning: an Introduction (MIT Press, 1998).
Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).
Bialek, W. Biophysics: Searching for Principles (Princeton University Press, 2012).
Weng, J., Zhang, Y. & Hwang, W. Candid covariance-free incremental principal component analysis. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1034–1040 (2003).
Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 888–905 (2000).
We are grateful to T. Behrens, I. Mommenejad, and K. Miller for helpful discussions, and to A. Mathis, H. Sanders, M. Chadwick, and D. Kumaran for comments on an earlier draft of the paper. This research was supported by the NSF Collaborative Research in Computational Neuroscience (CRCNS) Program Grant IIS-120 7833 and The John Templeton Foundation. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the funding agencies.
The authors declare no competing financial interests.
About this article
Cite this article
Stachenfeld, K., Botvinick, M. & Gershman, S. The hippocampus as a predictive map. Nat Neurosci 20, 1643–1653 (2017). https://doi.org/10.1038/nn.4650
Trends in Cognitive Sciences (2020)
Journal of Computational Neuroscience (2020)
Deep learning in business analytics and operations research: Models, applications and managerial implications
European Journal of Operational Research (2020)
Nature Communications (2020)
Nature Communications (2020)