Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Navigating for reward

Nature Reviews Neuroscience volume 22, pages 472–487 (2021)Cite this article

Subjects

A Publisher Correction to this article was published on 23 July 2021

This article has been updated

Abstract

An organism’s survival can depend on its ability to recall and navigate to spatial locations associated with rewards, such as food or a home. Accumulating research has revealed that computations of reward and its prediction occur on multiple levels across a complex set of interacting brain regions, including those that support memory and navigation. However, how the brain coordinates the encoding, recall and use of reward information to guide navigation remains incompletely understood. In this Review, we propose that the brain’s classical navigation centres — the hippocampus and the entorhinal cortex — are ideally suited to coordinate this larger network by representing both physical and mental space as a series of states. These states may be linked to reward via neuromodulatory inputs to the hippocampus–entorhinal cortex system. Hippocampal outputs can then broadcast sequences of states to the rest of the brain to store reward associations or to facilitate decision-making, potentially engaging additional value signals downstream. This proposal is supported by recent advances in both experimental and theoretical neuroscience. By discussing the neural systems traditionally tied to navigation and reward at their intersection, we aim to offer an integrated framework for understanding navigation to reward as a fundamental feature of many cognitive processes.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Modulations of hippocampal–entorhinal activity at reward-related behavioural timepoints.
Fig. 2: Dopaminergic signalling and innervation of the hippocampus.
Fig. 3: Hippocampal theta sequences and replay.
Fig. 4: Hypothesized interactions between brain systems in navigating to reward.

Similar content being viewed by others

Change history

References

  1. Robinson, T. E. & Berridge, K. C. The psychology and neurobiology of addiction: an incentive-sensitization view. Addiction 95, S91–S117 (2000).

    PubMed  Google Scholar 

  2. Crombag, H. S. & Shaham, Y. Renewal of drug seeking by contextual cues after prolonged extinction in rats. Behav. Neurosci. 116, 169–173 (2002).

    Article  CAS  PubMed  Google Scholar 

  3. O’Keefe, J. & Nadel, L. The Hippocampus as a Cognitive Map (Oxford University Press, 1978).

  4. Hafting, T., Fyhn, M., Molden, S., Moser, M. B. & Moser, E. I. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801–806 (2005).

    Article  CAS  PubMed  Google Scholar 

  5. McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I. & Moser, M. B. Path integration and the neural basis of the ‘cognitive map’. Nat. Rev. Neurosci. 7, 663–678 (2006).

    Article  CAS  PubMed  Google Scholar 

  6. Diehl, G. W., Hon, O. J., Leutgeb, S. & Leutgeb, J. K. Grid and nongrid cells in medial entorhinal cortex represent spatial location and environmental features with complementary coding schemes. Neuron 94, 83–92 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Taube, J. S. The head direction signal: origins and sensory-motor integration. Annu. Rev. Neurosci. 30, 181–207 (2007).

    Article  CAS  PubMed  Google Scholar 

  8. Taube, J. S., Muller, R. U. & Ranck, J. B. Jr. Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations. J. Neurosci. 10, 436–447 (1990).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Sargolini, F. et al. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science 312, 758–762 (2006).

    Article  CAS  PubMed  Google Scholar 

  10. Kropff, E., Carmichael, J. E., Moser, M. B. & Moser, E. I. Speed cells in the medial entorhinal cortex. Nature 523, 419–424 (2015).

    Article  CAS  PubMed  Google Scholar 

  11. Hoydal, O. A., Skytoen, E. R., Andersson, S. O., Moser, M. B. & Moser, E. I. Object-vector coding in the medial entorhinal cortex. Nature 568, 400–404 (2019).

    Article  CAS  PubMed  Google Scholar 

  12. Solstad, T., Boccara, C. N., Kropff, E., Moser, M. B. & Moser, E. I. Representation of geometric borders in the entorhinal cortex. Science 322, 1865–1868 (2008).

    Article  CAS  PubMed  Google Scholar 

  13. Savelli, F., Yoganarasimha, D. & Knierim, J. J. Influence of boundary removal on the spatial representations of the medial entorhinal cortex. Hippocampus 18, 1270–1282 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Tsao, A. et al. Integrating time from experience in the lateral entorhinal cortex. Nature 561, 57–62 (2018).

    Article  CAS  PubMed  Google Scholar 

  15. Deshmukh, S. S. & Knierim, J. J. Representation of non-spatial and spatial information in the lateral entorhinal cortex. Front. Behav. Neurosci. https://doi.org/10.3389/fnnrh.2011.00069 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Tsao, A., Moser, M. B. & Moser, E. I. Traces of experience in the lateral entorhinal cortex. Curr. Biol. 23, 399–405 (2013).

    Article  CAS  PubMed  Google Scholar 

  17. Fiete, I. R., Burak, Y. & Brookings, T. What grid cells convey about rat location. J. Neurosci. 28, 6858–6871 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Allen, K. et al. Impaired path integration and grid cell spatial periodicity in mice lacking GluA1-containing AMPA receptors. J. Neurosci. 34, 6245–6259 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Gil, M. et al. Impaired path integration in mice with disrupted grid cell firing. Nat. Neurosci. 1, 81–91 (2018).

    Article  CAS  Google Scholar 

  20. Rowland, D. C., Roudi, Y., Moser, M. B. & Moser, E. I. Ten years of grid cells. Annu. Rev. Neurosci. 39, 19–40 (2016).

    Article  CAS  PubMed  Google Scholar 

  21. Burwell, R. D. & Witter, M. P. in The Parahippocampal Region: Organization and Role in Cognitive Function (eds Witter, M. P. & Wouterlood, F. G.) (Oxford University Press, 2002).

  22. O’Keefe, J. & Dostrovsky, J. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34, 171–175 (1971).

    Article  PubMed  Google Scholar 

  23. O’Keefe, J. & Recce, M. L. Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus 3, 317–330 (1993).

    Article  PubMed  Google Scholar 

  24. Skaggs, W. E., McNaughton, B. L., Wilson, M. A. & Barnes, C. A. Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences. Hippocampus 6, 149–172 (1996).

    Article  CAS  PubMed  Google Scholar 

  25. Dragoi, G. & Buzsaki, G. Temporal encoding of place sequences by hippocampal cell assemblies. Neuron 50, 145–157 (2006).

    Article  CAS  PubMed  Google Scholar 

  26. Buzsaki, G. Hippocampal sharp wave-ripple: a cognitive biomarker for episodic memory and planning. Hippocampus 25, 1073–1188 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Wilson, M. A. & McNaughton, B. L. Reactivation of hippocampal ensemble memories during sleep. Science 265, 676–679 (1994).

    Article  CAS  PubMed  Google Scholar 

  28. Foster, D. J. & Wilson, M. A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440, 680–683 (2006).

    Article  CAS  PubMed  Google Scholar 

  29. Diba, K. & Buzsaki, G. Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 10, 1241–1242 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Lee, A. K. & Wilson, M. A. Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 36, 1183–1194 (2002).

    Article  CAS  PubMed  Google Scholar 

  31. Kentros, C. G., Agnihotri, N. T., Streater, S., Hawkins, R. D. & Kandel, E. R. Increased attention to spatial context increases both place field stability and spatial memory. Neuron 42, 283–295 (2004).

    Article  CAS  PubMed  Google Scholar 

  32. Dupret, D., O’Neill, J., Pleydell-Bouverie, B. & Csicsvari, J. The reorganization and reactiation of hippocampal maps predict spatial memory performance. Nat. Neurosci. 13, 995–1002 (2010). This landmark study establishes that the clustering of hippocampal place fields near reward locations requires plasticity during learning to retain the reorganized representation during memory retrieval and that reward memory is supported by reactivation of the reorganized representation during SWRs.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. de Lavilleon, G., Lacroix, M. M., Rondi-Reig, L. & Benchenane, K. Explicit memory creation during sleep demonstrates a causal role of place cells in navigation. Nat. Neurosci. 18, 493–495 (2015).

    Article  CAS  PubMed  Google Scholar 

  34. Robbe, D. & Buzsaki, G. Alteration of theta timescale dynamics of hippocampal place cells by a cannabinoid is associated with memory impairment. J. Neurosci. 29, 12597–12605 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Petersen, P. C. & Buzsaki, G. Cooling of medial septum reveals theta phase lag coordination of hippocampal cell assemblies. Neuron 107, 731–744 e733 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Bolding, K. A., Ferbinteanu, J., Fox, S. E. & Muller, R. U. Place cell firing cannot support navigation without intact septal circuits. Hippocampus 30, 175–191 (2020).

    Article  PubMed  Google Scholar 

  37. Jadhav, S. P., Kemere, C., German, P. W. & Frank, L. M. Awake hippocampal sharp-wave ripples support spatial memory. Science 336, 1454–1458 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Fernandez-Ruiz, A. et al. Long-duration hippocampal sharp wave ripples improve memory. Science 364, 1082–1086 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Gridchyn, I., Schoenenberger, P., O’Neill, J. & Csicsvari, J. Assembly-specific disruption of hippocampal replay leads to selective memory deficit. Neuron 106, 291–300.e6 (2020).

    Article  CAS  PubMed  Google Scholar 

  40. Ego-Stengel, V. & Wilson, M. A. Disruption of ripple-associated hippocampal activity during rest impairs spatial learning in the rat. Hippocampus 20, 1–10 (2010).

    PubMed  PubMed Central  Google Scholar 

  41. Girardeau, G., Benchenane, K., Wiener, S. I., Buzsaki, G. & Zugaro, M. B. Selective suppression of hippocampal ripples impairs spatial memory. Nat.Neurosci. 12, 1222–1223 (2009).

    Article  CAS  PubMed  Google Scholar 

  42. Robinson, N. T. M. et al. Targeted activation of hippocampal place cells drives memory-guided spatial behavior. Cell 183, 2041–2042 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Heys, J. G. & Dombeck, D. A. Evidence for a subcircuit in medial entorhinal cortex representing elapsed time during immobility. Nat. Neurosci. 21, 1574–1582 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Sun, C., Yang, W., Martin, J. & Tonegawa, S. Hippocampal neurons represent events as transferable units of experience. Nat. Neurosci. 23, 651–663 (2020).

    Article  CAS  PubMed  Google Scholar 

  45. Taxidis, J. et al. Differential emergence and stability of sensory and temporal representations in context-specific hippocampal sequences. Neuron 108, 984–998 e989 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. MacDonald, C. J., Lepage, K. Q., Eden, U. T. & Eichenbaum, H. Hippocampal “time cells” bridge the gap in memory for discontiguous events. Neuron 71, 737–749 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Pastalkova, E., Itskov, V., Amarasingham, A. & Buzsaki, G. Internally generated cell assembly sequences in the rat hippocampus. Science 321, 1322–1327 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Kraus, B. J. et al. Grid cells are time cells. SFN Neurosci. Abstr. 769, 19 (2013).

    Google Scholar 

  49. Shimbo, A., Izawa, E. I. & Fujisawa, S. Scalable representation of time in the hippocampus. Sci. Adv. 7, eabd7013 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Aronov, D., Nevers, R. & Tank, D. W. Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit. Nature 543, 719–722 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Terada, S., Sakurai, Y., Nakahara, H. & Fujisawa, S. Temporal and rate coding for discrete event sequences in the hippocampus. Neuron 94, 1248–1262 e1244 (2017).

    Article  CAS  PubMed  Google Scholar 

  52. Radvansky, B. A. & Dombeck, D. A. An olfactory virtual reality system for mice. Nat. Commun. 9, 839 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  53. Behrens, T. E. J. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).

    Article  CAS  PubMed  Google Scholar 

  54. Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. The hippocampus as a predictive map. Nat. Neurosci. 20, 1643–1653 (2017). This computational modelling article proposes that the HPC–EC system encodes an SR of predicted future states, unifying findings made during spatial navigation studies with a reinforcement learning framework.

    Article  CAS  PubMed  Google Scholar 

  55. Klukas, M., Lewis, M. & Fiete, I. Efficient and flexible representation of higher-dimensional cognitive variables with grid cells. PLoS Comput. Biol. 16, e1007796 (2020).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  56. Burgess, N., Maguire, E. A. & O’Keefe, J. The human hippocampus and spatial and episodic memory. Neuron 35, 625–641 (2002).

    Article  CAS  PubMed  Google Scholar 

  57. Singer, A. C. & Frank, L. M. Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron 64, 910–921 (2009). This key set of findings demonstrates a specific enhancement of hippocampal SWRs by receipt of reward in the awake state, with reward increasing both the prevalence of SWR events and the reactivation of place cells involved in the task.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  58. Sasaki, T. et al. Dentate network activity is necessary for spatial working memory by supporting CA3 sharp-wave ripple generation and prospective firing of CA3 neurons. Nat. Neurosci. 21, 258–269 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Ambrose, R. E., Pfeiffer, B. E. & Foster, D. J. Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron 91, 1124–1136 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  60. Sosa, M., Joo, H. R. & Frank, L. M. Dorsal and ventral hippocampal sharp-wave ripples activate distinct nucleus accumbens networks. Neuron 105, 725–741 e728 (2020).

    Article  CAS  PubMed  Google Scholar 

  61. Bhattarai, B., Lee, J. W. & Jung, M. W. Distinct effects of reward and navigation history on hippocampal forward and reverse replays. Proc. Natl Acad. Sci. USA 117, 689–697 (2020).

    Article  CAS  PubMed  Google Scholar 

  62. Eichenbaum, H., Kuperstein, M., Fagan, A. & Nagode, J. Cue-sampling and goal-approach correlates of hippocampal unit-activity in rats performing an odor-discrimination task. J. Neurosci. 7, 716–732 (1987).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Markus, E. J. et al. Interactions between location and task affect the spatial and directional firing of hippocampal neurons. J. Neurosci. 15, 7079–7094 (1995).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Aoki, Y., Igata, H., Ikegaya, Y. & Sasaki, T. The integration of goal-directed signals onto spatial maps of hippocampal place cells. Cell Rep. 27, 1516–1527 e1515 (2019).

    Article  CAS  PubMed  Google Scholar 

  65. Wood, E. R., Dudchenko, P. A., Robitsek, R. J. & Eichenbaum, H. Hippocampal neurons encode information about different types of memory episodes occurring in the same location. Neuron 27, 623–633 (2000).

    Article  CAS  PubMed  Google Scholar 

  66. Frank, L. M., Brown, E. N. & Wilson, M. Trajectory encoding in the hippocampus and entorhinal cortex. Neuron 27, 169–178 (2000). This study is one of the first (see also Wood et al. (2000)) to demonstrate prospective and retrospective coding in both the HPC and the MEC, indicating that cells previously thought to encode only spatial locations can reflect mnemonic processing of the animal’s future or past route.

    Article  CAS  PubMed  Google Scholar 

  67. Grieves, R. M., Wood, E. R. & Dudchenko, P. A. Place cells on a maze encode routes rather than destinations. eLife 5, e15986 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Ito, H. T., Zhang, S., Witter, M. P., Moser, E. I. & Moser, M. B. A prefrontal-thalamo-hippocampal circuit for goal directed spatial navigation. Nature 522, 50–55 (2015).

    Article  CAS  PubMed  Google Scholar 

  69. Lee, I., Griffin, A. L., Zilli, E. A., Eichenbaum, H. & Hasselmo, M. E. Gradual translocation of spatial correlates of neuronal firing in the hippocampus toward prospective reward locations. Neuron 51, 639–650 (2006).

    Article  CAS  PubMed  Google Scholar 

  70. Kennedy, P. J. & Shapiro, M. L. Motivational states activate distinct hippocampal representations to guide goal-directed behaviors. Proc. Natl. Acad. Sci. USA 106, 10805–10810 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  71. Lee, H., Ghim, J. W., Kim, H., Lee, D. & Jung, M. Hippocampal neural correlates for values of experienced events. J. Neurosci. 32, 15053–15065 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  72. Xu, H., Baracskay, P., O’Neill, J. & Csicsvari, J. Assembly responses of hippocampal CA1 place cells predict learned behavior in goal-directed spatial tasks on the radial eight-arm maze. Neuron 101, 119–132 e114 (2019).

    Article  CAS  PubMed  Google Scholar 

  73. Sarel, A., Finkelstein, A., Las, L. & Ulanovksy, N. Vectorial representation of spatial goals in the hippocampus of bats. Science 355, 176–180 (2017).

    Article  CAS  PubMed  Google Scholar 

  74. Hollup, S. A., Molden, S., Donnett, J. G., Moser, M. B. & Moser, E. I. Accumulation of hippocampal place fields at the goal location in an annular watermaze task. J. Neurosci. 21, 1635–1644 (2001). This article is the first to clearly demonstrate, using a ring-shaped water maze, that hippocampal place fields cluster near goal locations.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Mamad, O. et al. Place field assembly distribution encodes preferred locations. PLoS Biol. 15, e2002365 (2017). This study finds that optogenetic manipulation of VTA inputs to the dHPC can drive a behavioural place preference as well as a shift in place fields towards the location of the stimulation.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  76. Xiao, Z., Lin, K. & Fellous, J. M. Conjunctive reward-place coding properties of dorsal distal CA1 hippocampus cells. Biol. Cybern. 114, 285–301 (2020).

    Article  PubMed  Google Scholar 

  77. Danielson, N. B. et al. Sublayer-specific coding dynamics during spatial navigation and learning in hippocampal area CA1. Neuron 91, 652–665 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  78. Turi, G. F. et al. Vasoactive intestinal polypeptide-expressing interneurons in the hippocampus support goal-oriented spatial learning. Neuron 101, 1150–1165 e1158 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  79. Kaufman, A. M., Geiller, T. & Losonczy, A. A role for the locus coeruleus in hippocampal CA1 place cell reorganization during spatial reward learning. Neuron 105, 1018–1026 e1014 (2020). This elegant two-photon imaging work demonstrates for the first time that the activity of LC axons in the dHPC signals changes in a reward location, and that manipulating these inputs can modify the hippocampal population representation of a reward. Together with Mamad et al. (2017), this study implicates dopaminergic inputs in reorganizing the hippocampal map around reward sites.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Zaremba, J. D. et al. Impaired hippocampal place cell dynamics in a mouse model of the 22q11.2 deletion. Nat. Neurosci. 20, 1612–1623 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  81. Kobayashi, T., Nishijo, H., Fukuda, M., Bures, J. & Ono, T. Task-dependent representations in rat hippocampal place neurons. J. Neurophysiol. 78, 597–613 (1997).

    Article  CAS  PubMed  Google Scholar 

  82. Kobayashi, T., Tran, A. H., Nishijo, H., Ono, T. & Matsumoto, G. Contribution of hippocampal place cell activity to learning and formation of goal-directed navigation in rats. Neuroscience 117, 1025–1035 (2003).

    Article  CAS  PubMed  Google Scholar 

  83. Tryon, V. L. et al. Hippocampal neural activity reflects the economy of choices during goal-directed navigation. Hippocampus 27, 743–758 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  84. Mizuta, K., Nakai, J., Hayashi, Y. & Sato, M. Multiple coordinated cellular dynamics mediate CA1 map plasticity. Hippocampus 31, 235–243 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  85. Sato, M. et al. Distinct mechanisms of over-representation of landmarks and rewards in the hippocampus. Cell Rep. 32, 107864 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  86. McKenzie, S., Robinson, N. T., Herrera, L., Churchill, J. C. & Eichenbaum, H. Learning causes reorganization of neuronal firing patterns to represent related experiences within a hippocampal schema. J. Neurosci. 33, 10243–10256 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  87. Hok, V. et al. Goal-related activity in hippocampal place cells. J. Neurosci. 27, 472–482 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  88. Duvelle, E. et al. Insensitivity of place cells to the value of spatial goals in a two-choice flexible navigation task. J. Neurosci. 39, 2522–2541 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  89. Gauthier, J. L. & Tank, D. W. A dedicated population for reward coding in the hippocampus. Neuron 99, 179–193 (2018). This two-photon imaging study uncovers a subpopulation of hippocampal neurons specialized for encoding reward locations despite changes in location or environmental context, suggesting that a hippocampal reward signal can be dissociated from place firing.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  90. Kay, K. et al. A hippocampal network for spatial coding during immobility and sleep. Nature 531, 185–190 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  91. Lee, J. S., Briguglio, J. J., Cohen, J. D., Romani, S. & Lee, A. K. The statistical structure of the hippocampal code for space as a function of time, context, and value. Cell 183, 620–635 e622 (2020).

    Article  CAS  PubMed  Google Scholar 

  92. Lee, S. H. et al. Neural signals related to outcome evaluation are stronger in CA1 than CA3. Front. Neural Circuits 11, 40 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  93. Cembrowski, M. S. & Spruston, N. Heterogeneity within classical cell types is the rule: lessons from hippocampal pyramidal neurons. Nat. Rev. Neurosci. 20, 193–204 (2019).

    Article  CAS  PubMed  Google Scholar 

  94. Dupret, D., O’Neill, J. & Csicsvari, J. Dynamic reconfiguration of hippocampal interneuron circuits during spatial learning. Neuron 78, 166–180 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  95. Danielson, N. B. et al. In vivo imaging of dentate gyrus mossy cells in behaving mice. Neuron 93, 552–559 e554 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  96. Senzai, Y. & Buzsaki, G. Physiological properties and behavioral correlates of hippocampal granule cells and mossy cells. Neuron 93, 691–704 e695 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  97. GoodSmith, D. et al. Spatial representations of granule cells and mossy cells of the dentate gyrus. Neuron 93, 677–690 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  98. Woods, N. I. et al. The dentate gyrus classifies cortical representations of learned stimuli. Neuron 107, 173–184 e176 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  99. Azevedo, E. P. et al. A role of Drd2 hippocampal neurons in context-dependent food intake. Neuron 102, 873–886 e875 (2019).

    Article  CAS  PubMed  Google Scholar 

  100. Strange, B. A., Witter, M. P., Lein, E. S. & Moser, E. I. Functional organization of the hippocampal longitudinal axis. Nat. Rev. Neurosci. 15, 655–669 (2014).

    Article  CAS  PubMed  Google Scholar 

  101. Bryant, K. G. & Barker, J. M. Arbitration of approach-avoidance conflict by ventral hippocampus. Front. Neurosci. 14, 615337 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  102. Royer, S., Sirota, A., Patel, J. & Buzsaki, G. Distinct representations and theta dynamics in dorsal and ventral hippocampus. J. Neurosci. 30, 1777–1787 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  103. Ciocchi, S., Passecker, J., Malagon-Vina, H., Mikus, N. & Klausberger, T. Brain computation. Selective information routing by ventral hippocampal CA1 projection neurons. Science 348, 560–563 (2015).

    Article  CAS  PubMed  Google Scholar 

  104. Britt, J. P. et al. Synaptic and behavioral profile of multiple glutamatergic inputs to the nucleus accumbens. Neuron 76, 790–803 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  105. LeGates, T. A. et al. Reward behaviour is regulated by the strength of hippocampus-nucleus accumbens synapses. Nature 564, 258–262 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  106. Zhou, Y. et al. A ventral CA1 to nucleus accumbens core engram circuit mediates conditioned place preference for cocaine. Nat. Neurosci. 22, 1986–1999 (2019).

    Article  CAS  PubMed  Google Scholar 

  107. Meyers, R. A., Zavala, A. R. & Neisewander, J. L. Dorsal, but not ventral, hippocampal lesions disrupt cocaine place conditioning. Neuroreport 14, 2127–2131 (2003).

    Article  PubMed  Google Scholar 

  108. Riaz, S., Schumacher, A., Sivagurunathan, S., Van Der Meer, M. & Ito, R. Ventral, but not dorsal, hippocampus inactivation impairs reward memory expression and retrieval in contexts defined by proximal cues. Hippocampus 27, 822–836 (2017).

    Article  CAS  PubMed  Google Scholar 

  109. Sjulson, L., Peyrache, A., Cumpelik, A., Cassataro, D. & Buzsaki, G. Cocaine place conditioning strengthens location-specific hippocampal coupling to the nucleus accumbens. Neuron 98, 926–934.e925 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  110. Trouche, S. et al. A hippocampus-accumbens tripartite neuronal motif guides appetitive memory in space. Cell 176, 1393–1406 e1316 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  111. van der Meer, M. A. & Redish, A. D. Theta phase precession in rat ventral striatum links place and reward information. J. Neurosci. 31, 2843–2854 (2011).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  112. Gergues, M. M. et al. Circuit and molecular architecture of a ventral hippocampal network. Nat. Neurosci. 23, 1444–1452 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  113. Hardcastle, K., Maheswaranathan, N., Ganguli, S. & Giocomo, L. M. A multiplexed, heterogeneous, and adaptive code for navigation in medial entorhinal cortex. Neuron 94, 375–387 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  114. O’Neill, J., Boccara, C. N., Stella, F., Schoenenberger, P. & Csicsvari, J. Superficial layers of the medial entorhinal cortex replay independently of the hippocampus. Science 355, 184–188 (2017).

    Article  CAS  PubMed  Google Scholar 

  115. Lipton, P. A., White, J. A. & Eichenbaum, H. Disambiguation of overlapping experiences by neurons in the medial entorhinal cortex. J. Neurosci. 27, 5787–5795 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  116. Wilming, N., Konig, P., Konig, S. & Buffalo, E. A. Entorhinal cortex receptive fields are modulated by spatial attention, even without movement. eLife 7, e31745 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  117. Butler, W. N., Hardcastle, K. & Giocomo, L. M. Remembered reward locations restructure entorhinal spatial maps. Science 363, 1447–1452 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  118. Boccara, C. N., Nardin, M., Stella, F., O’Neill, J. & Csicsvari, J. The entorhinal cognitive map is attracted to goals. Science 363, 1443–1447 (2019). Using a memory-guided cheeseboard maze, this study finds that individuals fields of MEC grid cells can shift towards reward locations through learning, indicating that grid cells are more dynamically modulated by task demands than previously appreciated (see also Butler et al. (2019)).

    Article  CAS  PubMed  Google Scholar 

  119. Palacios-Filardo, J. & Mellor, J. R. Neuromodulation of hippocampal long-term synaptic plasticity. Curr. Opin. Neurobiol. 54, 37–43 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  120. Watabe-Uchida, M., Eshel, N. & Uchida, N. Neural circuitry of reward prediction error. Annu. Rev. Neurosci. 40, 373–394 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  121. Berke, J. D. What does dopamine mean? Nat. Neurosci. 21, 787–793 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  122. Keiflin, R. & Janak, P. H. Dopamine prediction errors in reward learning and addiction: from theory to neural circuitry. Neuron 88, 247–263 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  123. Bromberg-Martin, E. S., Matsumoto, M. & Hikosaka, O. Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815–834 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  124. Fields, H. L., Hjelmstad, G. O., Margolis, E. B. & Nicola, S. M. Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289–316 (2007).

    Article  CAS  PubMed  Google Scholar 

  125. Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  126. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  PubMed  Google Scholar 

  127. Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B. & Uchida, N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  128. Fiorillo, C. D., Tobler, P. N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).

    Article  CAS  PubMed  Google Scholar 

  129. Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  130. Sutton, R. S. & Barto, A. G. Reinforcement Learning (Adaptive Computation and Machine Learning) (MIT Press, 1998).

  131. Starkweather, C. K., Babayan, B. M., Uchida, N. & Gershman, S. J. Dopamine reward prediction errors reflect hidden-state inference across time. Nat. Neurosci. 20, 581–589 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  132. Lak, A., Nomoto, K., Keramati, M., Sakagami, M. & Kepecs, A. Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision. Curr. Biol. 27, 821–832 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  133. Dabney, W. et al. A distributional code for value in dopamine-based reinforcement learning. Nature 577, 671–675 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  134. Engelhard, B. et al. Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons. Nature 570, 509–513 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  135. Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).

    Article  CAS  PubMed  Google Scholar 

  136. Day, J. J., Roitman, M. F., Wightman, R. M. & Carelli, R. M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).

    Article  CAS  PubMed  Google Scholar 

  137. Floresco, S. B. The nucleus accumbens: an interface between cognition, emotion, and action. Annu. Rev. Psychol. 66, 25–52 (2015).

    Article  PubMed  Google Scholar 

  138. Hamid, A. A. et al. Mesolimbic dopamine signals the value of work. Nat. Neurosci. 19, 117–126 (2016).

    Article  CAS  PubMed  Google Scholar 

  139. Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E. & Graybiel, A. M. Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 500, 575–579 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  140. Kim, H. R. et al. A unified framework for dopamine signals across timescales. Cell 183, 1600–1616 e1625 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  141. Phillips, P. E., Stuber, G. D., Heien, M. L., Wightman, R. M. & Carelli, R. M. Subsecond dopamine release promotes cocaine seeking. Nature 422, 614–618 (2003).

    Article  CAS  PubMed  Google Scholar 

  142. Wassum, K. M., Ostlund, S. B. & Maidment, N. T. Phasic mesolimbic dopamine signaling precedes and predicts performance of a self-initiated action sequence task. Biol. Psychiatry 71, 846–854 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  143. Mohebi, A. et al. Dissociable dopamine dynamics for learning and motivation. Nature 570, 65–70 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  144. Nolan, S. O. et al. Direct dopamine terminal regulation by local striatal microcircuitry. J. Neurochem. 155, 475–493 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  145. Smith, C. C. & Greene, R. W. CNS dopamine transmission mediated by noradrenergic innervation. J. Neurosci. 32, 6072–6080 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  146. Poe, G. R. et al. Locus coeruleus: a new look at the blue spot. Nat. Rev. Neurosci. 21, 644–659 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  147. Sara, S. J. & Bouret, S. Orienting and reorienting: the locus coeruleus mediates cognition through arousal. Neuron 76, 130–141 (2012).

    Article  CAS  PubMed  Google Scholar 

  148. Bouret, S. & Sara, S. J. Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur. J. Neurosci. 20, 791–802 (2004).

    Article  PubMed  Google Scholar 

  149. Bouret, S. & Richmond, B. J. Sensitivity of locus ceruleus neurons to reward value for goal-directed actions. J. Neurosci. 35, 4005–4014 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  150. Xiang, L. et al. Behavioral correlates of activity of optogenetically identified locus coeruleus noradrenergic neurons in rats performing T-maze tasks. Sci. Rep. 9, 1361 (2019).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  151. Varazzani, C., San-Galli, A., Gilardeau, S. & Bouret, S. Noradrenaline and dopamine neurons in the reward/effort trade-off: a direct electrophysiological comparison in behaving monkeys. J. Neurosci. 35, 7866–7877 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  152. Trudeau, L. E. et al. The multilingual nature of dopamine neurons. Prog. Brain Res. 211, 141–164 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  153. Fields, H. L. & Margolis, E. B. Understanding opioid reward. Trends Neurosci. 38, 217–225 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  154. Fischer, A. G. & Ullsperger, M. An update on the role of serotonin and its interplay with dopamine for reward. Front. Hum. Neurosci. 11, 484 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  155. Teixeira, C. M. et al. Hippocampal 5-HT input regulates memory formation and schaffer collateral excitation. Neuron 98, 992–1004 e1004 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  156. Luchetti, A. et al. Two functionally distinct serotonergic projections into hippocampus. J. Neurosci. 40, 4936–4944 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  157. Hangya, B., Ranade, S. P., Lorenc, M. & Kepecs, A. Central cholinergic neurons are rapidly recruited by reinforcement feedback. Cell 162, 1155–1168 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  158. Takeuchi, T. et al. Locus coeruleus and dopaminergic consolidation of everyday memory. Nature 537, 357–362 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  159. Wagatsuma, A. et al. Locus coeruleus input to hippocampal CA3 drives single-trial learning of a novel context. Proc. Natl Acad. Sci. USA 115, E310–E316 (2018).

    Article  CAS  PubMed  Google Scholar 

  160. O’Carroll, C. M., Martin, S. J., Sandin, J., Frenguelli, B. & Morris, R. G. Dopaminergic modulation of the persistence of one-trial hippocampus-dependent memory. Learn. Mem. 13, 760–769 (2006).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  161. Gasbarri, A., Packard, M. G., Campana, E. & Pacitti, C. Anterograde and retrograde tracing of projections from the ventral tegmental area to the hippocampal formation in the rat. Brain Res. Bull. 33, 445–452 (1994).

    Article  CAS  PubMed  Google Scholar 

  162. Loughlin, S. E., Foote, S. L. & Grzanna, R. Efferent projections of nucleus locus coeruleus: morphologic subpopulations have different efferent targets. Neuroscience 18, 307–319 (1986).

    Article  CAS  PubMed  Google Scholar 

  163. Fallon, J. H., Koziell, D. A. & Moore, R. Y. Catecholamine innervation of the basal forebrain. II. Amygdala, suprarhinal cortex and entorhinal cortex. J. Comp. Neurol. 180, 509–532 (1978).

    Article  CAS  PubMed  Google Scholar 

  164. Kempadoo, K. A., Mosharov, E. V., Choi, S. J., Sulzer, D. & Kandel, E. R. Dopamine release from the locus coeruleus to the dorsal hippocampus promotes spatial learning and memory. Proc. Natl Acad. Sci. USA 113, 14835–14840 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  165. Rosen, Z. B., Cheung, S. & Siegelbaum, S. A. Midbrain dopamine neurons bidirectionally regulate CA3-CA1 synaptic drive. Nat. Neurosci. 18, 1763–1771 (2015).

    Article  CAS  PubMed  Google Scholar 

  166. Martig, A. K. & Mizumori, S. J. Ventral tegmental area disruption selectively affects CA1/CA2 but not CA3 place fields during a differential reward working memory task. Hippocampus 21, 172–184 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  167. McNamara, C. G. & Dupret, D. Two sources of dopamine for the hippocampus. Trends Neurosci. 40, 383–384 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  168. McNamara, C. G., Tejero-Cantero, A., Trouche, S., Campo-Urriza, N. & Dupret, D. Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence. Nat. Neurosci. 17, 1658–1660 (2014). This article finds that optogenetic stimulation of VTA axons in the dHPC increases the reactivation of place cell ensembles in subsequent SWRs during sleep, improving memory for reward locations.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  169. Retailleau, A. & Morris, G. Spatial rule learning and corresponding CA1 place cell reorientation depend on local dopamine release. Curr. Biol. 28, 836–846 e834 (2018).

    Article  CAS  PubMed  Google Scholar 

  170. Bethus, I., Tse, D. & Morris, R. G. Dopamine and memory: modulation of the persistence of memory for novel hippocampal NMDA receptor-dependent paired associates. J. Neurosci. 30, 1610–1618 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  171. Sara, S. J. & Segal, M. Plasticity of sensory responses of locus-ceruleus neurons in the behaving rat - implications for cognition. Prog. BraRes. 88, 571–585 (1991).

    CAS  Google Scholar 

  172. Sosa, M., Gillespie, A. K. & Frank, L. M. Neural activity patterns underlying spatial coding in the hippocampus. Curr. Top. Behav. Neurosci. 37, 43–100 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  173. Buzsaki, G. & Tingley, D. Space and time: the hippocampus as a sequence generator. Trends Cogn. Sci. 22, 853–869 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  174. Gupta, A. S., van der Meer, M. A., Touretzky, D. S. & Redish, A. D. Segmentation of spatial experience by hippocampal theta sequences. Nat. Neurosci. 15, 1032–1039 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  175. Foster, D. J. & Wilson, M. A. Hippocampal theta sequences. Hippocampus 17, 1093–1099 (2007).

    Article  PubMed  Google Scholar 

  176. Johnson, A. & Redish, A. D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  177. Wikenheiser, A. M. & Redish, A. D. Hippocampal theta sequences reflect current goals. Nat. Neurosci. 18, 289–294 (2015). This key study establishes theta sequences as a putative mechanism in spatial planning, finding that when an animal initiates an approach to goals at different distances, theta sequences flexibly extend their ‘look-ahead distance’ to predict the animal’s chosen goal.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  178. Kay, K. et al. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell 180, 552–567 e525 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  179. Wang, M., Foster, D. J. & Pfeiffer, B. E. Alternating sequences of future and past behavior encoded within hippocampal theta oscillations. Science 370, 247–250 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  180. Brandon, M. P., Bogaard, A. R., Schultheiss, N. W. & Hasselmo, M. E. Segregation of cortical head direction cell assemblies on alternating theta cycles. Nat. Neurosci. 16, 739–748 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  181. Kubie, J. L. & Fenton, A. A. Linear look-ahead in conjunctive cells: an entorhinal mechanism for vector-based navigation. Front. Neural Circuits 6, 20 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  182. Hasselmo, M. E., Bodelon, C. & Wyble, B. P. A proposed function for hippocampal theta rhythm: separate phases of encoding and retrieval enhance reversal of prior learning. Neural. Comput. 14, 793–817 (2002).

    Article  PubMed  Google Scholar 

  183. Davidson, T. J., Kloosterman, F. & Wilson, M. A. Hippocampal replay of extended experience. Neuron 63, 497–507 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  184. Joo, H. R. & Frank, L. M. The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation. Nat. Rev. Neurosci. 19, 744–757 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  185. Findlay, G., Tononi, G. & Cirelli, C. The evolving view of replay and its functions in wake and sleep. Sleep. Adv. 1, zpab002 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  186. Pfeiffer, B. E. & Foster, D. J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013). This impressive study finds that in a 2D environment, hippocampal replay events can flexibly predict the animal’s subsequent trajectory to remembered reward locations, providing evidence for a possible role of replay in planning.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  187. Karlsson, M. P. & Frank, L. M. Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 12, 913–918 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  188. Singer, A. C., Carr, M. F., Karlsson, M. P. & Frank, L. M. Hippocampal SWR activity predicts correct decisions during the initial learning of an alternation task. Neuron 77, 1163–1173 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  189. Gillespie, A. K. et al. Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice. bioRxiv https://doi.org/10.1101/2021.03.09.434621 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  190. Carey, A. A., Tanaka, Y. & van der Meer, M. A. A. Reward revaluation biases hippocampal replay content away from the preferred outcome. Nat. Neurosci. 22, 1450–1459 (2019).

    Article  CAS  PubMed  Google Scholar 

  191. Barron, H. C. et al. Neuronal computation underlying inferential reasoning in humans and mice. Cell 183, 228–243 e221 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  192. Ólafsdóttir, H. F., Carpenter, F. & Barry, C. Coordinated grid and place cell replay during rest. Nat. Neurosci. 19, 792–794 (2016).

    Article  PubMed  CAS  Google Scholar 

  193. Trimper, J. B., Trettel, S. G., Hwaun, E. & Colgin, L. L. Methodological caveats in the detection of coordinted replay between place cells and grid cells. Front. Syst. Neurosci. https://doi.org/10.3389/fnsys.2017.00057 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  194. Yamamoto, J. & Tonegawa, S. Direct medial entorhinal cortex input to hippocampal CA1 is crucial for extended quiet awake replay. Neuron 96, 217–227 e214 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  195. Todorova, R. & Zugaro, M. Hippocampal ripples as a mode of communication with cortical and subcortical areas. Hippocampus 30, 39–49 (2018).

    Article  PubMed  Google Scholar 

  196. Pezzulo, G., van der Meer, M. A., Lansink, C. S. & Pennartz, C. M. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci. 18, 647–657 (2014).

    Article  PubMed  Google Scholar 

  197. Logothetis, N. K. et al. Hippocampal-cortical interaction during periods of subcortical silence. Nature 491, 547–553 (2012).

    Article  CAS  PubMed  Google Scholar 

  198. Ji, D. & Wilson, M. A. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nat. Neurosci. 10, 100–107 (2007).

    Article  CAS  PubMed  Google Scholar 

  199. Rothschild, G., Eban, E. & Frank, L. M. A cortical-hippocampal-cortical loop of information processing during memory consolidation. Nat. Neurosci. 20, 251–259 (2017).

    Article  CAS  PubMed  Google Scholar 

  200. Abadchi, J. K. et al. Spatiotemporal patterns of neocortical activity around hippocampal sharp-wave ripples. eLife 9, e51972 (2020).

    Article  CAS  Google Scholar 

  201. Bendor, D. & Wilson, M. A. Biasing the content of hippocampal replay during sleep. Nat. Neurosci. 15, 1439–1444 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  202. Eichenbaum, H. Prefrontal-hippocampal interactions in episodic memory. Nat. Rev. Neurosci. 18, 547–558 (2017).

    Article  CAS  PubMed  Google Scholar 

  203. Hyman, J. M., Zilli, E. A., Paley, A. M. & Hasselmo, M. E. Medial prefrontal cortex cells show dynamic modulation with the hippocampal theta rhythm dependent on behavior. Hippocampus 15, 739–749 (2005).

    Article  PubMed  Google Scholar 

  204. Jung, M. W., Qin, Y., McNaughton, B. L. & Barnes, C. A. Firing characteristics of deep layer neurons in prefrontal cortex in rats performing spatial working memory tasks. Cereb. Cortex 8, 437–450 (1998).

    Article  CAS  PubMed  Google Scholar 

  205. Jadhav, S. P., Rothschild, G., Roumis, D. K. & Frank, L. M. Coordinated excitation and inhibition of prefrontal ensembles during awake hippocampal sharp-wave ripple events. Neuron 90, 113–127 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  206. Hok, V., Save, E., Lenck-Santini, P. P. & Poucet, B. Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex. Proc. Natl Acad. Sci. USA 102, 4602–4607 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  207. Yu, J. Y., Liu, D. F., Loback, A., Grossrubatscher, I. & Frank, L. M. Specific hippocampal representations are linked to generalized cortical representations in memory. Nat. Commun. 9, 2209 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  208. Niv, Y. Learning task-state representations. Nat. Neurosci. 22, 1544–1553 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  209. Siapas, A. G., Lubenov, E. V. & Wilson, M. A. Prefrontal phase-locking to hippocampal theta oscillations. Neuron 46, 141–151 (2005).

    Article  CAS  PubMed  Google Scholar 

  210. Benchenane, K. et al. Coherent theta oscillations and reorganization of spike timing in the hippocampal- prefrontal network upon learning. Neuron 66, 921–936 (2010).

    Article  CAS  PubMed  Google Scholar 

  211. Jones, M. W. & Wilson, M. A. Phase precession of medial prefrontal cortical activity relative to the hippocampal theta rhythm. Hippocampus 15, 867–873 (2005).

    Article  PubMed  Google Scholar 

  212. Zielinski, M. C., Shin, J. D. & Jadhav, S. P. Coherent coding of spatial position mediated by theta oscillations in the hippocampus and prefrontal cortex. J. Neurosci. 39, 4550–4565 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  213. Hasz, B. M. & Redish, A. D. Spatial encoding in dorsomedial prefrontal cortex and hippocampus is related during deliberation. Hippocampus 30, 1194–1208 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  214. Tang, W., Shin, J. D. & Jadhav, S. P. Multiple time-scales of decision making in the hippocampus and prefrontal cortex. eLife 10, e66227 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  215. Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I. & Battaglia, F. P. Replay of rule-learning related neural patterns in the prefrontal cortex during sleep. Nat. Neurosci. 12, 919–926 (2009).

    Article  CAS  PubMed  Google Scholar 

  216. Shin, J. D., Tang, W. & Jadhav, S. P. Dynamics of awake hippocampal-prefrontal replay for spatial learning and memory-guided decision making. Neuron 104, 1110–1125 e1117 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  217. Gomperts, S. N., Kloosterman, F. & Wilson, M. A. VTA neurons coordinate with the hippocampal reactivation of spatial experience. eLife 4, e05360 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  218. Mattar, M. G. & Daw, N. D. Prioritized memory access explains planning and hippocampal replay. Nat. Neurosci. 21, 1609–1617 (2018). This work provides an innovative computational framework for how forward and reverse replay events could assign values to states along spatial trajectories depending on the agent’s behavioural needs.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  219. Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L. & Pennartz, C. M. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol. 7, e1000173 (2009).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  220. Lansink, C. S. et al. Reward expectancy strengthens CA1 theta and beta band synchronization and hippocampal-ventral striatal coupling. J. Neurosci. 36, 10598–10610 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  221. Berke, J. D., Okatan, M., Skurski, J. & Eichenbaum, H. B. Oscillatory entrainment of striatal neurons in freely moving rats. Neuron 43, 883–896 (2004).

    Article  CAS  PubMed  Google Scholar 

  222. van der Meer, M. A. & Redish, A. D. Covert expectation-of-reward in rat ventral striatum at decision points. Front. Integr. Neurosci. 3, 1 (2009).

    PubMed  PubMed Central  Google Scholar 

  223. Wirtshafter, H. S. & Wilson, M. A. Locomotor and hippocampal processing converge in the lateral septum. Curr. Biol. 29, 3177–3192 e3173 (2019).

    Article  CAS  PubMed  Google Scholar 

  224. Girardeau, G., Inema, I. & Buzsaki, G. Reactivations of emotional memory in the hippocampus-amygdala system during sleep. Nat. Neurosci. 20, 1634–1642 (2017).

    Article  CAS  PubMed  Google Scholar 

  225. Mizumori, S. J. & Tryon, V. L. Integrative hippocampal and decision-making neurocircuitry during goal-relevant predictions and encoding. Prog. Brain Res. 219, 217–242 (2015).

    Article  PubMed  Google Scholar 

  226. Lisman, J. E. & Grace, A. A. The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron. 46, 703–713 (2005).

    Article  CAS  PubMed  Google Scholar 

  227. Gershman, S. J. The successor representation: its computational logic and neural substrates. J. Neurosci. 38, 7193–7200 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  228. Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural. Comput. 5, 613–624 (1993).

    Article  Google Scholar 

  229. Dordek, Y., Soudry, D., Meir, R. & Derdikman, D. Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis. eLife 5, e10094 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  230. Momennejad, I. Learning structures: predictive representations, replay, and generalization. Curr. Opin. Behav. Sci. 32, 155–166 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  231. Bakkour, A. et al. The hippocampus supports deliberation during value-based decisions. eLife 8, e46080 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  232. Biderman, N., Bakkour, A. & Shohamy, D. What are memories for? The hippocampus bridges past experience with future decisions. Trends Cogn. Sci. 24, 542–556 (2020).

    Article  PubMed  Google Scholar 

  233. Vikbladh, O. M. et al. Hippocampal contributions to model-based planning and spatial memory. Neuron 102, 683–693 e684 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  234. Jeong, Y. et al. Role of the hippocampal CA1 region in incremental value learning. Sci. Rep. 8, 9870 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  235. McNamee, D. C., Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. Flexible modulation of sequence generation in the entorhinal-hippocampal system. Nat. Neurosci. https://doi.org/10.1038/s41593-021-00831-7 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  236. Johnson, A., van der Meer, M. A. & Redish, A. D. Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697 (2007).

    Article  CAS  PubMed  Google Scholar 

  237. Jung, M. W., Lee, H., Jeong, Y., Lee, J. W. & Lee, I. Remembering rewarding futures: a simulation-selection model of the hippocampus. Hippocampus 28, 913–930 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  238. Allen, W. E. et al. Thirst regulates motivated behavior through modulation of brainwide neural population dynamics. Science 364, 253 (2019).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  239. Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364, 255 (2019).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  240. Musall, S., Kaufman, M. T., Juavinett, A. L., Gluf, S. & Churchland, A. K. Single-trial neural dynamics are dominated by richly varied movements. Nat. Neurosci. 22, 1677–1686 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  241. Otmakhova, N. A. & Lisman, J. E. D1/D5 dopamine receptor activation increases the magnitude of early long-term potentiation at CA1 hippocampal synapses. J. Neurosci. 16, 7478–7486 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  242. Li, S., Cullen, W. K., Anwyl, R. & Rowan, M. J. Dopamine-dependent facilitation of LTP induction in hippocampal CA1 by exposure to spatial novelty. Nat. Neurosci. 6, 526–531 (2003).

    Article  CAS  PubMed  Google Scholar 

  243. Huang, Y. Y. & Kandel, E. R. D1/D5 receptor agonists induce a protein synthesis-dependent late potentiation in the CA1 region of the hippocampus. Proc. Natl. Acad. Sci. USA 92, 2446–2450 (1995).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  244. Batallán-Burrowes, A. A. & Chapman, C. A. Dopamine suppresses persistent firing in layer III lateral entorhinal cortex neurons. Neurosci. Lett. 674, 70–74 (2018).

    Article  CAS  PubMed  Google Scholar 

  245. Rosenkranz, J. A. & Johnston, D. Dopaminergic regulation of neuronal excitability through modulation of Ih in layer V entorhinal cortex. J. Neurosci. 26, 3229–3244 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  246. Caruana, D. A., Sorge, R. E., Stewart, J. & Chapman, C. A. Dopamine has bidirectional effects on synaptic responses to cortical inputs in layer II of the lateral entorhinal cortex. J. Neurophysiol. 96, 3006–3015 (2006).

    Article  PubMed  Google Scholar 

  247. Glovaci, I., Caruana, D. A. & Chapman, C. A. Dopaminergic enhancement of excitatory synaptic transmission in layer II entorhinal neurons is dependent on D1-like receptor-mediated signaling. Neuroscience 258, 74–83 (2014).

    Article  CAS  PubMed  Google Scholar 

  248. Pralong, E. & Jones, R. S. Interactions of dopamine with glutamate- and GABA-mediated synaptic transmission in the rat entorhinal cortex in vitro. Eur. J. Neurosci. 5, 760–767 (1993).

    Article  CAS  PubMed  Google Scholar 

  249. Hutter, J. A. & Chapman, C. A. Exposure to cues associated with palatable food reward results in a dopamine D2 receptor-dependent suppression of evoked synaptic responses in the entorhinal cortex. Behav. Brain Funct. 9, 37 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  250. Jin, X. et al. Dopamine D2 receptors regulate the action potential threshold by modulating T-type calcium channels in stellate cells of the medial entorhinal cortex. J. Physiol. 597, 3363–3387 (2019).

    Article  CAS  PubMed  Google Scholar 

  251. Stenkamp, K., Heinemann, U. & Schmitz, D. Dopamine suppresses stimulus-induced field potentials in layer III of rat medial entorhinal cortex. Neurosci. Lett. 255, 119–121 (1998).

    Article  CAS  PubMed  Google Scholar 

  252. Mayne, E. W., Craig, M. T., McBain, C. J. & Paulsen, O. Dopamine suppresses persistent network activity via D1-like dopamine receptors in rat medial entorhinal cortex. Eur. J. Neurosci. 37, 1242–1247 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  253. Cilz, N. I., Kurada, L., Hu, B. & Lei, S. Dopaminergic modulation of GABAergic transmission in the entorhinal cortex: concerted roles of alpha1 adrenoreceptors, inward rectifier K+, and T-type Ca2+ channels. Cereb. Cortex 24, 3195–3208 (2014).

    Article  PubMed  Google Scholar 

  254. Li, H. B., Lin, L., Yang, L. Y. & Xie, C. Dopaminergic facilitation of GABAergic transmission in layer III of rat medial entorhinal cortex. Chin. J. Physiol. 58, 46–54 (2015).

    Article  CAS  PubMed  Google Scholar 

  255. Burak, Y. & Fiete, I. R. Accurate path integration in continuous attractor network models of grid cells. PLoS Comput. Biol. 5, e1000291 (2009).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  256. Couey, J. J. et al. Recurrent inhibitory circuitry as a mechanism for grid formation. Nat. Neurosci. 16, 318–324 (2013).

    Article  CAS  PubMed  Google Scholar 

  257. Silva, D., Feng, T. & Foster, D. J. Trajectory events across hippocampal place cells require previous experience. Nat. Neurosci. 18, 1772–1779 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  258. O’Neill, J., Senior, T. J., Allen, K., Huxter, J. R. & Csicsvari, J. Reactivation of experience-dependent cell assembly patterns in the hippocampus. Nat. Neurosci. 11, 209–215 (2008).

    Article  CAS  PubMed  Google Scholar 

  259. Roux, L., Hu, B., Eichler, R., Stark, E. & Buzsaki, G. Sharp wave ripples during learning stabilize the hippocampal spatial map. Nat. Neurosci. 20, 845–853 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  260. Sabatini, B. L. & Tian, L. Imaging neurotransmitter and neuromodulator dynamics in vivo with genetically encoded indicators. Neuron 108, 17–32 (2020).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

The authors thank A. Mohebi for feedback on the manuscript, M.H. Plitt and T.G. Fisher for insightful discussions and E. Duvelle for helpful correspondence. This work was supported by the US Office of Naval Research (N00141812690), NIDA (DA042012), the Simons Foundation (542987SPI), the Vallee Foundation, the James S. McDonnell Foundation (L.M.G.) and the Helen Hay Whitney Foundation (M.S.).

Author information

Authors and Affiliations

  1. Department of Neurobiology, Stanford University School of Medicine, Stanford, CA, USA

    Marielena Sosa & Lisa M. Giocomo

Authors
  1. Marielena Sosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lisa M. Giocomo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to all aspects of the article.

Corresponding authors

Correspondence to Marielena Sosa or Lisa M. Giocomo.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Neuroscience thanks M.W. Jung, A. Losonczy and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Glossary

Grid cells

Entorhinal cortex cells that fire in triangularly spaced fields that tile the whole environment.

Place cells

Hippocampal cells that fire maximally in one or a few discrete regions of space (each cell’s ‘place field’).

Theta sequences

Sequential spikes of multiple place cells that together encode a trajectory through space, ordered by the theta phase of each spike. Theta sequences occur during times of high theta power, typically during movement.

Sharp-wave ripples

(SWRs). High-frequency oscillations (about 150–250 Hz) coincident with a sharp, low-frequency deflection in the local field potential. These events reflect the coincident activation of many hippocampal cells in a short period (about 50–200 ms) and typically occur during immobility.

Replay events

Sequential spikes of multiple place cells that typically occur locked to sharp-wave ripples during immobility and that together encode a trajectory through space. In high-fidelity replay events, place cells in the sequence are reactivated according to the order in which they fired during a previous run.

States

Snapshots of a situation discretizing a longer continuous process that comprises an experience. As an analogy, if this snapshot were taken by a camera, the duration of the state would be the exposure time and would vary depending on the situation (for example, how dark it is outside).

Value

How much an outcome, or state that predicts an outcome, is ‘worth’. This worth includes the amount and likelihood of the reward predicted.

Probabilistic value

The probability that a reward will be delivered given a certain choice. Even if a choice is correct according to the task, changing the probability of reward delivery can modulate the value of preceding states.

Cheeseboard maze

A spatial task in which rewards are hidden in a subset of holes or wells in the floor of an open arena. This task is used as a spatial memory paradigm because the animal has to remember which wells are rewarded on the basis of their position in the environment, and the reward locations can change across sessions or days.

Reward prediction error

(RPE). The difference between the reward received and the reward expected. Positive RPEs indicate larger rewards than expected (including a reward when none was expected), whereas negative RPEs indicate smaller rewards than expected (including the absence of a reward when it was expected).

Reinforcement learning

A set of computational theories often used for machine learning to describe how states and actions are assigned values that inform how an agent can receive the maximal reward.

Temporal difference reinforcement learning

(TD-RL). A type of reinforcement learning in which values are updated by a reward prediction error between temporally adjacent states, such that states preceding the reward receive a ‘cached’ value prediction.

Motivation

The impetus an agent feels to perform reward-seeking actions. Value is used to inform motivation and invigorate reward-seeking actions (make them faster and more efficient).

Value function

A function of adjacent states, or states paired with actions, that computes the expected future reward in each state.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sosa, M., Giocomo, L.M. Navigating for reward. Nat Rev Neurosci 22, 472–487 (2021). https://doi.org/10.1038/s41583-021-00479-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41583-021-00479-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing