An organism’s survival can depend on its ability to recall and navigate to spatial locations associated with rewards, such as food or a home. Accumulating research has revealed that computations of reward and its prediction occur on multiple levels across a complex set of interacting brain regions, including those that support memory and navigation. However, how the brain coordinates the encoding, recall and use of reward information to guide navigation remains incompletely understood. In this Review, we propose that the brain’s classical navigation centres — the hippocampus and the entorhinal cortex — are ideally suited to coordinate this larger network by representing both physical and mental space as a series of states. These states may be linked to reward via neuromodulatory inputs to the hippocampus–entorhinal cortex system. Hippocampal outputs can then broadcast sequences of states to the rest of the brain to store reward associations or to facilitate decision-making, potentially engaging additional value signals downstream. This proposal is supported by recent advances in both experimental and theoretical neuroscience. By discussing the neural systems traditionally tied to navigation and reward at their intersection, we aim to offer an integrated framework for understanding navigation to reward as a fundamental feature of many cognitive processes.
Subscribe to Journal
Get full journal access for 1 year
only $4.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Robinson, T. E. & Berridge, K. C. The psychology and neurobiology of addiction: an incentive-sensitization view. Addiction 95, S91–S117 (2000).
Crombag, H. S. & Shaham, Y. Renewal of drug seeking by contextual cues after prolonged extinction in rats. Behav. Neurosci. 116, 169–173 (2002).
O’Keefe, J. & Nadel, L. The Hippocampus as a Cognitive Map (Oxford University Press, 1978).
Hafting, T., Fyhn, M., Molden, S., Moser, M. B. & Moser, E. I. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801–806 (2005).
McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I. & Moser, M. B. Path integration and the neural basis of the ‘cognitive map’. Nat. Rev. Neurosci. 7, 663–678 (2006).
Diehl, G. W., Hon, O. J., Leutgeb, S. & Leutgeb, J. K. Grid and nongrid cells in medial entorhinal cortex represent spatial location and environmental features with complementary coding schemes. Neuron 94, 83–92 (2017).
Taube, J. S. The head direction signal: origins and sensory-motor integration. Annu. Rev. Neurosci. 30, 181–207 (2007).
Taube, J. S., Muller, R. U. & Ranck, J. B. Jr. Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations. J. Neurosci. 10, 436–447 (1990).
Sargolini, F. et al. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science 312, 758–762 (2006).
Kropff, E., Carmichael, J. E., Moser, M. B. & Moser, E. I. Speed cells in the medial entorhinal cortex. Nature 523, 419–424 (2015).
Hoydal, O. A., Skytoen, E. R., Andersson, S. O., Moser, M. B. & Moser, E. I. Object-vector coding in the medial entorhinal cortex. Nature 568, 400–404 (2019).
Solstad, T., Boccara, C. N., Kropff, E., Moser, M. B. & Moser, E. I. Representation of geometric borders in the entorhinal cortex. Science 322, 1865–1868 (2008).
Savelli, F., Yoganarasimha, D. & Knierim, J. J. Influence of boundary removal on the spatial representations of the medial entorhinal cortex. Hippocampus 18, 1270–1282 (2008).
Tsao, A. et al. Integrating time from experience in the lateral entorhinal cortex. Nature 561, 57–62 (2018).
Deshmukh, S. S. & Knierim, J. J. Representation of non-spatial and spatial information in the lateral entorhinal cortex. Front. Behav. Neurosci. https://doi.org/10.3389/fnnrh.2011.00069 (2011).
Tsao, A., Moser, M. B. & Moser, E. I. Traces of experience in the lateral entorhinal cortex. Curr. Biol. 23, 399–405 (2013).
Fiete, I. R., Burak, Y. & Brookings, T. What grid cells convey about rat location. J. Neurosci. 28, 6858–6871 (2008).
Allen, K. et al. Impaired path integration and grid cell spatial periodicity in mice lacking GluA1-containing AMPA receptors. J. Neurosci. 34, 6245–6259 (2014).
Gil, M. et al. Impaired path integration in mice with disrupted grid cell firing. Nat. Neurosci. 1, 81–91 (2018).
Rowland, D. C., Roudi, Y., Moser, M. B. & Moser, E. I. Ten years of grid cells. Annu. Rev. Neurosci. 39, 19–40 (2016).
Burwell, R. D. & Witter, M. P. in The Parahippocampal Region: Organization and Role in Cognitive Function (eds Witter, M. P. & Wouterlood, F. G.) (Oxford University Press, 2002).
O’Keefe, J. & Dostrovsky, J. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34, 171–175 (1971).
O’Keefe, J. & Recce, M. L. Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus 3, 317–330 (1993).
Skaggs, W. E., McNaughton, B. L., Wilson, M. A. & Barnes, C. A. Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences. Hippocampus 6, 149–172 (1996).
Dragoi, G. & Buzsaki, G. Temporal encoding of place sequences by hippocampal cell assemblies. Neuron 50, 145–157 (2006).
Buzsaki, G. Hippocampal sharp wave-ripple: a cognitive biomarker for episodic memory and planning. Hippocampus 25, 1073–1188 (2015).
Wilson, M. A. & McNaughton, B. L. Reactivation of hippocampal ensemble memories during sleep. Science 265, 676–679 (1994).
Foster, D. J. & Wilson, M. A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440, 680–683 (2006).
Diba, K. & Buzsaki, G. Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 10, 1241–1242 (2007).
Lee, A. K. & Wilson, M. A. Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 36, 1183–1194 (2002).
Kentros, C. G., Agnihotri, N. T., Streater, S., Hawkins, R. D. & Kandel, E. R. Increased attention to spatial context increases both place field stability and spatial memory. Neuron 42, 283–295 (2004).
Dupret, D., O’Neill, J., Pleydell-Bouverie, B. & Csicsvari, J. The reorganization and reactiation of hippocampal maps predict spatial memory performance. Nat. Neurosci. 13, 995–1002 (2010). This landmark study establishes that the clustering of hippocampal place fields near reward locations requires plasticity during learning to retain the reorganized representation during memory retrieval and that reward memory is supported by reactivation of the reorganized representation during SWRs.
de Lavilleon, G., Lacroix, M. M., Rondi-Reig, L. & Benchenane, K. Explicit memory creation during sleep demonstrates a causal role of place cells in navigation. Nat. Neurosci. 18, 493–495 (2015).
Robbe, D. & Buzsaki, G. Alteration of theta timescale dynamics of hippocampal place cells by a cannabinoid is associated with memory impairment. J. Neurosci. 29, 12597–12605 (2009).
Petersen, P. C. & Buzsaki, G. Cooling of medial septum reveals theta phase lag coordination of hippocampal cell assemblies. Neuron 107, 731–744 e733 (2020).
Bolding, K. A., Ferbinteanu, J., Fox, S. E. & Muller, R. U. Place cell firing cannot support navigation without intact septal circuits. Hippocampus 30, 175–191 (2020).
Jadhav, S. P., Kemere, C., German, P. W. & Frank, L. M. Awake hippocampal sharp-wave ripples support spatial memory. Science 336, 1454–1458 (2012).
Fernandez-Ruiz, A. et al. Long-duration hippocampal sharp wave ripples improve memory. Science 364, 1082–1086 (2019).
Gridchyn, I., Schoenenberger, P., O’Neill, J. & Csicsvari, J. Assembly-specific disruption of hippocampal replay leads to selective memory deficit. Neuron 106, 291–300.e6 (2020).
Ego-Stengel, V. & Wilson, M. A. Disruption of ripple-associated hippocampal activity during rest impairs spatial learning in the rat. Hippocampus 20, 1–10 (2010).
Girardeau, G., Benchenane, K., Wiener, S. I., Buzsaki, G. & Zugaro, M. B. Selective suppression of hippocampal ripples impairs spatial memory. Nat.Neurosci. 12, 1222–1223 (2009).
Robinson, N. T. M. et al. Targeted activation of hippocampal place cells drives memory-guided spatial behavior. Cell 183, 2041–2042 (2020).
Heys, J. G. & Dombeck, D. A. Evidence for a subcircuit in medial entorhinal cortex representing elapsed time during immobility. Nat. Neurosci. 21, 1574–1582 (2018).
Sun, C., Yang, W., Martin, J. & Tonegawa, S. Hippocampal neurons represent events as transferable units of experience. Nat. Neurosci. 23, 651–663 (2020).
Taxidis, J. et al. Differential emergence and stability of sensory and temporal representations in context-specific hippocampal sequences. Neuron 108, 984–998 e989 (2020).
MacDonald, C. J., Lepage, K. Q., Eden, U. T. & Eichenbaum, H. Hippocampal “time cells” bridge the gap in memory for discontiguous events. Neuron 71, 737–749 (2011).
Pastalkova, E., Itskov, V., Amarasingham, A. & Buzsaki, G. Internally generated cell assembly sequences in the rat hippocampus. Science 321, 1322–1327 (2008).
Kraus, B. J. et al. Grid cells are time cells. SFN Neurosci. Abstr. 769, 19 (2013).
Shimbo, A., Izawa, E. I. & Fujisawa, S. Scalable representation of time in the hippocampus. Sci. Adv. 7, eabd7013 (2021).
Aronov, D., Nevers, R. & Tank, D. W. Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit. Nature 543, 719–722 (2017).
Terada, S., Sakurai, Y., Nakahara, H. & Fujisawa, S. Temporal and rate coding for discrete event sequences in the hippocampus. Neuron 94, 1248–1262 e1244 (2017).
Radvansky, B. A. & Dombeck, D. A. An olfactory virtual reality system for mice. Nat. Commun. 9, 839 (2018).
Behrens, T. E. J. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).
Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. The hippocampus as a predictive map. Nat. Neurosci. 20, 1643–1653 (2017). This computational modelling article proposes that the HPC–EC system encodes an SR of predicted future states, unifying findings made during spatial navigation studies with a reinforcement learning framework.
Klukas, M., Lewis, M. & Fiete, I. Efficient and flexible representation of higher-dimensional cognitive variables with grid cells. PLoS Comput. Biol. 16, e1007796 (2020).
Burgess, N., Maguire, E. A. & O’Keefe, J. The human hippocampus and spatial and episodic memory. Neuron 35, 625–641 (2002).
Singer, A. C. & Frank, L. M. Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron 64, 910–921 (2009). This key set of findings demonstrates a specific enhancement of hippocampal SWRs by receipt of reward in the awake state, with reward increasing both the prevalence of SWR events and the reactivation of place cells involved in the task.
Sasaki, T. et al. Dentate network activity is necessary for spatial working memory by supporting CA3 sharp-wave ripple generation and prospective firing of CA3 neurons. Nat. Neurosci. 21, 258–269 (2018).
Ambrose, R. E., Pfeiffer, B. E. & Foster, D. J. Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron 91, 1124–1136 (2016).
Sosa, M., Joo, H. R. & Frank, L. M. Dorsal and ventral hippocampal sharp-wave ripples activate distinct nucleus accumbens networks. Neuron 105, 725–741 e728 (2020).
Bhattarai, B., Lee, J. W. & Jung, M. W. Distinct effects of reward and navigation history on hippocampal forward and reverse replays. Proc. Natl Acad. Sci. USA 117, 689–697 (2020).
Eichenbaum, H., Kuperstein, M., Fagan, A. & Nagode, J. Cue-sampling and goal-approach correlates of hippocampal unit-activity in rats performing an odor-discrimination task. J. Neurosci. 7, 716–732 (1987).
Markus, E. J. et al. Interactions between location and task affect the spatial and directional firing of hippocampal neurons. J. Neurosci. 15, 7079–7094 (1995).
Aoki, Y., Igata, H., Ikegaya, Y. & Sasaki, T. The integration of goal-directed signals onto spatial maps of hippocampal place cells. Cell Rep. 27, 1516–1527 e1515 (2019).
Wood, E. R., Dudchenko, P. A., Robitsek, R. J. & Eichenbaum, H. Hippocampal neurons encode information about different types of memory episodes occurring in the same location. Neuron 27, 623–633 (2000).
Frank, L. M., Brown, E. N. & Wilson, M. Trajectory encoding in the hippocampus and entorhinal cortex. Neuron 27, 169–178 (2000). This study is one of the first (see also Wood et al. (2000)) to demonstrate prospective and retrospective coding in both the HPC and the MEC, indicating that cells previously thought to encode only spatial locations can reflect mnemonic processing of the animal’s future or past route.
Grieves, R. M., Wood, E. R. & Dudchenko, P. A. Place cells on a maze encode routes rather than destinations. eLife 5, e15986 (2016).
Ito, H. T., Zhang, S., Witter, M. P., Moser, E. I. & Moser, M. B. A prefrontal-thalamo-hippocampal circuit for goal directed spatial navigation. Nature 522, 50–55 (2015).
Lee, I., Griffin, A. L., Zilli, E. A., Eichenbaum, H. & Hasselmo, M. E. Gradual translocation of spatial correlates of neuronal firing in the hippocampus toward prospective reward locations. Neuron 51, 639–650 (2006).
Kennedy, P. J. & Shapiro, M. L. Motivational states activate distinct hippocampal representations to guide goal-directed behaviors. Proc. Natl. Acad. Sci. USA 106, 10805–10810 (2009).
Lee, H., Ghim, J. W., Kim, H., Lee, D. & Jung, M. Hippocampal neural correlates for values of experienced events. J. Neurosci. 32, 15053–15065 (2012).
Xu, H., Baracskay, P., O’Neill, J. & Csicsvari, J. Assembly responses of hippocampal CA1 place cells predict learned behavior in goal-directed spatial tasks on the radial eight-arm maze. Neuron 101, 119–132 e114 (2019).
Sarel, A., Finkelstein, A., Las, L. & Ulanovksy, N. Vectorial representation of spatial goals in the hippocampus of bats. Science 355, 176–180 (2017).
Hollup, S. A., Molden, S., Donnett, J. G., Moser, M. B. & Moser, E. I. Accumulation of hippocampal place fields at the goal location in an annular watermaze task. J. Neurosci. 21, 1635–1644 (2001). This article is the first to clearly demonstrate, using a ring-shaped water maze, that hippocampal place fields cluster near goal locations.
Mamad, O. et al. Place field assembly distribution encodes preferred locations. PLoS Biol. 15, e2002365 (2017). This study finds that optogenetic manipulation of VTA inputs to the dHPC can drive a behavioural place preference as well as a shift in place fields towards the location of the stimulation.
Xiao, Z., Lin, K. & Fellous, J. M. Conjunctive reward-place coding properties of dorsal distal CA1 hippocampus cells. Biol. Cybern. 114, 285–301 (2020).
Danielson, N. B. et al. Sublayer-specific coding dynamics during spatial navigation and learning in hippocampal area CA1. Neuron 91, 652–665 (2016).
Turi, G. F. et al. Vasoactive intestinal polypeptide-expressing interneurons in the hippocampus support goal-oriented spatial learning. Neuron 101, 1150–1165 e1158 (2019).
Kaufman, A. M., Geiller, T. & Losonczy, A. A role for the locus coeruleus in hippocampal CA1 place cell reorganization during spatial reward learning. Neuron 105, 1018–1026 e1014 (2020). This elegant two-photon imaging work demonstrates for the first time that the activity of LC axons in the dHPC signals changes in a reward location, and that manipulating these inputs can modify the hippocampal population representation of a reward. Together with Mamad et al. (2017), this study implicates dopaminergic inputs in reorganizing the hippocampal map around reward sites.
Zaremba, J. D. et al. Impaired hippocampal place cell dynamics in a mouse model of the 22q11.2 deletion. Nat. Neurosci. 20, 1612–1623 (2017).
Kobayashi, T., Nishijo, H., Fukuda, M., Bures, J. & Ono, T. Task-dependent representations in rat hippocampal place neurons. J. Neurophysiol. 78, 597–613 (1997).
Kobayashi, T., Tran, A. H., Nishijo, H., Ono, T. & Matsumoto, G. Contribution of hippocampal place cell activity to learning and formation of goal-directed navigation in rats. Neuroscience 117, 1025–1035 (2003).
Tryon, V. L. et al. Hippocampal neural activity reflects the economy of choices during goal-directed navigation. Hippocampus 27, 743–758 (2017).
Mizuta, K., Nakai, J., Hayashi, Y. & Sato, M. Multiple coordinated cellular dynamics mediate CA1 map plasticity. Hippocampus 31, 235–243 (2021).
Sato, M. et al. Distinct mechanisms of over-representation of landmarks and rewards in the hippocampus. Cell Rep. 32, 107864 (2020).
McKenzie, S., Robinson, N. T., Herrera, L., Churchill, J. C. & Eichenbaum, H. Learning causes reorganization of neuronal firing patterns to represent related experiences within a hippocampal schema. J. Neurosci. 33, 10243–10256 (2013).
Hok, V. et al. Goal-related activity in hippocampal place cells. J. Neurosci. 27, 472–482 (2007).
Duvelle, E. et al. Insensitivity of place cells to the value of spatial goals in a two-choice flexible navigation task. J. Neurosci. 39, 2522–2541 (2019).
Gauthier, J. L. & Tank, D. W. A dedicated population for reward coding in the hippocampus. Neuron 99, 179–193 (2018). This two-photon imaging study uncovers a subpopulation of hippocampal neurons specialized for encoding reward locations despite changes in location or environmental context, suggesting that a hippocampal reward signal can be dissociated from place firing.
Kay, K. et al. A hippocampal network for spatial coding during immobility and sleep. Nature 531, 185–190 (2016).
Lee, J. S., Briguglio, J. J., Cohen, J. D., Romani, S. & Lee, A. K. The statistical structure of the hippocampal code for space as a function of time, context, and value. Cell 183, 620–635 e622 (2020).
Lee, S. H. et al. Neural signals related to outcome evaluation are stronger in CA1 than CA3. Front. Neural Circuits 11, 40 (2017).
Cembrowski, M. S. & Spruston, N. Heterogeneity within classical cell types is the rule: lessons from hippocampal pyramidal neurons. Nat. Rev. Neurosci. 20, 193–204 (2019).
Dupret, D., O’Neill, J. & Csicsvari, J. Dynamic reconfiguration of hippocampal interneuron circuits during spatial learning. Neuron 78, 166–180 (2013).
Danielson, N. B. et al. In vivo imaging of dentate gyrus mossy cells in behaving mice. Neuron 93, 552–559 e554 (2017).
Senzai, Y. & Buzsaki, G. Physiological properties and behavioral correlates of hippocampal granule cells and mossy cells. Neuron 93, 691–704 e695 (2017).
GoodSmith, D. et al. Spatial representations of granule cells and mossy cells of the dentate gyrus. Neuron 93, 677–690 (2017).
Woods, N. I. et al. The dentate gyrus classifies cortical representations of learned stimuli. Neuron 107, 173–184 e176 (2020).
Azevedo, E. P. et al. A role of Drd2 hippocampal neurons in context-dependent food intake. Neuron 102, 873–886 e875 (2019).
Strange, B. A., Witter, M. P., Lein, E. S. & Moser, E. I. Functional organization of the hippocampal longitudinal axis. Nat. Rev. Neurosci. 15, 655–669 (2014).
Bryant, K. G. & Barker, J. M. Arbitration of approach-avoidance conflict by ventral hippocampus. Front. Neurosci. 14, 615337 (2020).
Royer, S., Sirota, A., Patel, J. & Buzsaki, G. Distinct representations and theta dynamics in dorsal and ventral hippocampus. J. Neurosci. 30, 1777–1787 (2010).
Ciocchi, S., Passecker, J., Malagon-Vina, H., Mikus, N. & Klausberger, T. Brain computation. Selective information routing by ventral hippocampal CA1 projection neurons. Science 348, 560–563 (2015).
Britt, J. P. et al. Synaptic and behavioral profile of multiple glutamatergic inputs to the nucleus accumbens. Neuron 76, 790–803 (2012).
LeGates, T. A. et al. Reward behaviour is regulated by the strength of hippocampus-nucleus accumbens synapses. Nature 564, 258–262 (2018).
Zhou, Y. et al. A ventral CA1 to nucleus accumbens core engram circuit mediates conditioned place preference for cocaine. Nat. Neurosci. 22, 1986–1999 (2019).
Meyers, R. A., Zavala, A. R. & Neisewander, J. L. Dorsal, but not ventral, hippocampal lesions disrupt cocaine place conditioning. Neuroreport 14, 2127–2131 (2003).
Riaz, S., Schumacher, A., Sivagurunathan, S., Van Der Meer, M. & Ito, R. Ventral, but not dorsal, hippocampus inactivation impairs reward memory expression and retrieval in contexts defined by proximal cues. Hippocampus 27, 822–836 (2017).
Sjulson, L., Peyrache, A., Cumpelik, A., Cassataro, D. & Buzsaki, G. Cocaine place conditioning strengthens location-specific hippocampal coupling to the nucleus accumbens. Neuron 98, 926–934.e925 (2018).
Trouche, S. et al. A hippocampus-accumbens tripartite neuronal motif guides appetitive memory in space. Cell 176, 1393–1406 e1316 (2019).
van der Meer, M. A. & Redish, A. D. Theta phase precession in rat ventral striatum links place and reward information. J. Neurosci. 31, 2843–2854 (2011).
Gergues, M. M. et al. Circuit and molecular architecture of a ventral hippocampal network. Nat. Neurosci. 23, 1444–1452 (2020).
Hardcastle, K., Maheswaranathan, N., Ganguli, S. & Giocomo, L. M. A multiplexed, heterogeneous, and adaptive code for navigation in medial entorhinal cortex. Neuron 94, 375–387 (2017).
O’Neill, J., Boccara, C. N., Stella, F., Schoenenberger, P. & Csicsvari, J. Superficial layers of the medial entorhinal cortex replay independently of the hippocampus. Science 355, 184–188 (2017).
Lipton, P. A., White, J. A. & Eichenbaum, H. Disambiguation of overlapping experiences by neurons in the medial entorhinal cortex. J. Neurosci. 27, 5787–5795 (2007).
Wilming, N., Konig, P., Konig, S. & Buffalo, E. A. Entorhinal cortex receptive fields are modulated by spatial attention, even without movement. eLife 7, e31745 (2018).
Butler, W. N., Hardcastle, K. & Giocomo, L. M. Remembered reward locations restructure entorhinal spatial maps. Science 363, 1447–1452 (2019).
Boccara, C. N., Nardin, M., Stella, F., O’Neill, J. & Csicsvari, J. The entorhinal cognitive map is attracted to goals. Science 363, 1443–1447 (2019). Using a memory-guided cheeseboard maze, this study finds that individuals fields of MEC grid cells can shift towards reward locations through learning, indicating that grid cells are more dynamically modulated by task demands than previously appreciated (see also Butler et al. (2019)).
Palacios-Filardo, J. & Mellor, J. R. Neuromodulation of hippocampal long-term synaptic plasticity. Curr. Opin. Neurobiol. 54, 37–43 (2019).
Watabe-Uchida, M., Eshel, N. & Uchida, N. Neural circuitry of reward prediction error. Annu. Rev. Neurosci. 40, 373–394 (2017).
Berke, J. D. What does dopamine mean? Nat. Neurosci. 21, 787–793 (2018).
Keiflin, R. & Janak, P. H. Dopamine prediction errors in reward learning and addiction: from theory to neural circuitry. Neuron 88, 247–263 (2015).
Bromberg-Martin, E. S., Matsumoto, M. & Hikosaka, O. Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815–834 (2010).
Fields, H. L., Hjelmstad, G. O., Margolis, E. B. & Nicola, S. M. Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289–316 (2007).
Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).
Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B. & Uchida, N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012).
Fiorillo, C. D., Tobler, P. N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
Sutton, R. S. & Barto, A. G. Reinforcement Learning (Adaptive Computation and Machine Learning) (MIT Press, 1998).
Starkweather, C. K., Babayan, B. M., Uchida, N. & Gershman, S. J. Dopamine reward prediction errors reflect hidden-state inference across time. Nat. Neurosci. 20, 581–589 (2017).
Lak, A., Nomoto, K., Keramati, M., Sakagami, M. & Kepecs, A. Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision. Curr. Biol. 27, 821–832 (2017).
Dabney, W. et al. A distributional code for value in dopamine-based reinforcement learning. Nature 577, 671–675 (2020).
Engelhard, B. et al. Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons. Nature 570, 509–513 (2019).
Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).
Day, J. J., Roitman, M. F., Wightman, R. M. & Carelli, R. M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).
Floresco, S. B. The nucleus accumbens: an interface between cognition, emotion, and action. Annu. Rev. Psychol. 66, 25–52 (2015).
Hamid, A. A. et al. Mesolimbic dopamine signals the value of work. Nat. Neurosci. 19, 117–126 (2016).
Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E. & Graybiel, A. M. Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 500, 575–579 (2013).
Kim, H. R. et al. A unified framework for dopamine signals across timescales. Cell 183, 1600–1616 e1625 (2020).
Phillips, P. E., Stuber, G. D., Heien, M. L., Wightman, R. M. & Carelli, R. M. Subsecond dopamine release promotes cocaine seeking. Nature 422, 614–618 (2003).
Wassum, K. M., Ostlund, S. B. & Maidment, N. T. Phasic mesolimbic dopamine signaling precedes and predicts performance of a self-initiated action sequence task. Biol. Psychiatry 71, 846–854 (2012).
Mohebi, A. et al. Dissociable dopamine dynamics for learning and motivation. Nature 570, 65–70 (2019).
Nolan, S. O. et al. Direct dopamine terminal regulation by local striatal microcircuitry. J. Neurochem. 155, 475–493 (2020).
Smith, C. C. & Greene, R. W. CNS dopamine transmission mediated by noradrenergic innervation. J. Neurosci. 32, 6072–6080 (2012).
Poe, G. R. et al. Locus coeruleus: a new look at the blue spot. Nat. Rev. Neurosci. 21, 644–659 (2020).
Sara, S. J. & Bouret, S. Orienting and reorienting: the locus coeruleus mediates cognition through arousal. Neuron 76, 130–141 (2012).
Bouret, S. & Sara, S. J. Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur. J. Neurosci. 20, 791–802 (2004).
Bouret, S. & Richmond, B. J. Sensitivity of locus ceruleus neurons to reward value for goal-directed actions. J. Neurosci. 35, 4005–4014 (2015).
Xiang, L. et al. Behavioral correlates of activity of optogenetically identified locus coeruleus noradrenergic neurons in rats performing T-maze tasks. Sci. Rep. 9, 1361 (2019).
Varazzani, C., San-Galli, A., Gilardeau, S. & Bouret, S. Noradrenaline and dopamine neurons in the reward/effort trade-off: a direct electrophysiological comparison in behaving monkeys. J. Neurosci. 35, 7866–7877 (2015).
Trudeau, L. E. et al. The multilingual nature of dopamine neurons. Prog. Brain Res. 211, 141–164 (2014).
Fields, H. L. & Margolis, E. B. Understanding opioid reward. Trends Neurosci. 38, 217–225 (2015).
Fischer, A. G. & Ullsperger, M. An update on the role of serotonin and its interplay with dopamine for reward. Front. Hum. Neurosci. 11, 484 (2017).
Teixeira, C. M. et al. Hippocampal 5-HT input regulates memory formation and schaffer collateral excitation. Neuron 98, 992–1004 e1004 (2018).
Luchetti, A. et al. Two functionally distinct serotonergic projections into hippocampus. J. Neurosci. 40, 4936–4944 (2020).
Hangya, B., Ranade, S. P., Lorenc, M. & Kepecs, A. Central cholinergic neurons are rapidly recruited by reinforcement feedback. Cell 162, 1155–1168 (2015).
Takeuchi, T. et al. Locus coeruleus and dopaminergic consolidation of everyday memory. Nature 537, 357–362 (2016).
Wagatsuma, A. et al. Locus coeruleus input to hippocampal CA3 drives single-trial learning of a novel context. Proc. Natl Acad. Sci. USA 115, E310–E316 (2018).
O’Carroll, C. M., Martin, S. J., Sandin, J., Frenguelli, B. & Morris, R. G. Dopaminergic modulation of the persistence of one-trial hippocampus-dependent memory. Learn. Mem. 13, 760–769 (2006).
Gasbarri, A., Packard, M. G., Campana, E. & Pacitti, C. Anterograde and retrograde tracing of projections from the ventral tegmental area to the hippocampal formation in the rat. Brain Res. Bull. 33, 445–452 (1994).
Loughlin, S. E., Foote, S. L. & Grzanna, R. Efferent projections of nucleus locus coeruleus: morphologic subpopulations have different efferent targets. Neuroscience 18, 307–319 (1986).
Fallon, J. H., Koziell, D. A. & Moore, R. Y. Catecholamine innervation of the basal forebrain. II. Amygdala, suprarhinal cortex and entorhinal cortex. J. Comp. Neurol. 180, 509–532 (1978).
Kempadoo, K. A., Mosharov, E. V., Choi, S. J., Sulzer, D. & Kandel, E. R. Dopamine release from the locus coeruleus to the dorsal hippocampus promotes spatial learning and memory. Proc. Natl Acad. Sci. USA 113, 14835–14840 (2016).
Rosen, Z. B., Cheung, S. & Siegelbaum, S. A. Midbrain dopamine neurons bidirectionally regulate CA3-CA1 synaptic drive. Nat. Neurosci. 18, 1763–1771 (2015).
Martig, A. K. & Mizumori, S. J. Ventral tegmental area disruption selectively affects CA1/CA2 but not CA3 place fields during a differential reward working memory task. Hippocampus 21, 172–184 (2011).
McNamara, C. G. & Dupret, D. Two sources of dopamine for the hippocampus. Trends Neurosci. 40, 383–384 (2017).
McNamara, C. G., Tejero-Cantero, A., Trouche, S., Campo-Urriza, N. & Dupret, D. Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence. Nat. Neurosci. 17, 1658–1660 (2014). This article finds that optogenetic stimulation of VTA axons in the dHPC increases the reactivation of place cell ensembles in subsequent SWRs during sleep, improving memory for reward locations.
Retailleau, A. & Morris, G. Spatial rule learning and corresponding CA1 place cell reorientation depend on local dopamine release. Curr. Biol. 28, 836–846 e834 (2018).
Bethus, I., Tse, D. & Morris, R. G. Dopamine and memory: modulation of the persistence of memory for novel hippocampal NMDA receptor-dependent paired associates. J. Neurosci. 30, 1610–1618 (2010).
Sara, S. J. & Segal, M. Plasticity of sensory responses of locus-ceruleus neurons in the behaving rat - implications for cognition. Prog. BraRes. 88, 571–585 (1991).
Sosa, M., Gillespie, A. K. & Frank, L. M. Neural activity patterns underlying spatial coding in the hippocampus. Curr. Top. Behav. Neurosci. 37, 43–100 (2018).
Buzsaki, G. & Tingley, D. Space and time: the hippocampus as a sequence generator. Trends Cogn. Sci. 22, 853–869 (2018).
Gupta, A. S., van der Meer, M. A., Touretzky, D. S. & Redish, A. D. Segmentation of spatial experience by hippocampal theta sequences. Nat. Neurosci. 15, 1032–1039 (2012).
Foster, D. J. & Wilson, M. A. Hippocampal theta sequences. Hippocampus 17, 1093–1099 (2007).
Johnson, A. & Redish, A. D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).
Wikenheiser, A. M. & Redish, A. D. Hippocampal theta sequences reflect current goals. Nat. Neurosci. 18, 289–294 (2015). This key study establishes theta sequences as a putative mechanism in spatial planning, finding that when an animal initiates an approach to goals at different distances, theta sequences flexibly extend their ‘look-ahead distance’ to predict the animal’s chosen goal.
Kay, K. et al. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell 180, 552–567 e525 (2020).
Wang, M., Foster, D. J. & Pfeiffer, B. E. Alternating sequences of future and past behavior encoded within hippocampal theta oscillations. Science 370, 247–250 (2020).
Brandon, M. P., Bogaard, A. R., Schultheiss, N. W. & Hasselmo, M. E. Segregation of cortical head direction cell assemblies on alternating theta cycles. Nat. Neurosci. 16, 739–748 (2013).
Kubie, J. L. & Fenton, A. A. Linear look-ahead in conjunctive cells: an entorhinal mechanism for vector-based navigation. Front. Neural Circuits 6, 20 (2012).
Hasselmo, M. E., Bodelon, C. & Wyble, B. P. A proposed function for hippocampal theta rhythm: separate phases of encoding and retrieval enhance reversal of prior learning. Neural. Comput. 14, 793–817 (2002).
Davidson, T. J., Kloosterman, F. & Wilson, M. A. Hippocampal replay of extended experience. Neuron 63, 497–507 (2009).
Joo, H. R. & Frank, L. M. The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation. Nat. Rev. Neurosci. 19, 744–757 (2018).
Findlay, G., Tononi, G. & Cirelli, C. The evolving view of replay and its functions in wake and sleep. Sleep. Adv. 1, zpab002 (2021).
Pfeiffer, B. E. & Foster, D. J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013). This impressive study finds that in a 2D environment, hippocampal replay events can flexibly predict the animal’s subsequent trajectory to remembered reward locations, providing evidence for a possible role of replay in planning.
Karlsson, M. P. & Frank, L. M. Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 12, 913–918 (2009).
Singer, A. C., Carr, M. F., Karlsson, M. P. & Frank, L. M. Hippocampal SWR activity predicts correct decisions during the initial learning of an alternation task. Neuron 77, 1163–1173 (2013).
Gillespie, A. K. et al. Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice. bioRxiv https://doi.org/10.1101/2021.03.09.434621 (2021).
Carey, A. A., Tanaka, Y. & van der Meer, M. A. A. Reward revaluation biases hippocampal replay content away from the preferred outcome. Nat. Neurosci. 22, 1450–1459 (2019).
Barron, H. C. et al. Neuronal computation underlying inferential reasoning in humans and mice. Cell 183, 228–243 e221 (2020).
Ólafsdóttir, H. F., Carpenter, F. & Barry, C. Coordinated grid and place cell replay during rest. Nat. Neurosci. 19, 792–794 (2016).
Trimper, J. B., Trettel, S. G., Hwaun, E. & Colgin, L. L. Methodological caveats in the detection of coordinted replay between place cells and grid cells. Front. Syst. Neurosci. https://doi.org/10.3389/fnsys.2017.00057 (2017).
Yamamoto, J. & Tonegawa, S. Direct medial entorhinal cortex input to hippocampal CA1 is crucial for extended quiet awake replay. Neuron 96, 217–227 e214 (2017).
Todorova, R. & Zugaro, M. Hippocampal ripples as a mode of communication with cortical and subcortical areas. Hippocampus 30, 39–49 (2018).
Pezzulo, G., van der Meer, M. A., Lansink, C. S. & Pennartz, C. M. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci. 18, 647–657 (2014).
Logothetis, N. K. et al. Hippocampal-cortical interaction during periods of subcortical silence. Nature 491, 547–553 (2012).
Ji, D. & Wilson, M. A. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nat. Neurosci. 10, 100–107 (2007).
Rothschild, G., Eban, E. & Frank, L. M. A cortical-hippocampal-cortical loop of information processing during memory consolidation. Nat. Neurosci. 20, 251–259 (2017).
Abadchi, J. K. et al. Spatiotemporal patterns of neocortical activity around hippocampal sharp-wave ripples. eLife 9, e51972 (2020).
Bendor, D. & Wilson, M. A. Biasing the content of hippocampal replay during sleep. Nat. Neurosci. 15, 1439–1444 (2012).
Eichenbaum, H. Prefrontal-hippocampal interactions in episodic memory. Nat. Rev. Neurosci. 18, 547–558 (2017).
Hyman, J. M., Zilli, E. A., Paley, A. M. & Hasselmo, M. E. Medial prefrontal cortex cells show dynamic modulation with the hippocampal theta rhythm dependent on behavior. Hippocampus 15, 739–749 (2005).
Jung, M. W., Qin, Y., McNaughton, B. L. & Barnes, C. A. Firing characteristics of deep layer neurons in prefrontal cortex in rats performing spatial working memory tasks. Cereb. Cortex 8, 437–450 (1998).
Jadhav, S. P., Rothschild, G., Roumis, D. K. & Frank, L. M. Coordinated excitation and inhibition of prefrontal ensembles during awake hippocampal sharp-wave ripple events. Neuron 90, 113–127 (2016).
Hok, V., Save, E., Lenck-Santini, P. P. & Poucet, B. Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex. Proc. Natl Acad. Sci. USA 102, 4602–4607 (2005).
Yu, J. Y., Liu, D. F., Loback, A., Grossrubatscher, I. & Frank, L. M. Specific hippocampal representations are linked to generalized cortical representations in memory. Nat. Commun. 9, 2209 (2018).
Niv, Y. Learning task-state representations. Nat. Neurosci. 22, 1544–1553 (2019).
Siapas, A. G., Lubenov, E. V. & Wilson, M. A. Prefrontal phase-locking to hippocampal theta oscillations. Neuron 46, 141–151 (2005).
Benchenane, K. et al. Coherent theta oscillations and reorganization of spike timing in the hippocampal- prefrontal network upon learning. Neuron 66, 921–936 (2010).
Jones, M. W. & Wilson, M. A. Phase precession of medial prefrontal cortical activity relative to the hippocampal theta rhythm. Hippocampus 15, 867–873 (2005).
Zielinski, M. C., Shin, J. D. & Jadhav, S. P. Coherent coding of spatial position mediated by theta oscillations in the hippocampus and prefrontal cortex. J. Neurosci. 39, 4550–4565 (2019).
Hasz, B. M. & Redish, A. D. Spatial encoding in dorsomedial prefrontal cortex and hippocampus is related during deliberation. Hippocampus 30, 1194–1208 (2020).
Tang, W., Shin, J. D. & Jadhav, S. P. Multiple time-scales of decision making in the hippocampus and prefrontal cortex. eLife 10, e66227 (2021).
Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I. & Battaglia, F. P. Replay of rule-learning related neural patterns in the prefrontal cortex during sleep. Nat. Neurosci. 12, 919–926 (2009).
Shin, J. D., Tang, W. & Jadhav, S. P. Dynamics of awake hippocampal-prefrontal replay for spatial learning and memory-guided decision making. Neuron 104, 1110–1125 e1117 (2019).
Gomperts, S. N., Kloosterman, F. & Wilson, M. A. VTA neurons coordinate with the hippocampal reactivation of spatial experience. eLife 4, e05360 (2015).
Mattar, M. G. & Daw, N. D. Prioritized memory access explains planning and hippocampal replay. Nat. Neurosci. 21, 1609–1617 (2018). This work provides an innovative computational framework for how forward and reverse replay events could assign values to states along spatial trajectories depending on the agent’s behavioural needs.
Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L. & Pennartz, C. M. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol. 7, e1000173 (2009).
Lansink, C. S. et al. Reward expectancy strengthens CA1 theta and beta band synchronization and hippocampal-ventral striatal coupling. J. Neurosci. 36, 10598–10610 (2016).
Berke, J. D., Okatan, M., Skurski, J. & Eichenbaum, H. B. Oscillatory entrainment of striatal neurons in freely moving rats. Neuron 43, 883–896 (2004).
van der Meer, M. A. & Redish, A. D. Covert expectation-of-reward in rat ventral striatum at decision points. Front. Integr. Neurosci. 3, 1 (2009).
Wirtshafter, H. S. & Wilson, M. A. Locomotor and hippocampal processing converge in the lateral septum. Curr. Biol. 29, 3177–3192 e3173 (2019).
Girardeau, G., Inema, I. & Buzsaki, G. Reactivations of emotional memory in the hippocampus-amygdala system during sleep. Nat. Neurosci. 20, 1634–1642 (2017).
Mizumori, S. J. & Tryon, V. L. Integrative hippocampal and decision-making neurocircuitry during goal-relevant predictions and encoding. Prog. Brain Res. 219, 217–242 (2015).
Lisman, J. E. & Grace, A. A. The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron. 46, 703–713 (2005).
Gershman, S. J. The successor representation: its computational logic and neural substrates. J. Neurosci. 38, 7193–7200 (2018).
Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural. Comput. 5, 613–624 (1993).
Dordek, Y., Soudry, D., Meir, R. & Derdikman, D. Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis. eLife 5, e10094 (2016).
Momennejad, I. Learning structures: predictive representations, replay, and generalization. Curr. Opin. Behav. Sci. 32, 155–166 (2020).
Bakkour, A. et al. The hippocampus supports deliberation during value-based decisions. eLife 8, e46080 (2019).
Biderman, N., Bakkour, A. & Shohamy, D. What are memories for? The hippocampus bridges past experience with future decisions. Trends Cogn. Sci. 24, 542–556 (2020).
Vikbladh, O. M. et al. Hippocampal contributions to model-based planning and spatial memory. Neuron 102, 683–693 e684 (2019).
Jeong, Y. et al. Role of the hippocampal CA1 region in incremental value learning. Sci. Rep. 8, 9870 (2018).
McNamee, D. C., Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. Flexible modulation of sequence generation in the entorhinal-hippocampal system. Nat. Neurosci. https://doi.org/10.1038/s41593-021-00831-7 (2021).
Johnson, A., van der Meer, M. A. & Redish, A. D. Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697 (2007).
Jung, M. W., Lee, H., Jeong, Y., Lee, J. W. & Lee, I. Remembering rewarding futures: a simulation-selection model of the hippocampus. Hippocampus 28, 913–930 (2018).
Allen, W. E. et al. Thirst regulates motivated behavior through modulation of brainwide neural population dynamics. Science 364, 253 (2019).
Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364, 255 (2019).
Musall, S., Kaufman, M. T., Juavinett, A. L., Gluf, S. & Churchland, A. K. Single-trial neural dynamics are dominated by richly varied movements. Nat. Neurosci. 22, 1677–1686 (2019).
Otmakhova, N. A. & Lisman, J. E. D1/D5 dopamine receptor activation increases the magnitude of early long-term potentiation at CA1 hippocampal synapses. J. Neurosci. 16, 7478–7486 (1996).
Li, S., Cullen, W. K., Anwyl, R. & Rowan, M. J. Dopamine-dependent facilitation of LTP induction in hippocampal CA1 by exposure to spatial novelty. Nat. Neurosci. 6, 526–531 (2003).
Huang, Y. Y. & Kandel, E. R. D1/D5 receptor agonists induce a protein synthesis-dependent late potentiation in the CA1 region of the hippocampus. Proc. Natl. Acad. Sci. USA 92, 2446–2450 (1995).
Batallán-Burrowes, A. A. & Chapman, C. A. Dopamine suppresses persistent firing in layer III lateral entorhinal cortex neurons. Neurosci. Lett. 674, 70–74 (2018).
Rosenkranz, J. A. & Johnston, D. Dopaminergic regulation of neuronal excitability through modulation of Ih in layer V entorhinal cortex. J. Neurosci. 26, 3229–3244 (2006).
Caruana, D. A., Sorge, R. E., Stewart, J. & Chapman, C. A. Dopamine has bidirectional effects on synaptic responses to cortical inputs in layer II of the lateral entorhinal cortex. J. Neurophysiol. 96, 3006–3015 (2006).
Glovaci, I., Caruana, D. A. & Chapman, C. A. Dopaminergic enhancement of excitatory synaptic transmission in layer II entorhinal neurons is dependent on D1-like receptor-mediated signaling. Neuroscience 258, 74–83 (2014).
Pralong, E. & Jones, R. S. Interactions of dopamine with glutamate- and GABA-mediated synaptic transmission in the rat entorhinal cortex in vitro. Eur. J. Neurosci. 5, 760–767 (1993).
Hutter, J. A. & Chapman, C. A. Exposure to cues associated with palatable food reward results in a dopamine D2 receptor-dependent suppression of evoked synaptic responses in the entorhinal cortex. Behav. Brain Funct. 9, 37 (2013).
Jin, X. et al. Dopamine D2 receptors regulate the action potential threshold by modulating T-type calcium channels in stellate cells of the medial entorhinal cortex. J. Physiol. 597, 3363–3387 (2019).
Stenkamp, K., Heinemann, U. & Schmitz, D. Dopamine suppresses stimulus-induced field potentials in layer III of rat medial entorhinal cortex. Neurosci. Lett. 255, 119–121 (1998).
Mayne, E. W., Craig, M. T., McBain, C. J. & Paulsen, O. Dopamine suppresses persistent network activity via D1-like dopamine receptors in rat medial entorhinal cortex. Eur. J. Neurosci. 37, 1242–1247 (2013).
Cilz, N. I., Kurada, L., Hu, B. & Lei, S. Dopaminergic modulation of GABAergic transmission in the entorhinal cortex: concerted roles of alpha1 adrenoreceptors, inward rectifier K+, and T-type Ca2+ channels. Cereb. Cortex 24, 3195–3208 (2014).
Li, H. B., Lin, L., Yang, L. Y. & Xie, C. Dopaminergic facilitation of GABAergic transmission in layer III of rat medial entorhinal cortex. Chin. J. Physiol. 58, 46–54 (2015).
Burak, Y. & Fiete, I. R. Accurate path integration in continuous attractor network models of grid cells. PLoS Comput. Biol. 5, e1000291 (2009).
Couey, J. J. et al. Recurrent inhibitory circuitry as a mechanism for grid formation. Nat. Neurosci. 16, 318–324 (2013).
Silva, D., Feng, T. & Foster, D. J. Trajectory events across hippocampal place cells require previous experience. Nat. Neurosci. 18, 1772–1779 (2015).
O’Neill, J., Senior, T. J., Allen, K., Huxter, J. R. & Csicsvari, J. Reactivation of experience-dependent cell assembly patterns in the hippocampus. Nat. Neurosci. 11, 209–215 (2008).
Roux, L., Hu, B., Eichler, R., Stark, E. & Buzsaki, G. Sharp wave ripples during learning stabilize the hippocampal spatial map. Nat. Neurosci. 20, 845–853 (2017).
Sabatini, B. L. & Tian, L. Imaging neurotransmitter and neuromodulator dynamics in vivo with genetically encoded indicators. Neuron 108, 17–32 (2020).
The authors thank A. Mohebi for feedback on the manuscript, M.H. Plitt and T.G. Fisher for insightful discussions and E. Duvelle for helpful correspondence. This work was supported by the US Office of Naval Research (N00141812690), NIDA (DA042012), the Simons Foundation (542987SPI), the Vallee Foundation, the James S. McDonnell Foundation (L.M.G.) and the Helen Hay Whitney Foundation (M.S.).
The authors declare no competing interests.
Peer review information
Nature Reviews Neuroscience thanks M.W. Jung, A. Losonczy and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- Grid cells
Entorhinal cortex cells that fire in triangularly spaced fields that tile the whole environment.
- Place cells
Hippocampal cells that fire maximally in one or a few discrete regions of space (each cell’s ‘place field’).
- Theta sequences
Sequential spikes of multiple place cells that together encode a trajectory through space, ordered by the theta phase of each spike. Theta sequences occur during times of high theta power, typically during movement.
- Sharp-wave ripples
(SWRs). High-frequency oscillations (about 150–250 Hz) coincident with a sharp, low-frequency deflection in the local field potential. These events reflect the coincident activation of many hippocampal cells in a short period (about 50–200 ms) and typically occur during immobility.
- Replay events
Sequential spikes of multiple place cells that typically occur locked to sharp-wave ripples during immobility and that together encode a trajectory through space. In high-fidelity replay events, place cells in the sequence are reactivated according to the order in which they fired during a previous run.
Snapshots of a situation discretizing a longer continuous process that comprises an experience. As an analogy, if this snapshot were taken by a camera, the duration of the state would be the exposure time and would vary depending on the situation (for example, how dark it is outside).
How much an outcome, or state that predicts an outcome, is ‘worth’. This worth includes the amount and likelihood of the reward predicted.
- Probabilistic value
The probability that a reward will be delivered given a certain choice. Even if a choice is correct according to the task, changing the probability of reward delivery can modulate the value of preceding states.
- Cheeseboard maze
A spatial task in which rewards are hidden in a subset of holes or wells in the floor of an open arena. This task is used as a spatial memory paradigm because the animal has to remember which wells are rewarded on the basis of their position in the environment, and the reward locations can change across sessions or days.
- Reward prediction error
(RPE). The difference between the reward received and the reward expected. Positive RPEs indicate larger rewards than expected (including a reward when none was expected), whereas negative RPEs indicate smaller rewards than expected (including the absence of a reward when it was expected).
- Reinforcement learning
A set of computational theories often used for machine learning to describe how states and actions are assigned values that inform how an agent can receive the maximal reward.
- Temporal difference reinforcement learning
(TD-RL). A type of reinforcement learning in which values are updated by a reward prediction error between temporally adjacent states, such that states preceding the reward receive a ‘cached’ value prediction.
The impetus an agent feels to perform reward-seeking actions. Value is used to inform motivation and invigorate reward-seeking actions (make them faster and more efficient).
- Value function
A function of adjacent states, or states paired with actions, that computes the expected future reward in each state.
About this article
Cite this article
Sosa, M., Giocomo, L.M. Navigating for reward. Nat Rev Neurosci 22, 472–487 (2021). https://doi.org/10.1038/s41583-021-00479-z