Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Vicarious trial and error

Key Points

  • Vicarious trial and error (VTE) is an occasional behaviour observed in rats at choice points on mazes. During VTE events, rats pause and look back and forth on the maze.

  • Decision making in mammals is accomplished through an interaction of multiple action-selection systems, including a deliberative system and a procedural system. Deliberative decision making entails a concrete, serial search-and-evaluate process that requires an explicit imagination of future outcomes, whereas procedural decision making entails cached action-chains that the animal learns to carry out in specific situations.

  • VTE occurs during times in which rats show flexible behaviour. When decisions can be made using either the deliberative system or the procedural system (that is, when these decisions are placed in conflict), VTE occurs when decisions are driven by deliberative systems.

  • During VTE, hippocampal representations sweep down the potential trajectories, one at a time. During VTE, ventral striatal and orbitofrontal representations reactivate reward-related signals.

  • As animals automate behaviours, both VTE and the hippocampal and ventral striatal processes diminish and other neurophysiological processes reflecting procedural decision making are observed.

  • The hypothesis put forward here is that VTE is a behavioural reflection of underlying deliberative processes.

Abstract

When rats come to a decision point, they sometimes pause and look back and forth as if deliberating over the choice; at other times, they proceed as if they have already made their decision. In the 1930s, this pause-and-look behaviour was termed 'vicarious trial and error' (VTE), with the implication that the rat was 'thinking about the future'. The discovery in 2007 that the firing of hippocampal place cells gives rise to alternating representations of each of the potential path options in a serial manner during VTE suggested a possible neural mechanism that could underlie the representations of future outcomes. More-recent experiments examining VTE in rats suggest that there are direct parallels to human processes of deliberative decision making, working memory and mental time travel.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Vicarious trial and error.

References

  1. Muenzinger, K. F. & Gentry, E. Tone discrimination in white rats. J. Comp. Psychol. 12, 195–206 (1931).

    Article  Google Scholar 

  2. Tolman, E. C. Prediction of vicarious trial and error by means of the schematic sowbug. Psychol. Rev. 46, 318–336 (1939).

    Article  Google Scholar 

  3. Tolman, E. C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).

    Article  CAS  PubMed  Google Scholar 

  4. Hull, C. L. Principles of Behavior (Appleton-Century-Crofts, 1943).

    Google Scholar 

  5. Turing, A. On computable numbers, with an application to the entscheidungs problem. Proc. Lond. Math. Soc. 42, 230–265 (1937).

    Article  Google Scholar 

  6. Wiener, N. Cybernetics, or Control and Communications in the Animal and the Machine (Hermann, 1948).

    Google Scholar 

  7. Shannon, C. A mathematical theory of communication. Bell System Techn. J. 27, 379–423, 623–656 (1948).

    Article  Google Scholar 

  8. Newell, A., Shaw, J. C. & Simon, H. A. in Proc. Int. Conf. Information Process. [online], (UNESCO, 1959).

    Google Scholar 

  9. Simon, H. A behavioral model of rational choice. Q. J. Econ. 69, 99–118 (1955).

    Article  Google Scholar 

  10. O'Keefe, J. & Nadel, L. The Hippocampus as a Cognitive Map (Clarendon Press, 1978). This is a comprehensive book laying out the hypothesis that the hippocampus has a key role in Tolman's cognitive map. Also, this book explains one of the first proposed algorithmic differences between what is now called deliberative and procedural learning.

    Google Scholar 

  11. Wilson, M. A. & McNaughton, B. L. Dynamics of the hippocampal ensemble code for space. Science 261, 1055–1058 (1993).

    Article  CAS  PubMed  Google Scholar 

  12. Zhang, K. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. J. Neurosci. 16, 2112–2126 (1996).

    Article  CAS  PubMed  Google Scholar 

  13. Brown, E. N., Frank, L. M., Tang, D., Quirk, M. C. & Wilson, M. A. A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells. J. Neurosci. 18, 7411–7425 (1998).

    Article  CAS  PubMed  Google Scholar 

  14. Johnson, A. & Redish, A. D. Neural ensembles CA3 transiently encode paths forward animal decision point. J. Neurosci. 27, 12176–12189 (2007). This paper shows for the first time that hippocampal sequences encode future outcomes at choice points.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Foster, D. J. & Wilson, M. A. Hippocampal theta sequences. Hippocampus 17, 1093–1099 (2007).

    Article  PubMed  Google Scholar 

  16. Pfeiffer, B. E. & Foster, D. J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. O'Craven, K. M. & Kanwisher, N. Mental imagery of faces and places activates corresponding stimulus-specific brain regions. J. Cogn. Neurosci. 12, 1013–1023 (2000).

    Article  CAS  PubMed  Google Scholar 

  18. Pearson J., Naselaris, T., Holmes, E. A. & Kosslyn S. M. Mental imagery: functional mechanisms clinical application. Trends Cogn. Sci. 19, 590–602 (2015). This is a clear review of the now established fact that imagination activates the same circuits as perception in humans, which can be used (in reference 19) to identify mental time travel in non-human animals.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Johnson, A., Fenton, A. A., Kentros, C. & Redish, A. D. Looking for cognition in the structure within the noise. Trends Cogn. Sci. 13, 55–64 (2009). This paper lays out the logic and mathematics allowing for identification of neural representations of cognitive events (such as mental time travel) from neural signals in non-human animals.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Payne, J., Bettman, J. & Johnson, E. The Adaptive Decision Maker (Cambridge Univ. Press, 1993).

    Book  Google Scholar 

  21. Gilbert, D. T. & Wilson, T. D. Prospection: experiencing the future. Science 317, 1351–1354 (2007).

    Article  CAS  PubMed  Google Scholar 

  22. Rangel, A., Camerer, C. & Montague, P. R. A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 9, 545–556 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Kurth-Nelson, Z., Bickel, W. K. & Redish, A. D. A theoretical account of cognitive effects in delay discounting. Eur. J. Neurosci. 35, 1052–1064 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  24. Redish, A. D. The Mind within the Brain: How we Make Decisions and How Those Decisions Go Wrong (Oxford Univ. Press, 2013). This is a thorough review of the concepts and theory underlying the multiple-decision systems hypothesis.

    Google Scholar 

  25. Buckner, R. L. & Carroll, D. C. Self-projection and the brain. Trends Cognitive Sci. 11, 49–57 (2007).

    Article  Google Scholar 

  26. Schacter, D. L. & Addis, D. R. in Predictions in the Brain: Using Our Past to Generate a Future (ed. Bar, M.) 58–69 (Oxford Univ. Press, 2011).

    Book  Google Scholar 

  27. Gilbert, D. T. & Wilson, T. D. Why the brain talks to itself: sources of error in emotional prediction. Phil. Trans. R Soc. B 364, 1335–1341 (2009).

    Article  PubMed  Google Scholar 

  28. Phelps, E., Lempert, K. M. & Sokol-Hessner, P. Emotion and decision making: multiple modulatory circuits. Annu. Rev. Neurosci. 37, 263–287 (2014).

    Article  CAS  PubMed  Google Scholar 

  29. Redish, A. D., Schultheiss, N. W. & Carter, E. C. The computational complexity of valuation and motivational forces in decision-making processes. Curr. Top. Behav. Neurosci. http://dx.doi.org/10.1007/7854_2015_375 (2015).

  30. Redish, A. D., Jensen, S., Johnson, A. & Kurth-Nelson, Z. Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychol. Rev. 114, 784–805 (2007).

    Article  PubMed  Google Scholar 

  31. Gershman, S. J., Blei, D. & Niv, Y. Context, learning and extinction. Psychol. Rev. 117, 197–209 (2010).

    Article  PubMed  Google Scholar 

  32. Niv, Y. et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J. Neurosci. 35, 8145–8157 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Johnson, A. & Crowe, D. A. Revisiting Tolman, his theories and cognitive maps. Cogn. Critique 1, 43–72 (2009).

    Google Scholar 

  34. Redish, A. D. Beyond the Cognitive Map: From Place Cells to Episodic Memory (MIT Press, 1999).

    Book  Google Scholar 

  35. Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    Article  CAS  PubMed  Google Scholar 

  36. Winstanley, C. A. et al. in Cognitive Search: Evolution, Algorithms, and the Brain (eds Hills, T. et al.) 125–156 (MIT Press, 2012).

    Google Scholar 

  37. McKenzie, S. et al. Hippocampal representation of related and opposing memories develop within distinct, hierarchically organized neural schemas. Neuron 83, 202–215 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Lichtenstein, S. & Slovic, P. (eds) The Construction of Preference (Cambridge Univ. Press, 2006).

    Book  Google Scholar 

  39. Kahneman, D. Thinking, Fast, and Slow (Farrar, 2011).

    Google Scholar 

  40. Benoit, R. G., Gilbert, S. J. & Burgess, P. W. A neural mechanism mediating the impact of episodic prospection on farsighted decisions. J. Neurosci. 31, 6771–6779 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Peters, J. & Büchel, C. Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal–mediotemporal interactions. Neuron 66, 138–148 (2010).

    Article  CAS  PubMed  Google Scholar 

  42. Kwan, D. et al. Future decision-making without episodic mental time travel. Hippocampus 22, 1215–1219 (2012).

    Article  PubMed  Google Scholar 

  43. Hassabis, D., Kumaran, D., Vann, S. D. & Maguire, E. A. Patients with hippocampal amnesia cannot imagine experiences. Proc. Natl Acad. Sci. USA 104, 1726–1731 (2007). This paper shows that the hippocampus is critical for the ability to create episodic futures in humans.

    Article  CAS  PubMed  Google Scholar 

  44. Lebreton, M. et al. A critical role for the hippocampus in the valuation of imagined outcomes. PLoS Biol. 11, e1001684 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Wang, J. X., Cohen, N. J. & Voss, J. L. Covert rapid action-memory simulation (crams): a hypothesis of hippocampal–prefrontal interactions for adaptive behavior. Neurobiol. Learn. Mem. 117, 22–33 (2015).

    Article  PubMed  Google Scholar 

  46. Spiers, H. J. & Gilbert, S. J. Solving the detour problem in navigation: a model of prefrontal and hippocampal interactions. Front. Hum. Neurosci. 9, 125 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  47. Hikosaka, O. et al. Parallel neural networks for learning sequential procedures. Trends Neurosci. 22, 464–471 (1999).

    Article  CAS  PubMed  Google Scholar 

  48. Lee, S. W., Shimoko, S. & O'Doherty, J. P. Neural computations underlying arbitration between model-based and model-free learning. Neuron 81, 687–699 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Dezfouli, A. & Balleine, B. Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 35, 1036–1051 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Johnson, A., van der Meer, M. A. & Redish, A. D. Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697 (2007).

    Article  CAS  PubMed  Google Scholar 

  51. Gardner, R. S. et al. A secondary working memory challenge preserves primary place strategies despite overtraining. Learn. Mem. 20, 648–656 (2013). On the classic Tolman–Hull plus maze, VTE arises with deliberative choices.

    Article  PubMed  Google Scholar 

  52. Schmidt, B. J., Papale, A. E., Redish, A. D. & Markus, E. J. Conflict between place and response navigation strategies: effects on vicarious trial and error (VTE) behaviors. Learn. Mem. 20, 130–138 (2013).

    Article  PubMed  Google Scholar 

  53. Smith, K. S. & Graybiel, A. M. A dual operator view of habitual behavior reflecting cortical striatal dynamics. Neuron 79, 361–374 (2013). This paper shows that VTE is negatively related to dorsolateral striatal representations that reflect procedural strategies (task bracketing).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Meer, M. A. A., Johnson, A., Schmitzer-Torbert, N. C. & Redish, A. D. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 67, 25–32 (2010). This paper shows that hippocampal representations encode future options during VTE, ventral striatal representations encode potential rewards during VTE, and dorsolateral striatum does neither, instead slowly developing situation-action representations in line with the automation of behaviour.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Thorn, C. A., Atallah, H., Howe, M. & Graybiel, A. M. Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 66, 781–795 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  56. Regier, P. S., Amemiya, S. & Redish, A. D. Hippocampus and subregions of the dorsal striatum respond differently to a behavioral strategy change on a spatial navigation task. J. Neurophysiol. 114, 1399–1416 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  57. Niv, Y., Joel, D. & Dayan, P. A normative perspective on motivation. Trends Cogn. Sci. 10, 375–381 (2006). This paper gives an explication of the difference between search-and-evaluate and cached-action-chain decision systems.

    Article  PubMed  Google Scholar 

  58. Johnson, A., Varberg, Z., Benhardus, J., Maahs, A. & Schrater, P. The hippocampus and exploration: dynamically evolving behavior and neural representations. Front. Hum. Neurosci. 6, 216 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  59. Balleine, B. W. & Dickinson, A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–419 (1998).

    Article  CAS  PubMed  Google Scholar 

  60. Liljeholm, M., Tricomi, E., O'Doherty, J. P. & Balleine, B. W. Neural correlates of instrumental contingency learning: differential effects of actionreward conjunction and disjunction. J. Neurosci. 31, 2474–2480 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  61. Krajbich, I. & Rangel, A. Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions. Proc. Natl Acad. Sci. USA 108, 13852–13857 (2011). Investigating human saccade–fixate–saccade sequences, this paper suggests a decision model and shows that it is consistent with the behavioural selections made by subjects.

    Article  CAS  PubMed  Google Scholar 

  62. Packard, M. G. & McGaugh, J. L. Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol. Learn. Mem. 65, 65–72 (1996).

    Article  CAS  PubMed  Google Scholar 

  63. Yin, H. H., Knowlton, B. & Balleine, B. W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 19, 181–189 (2004).

    Article  PubMed  Google Scholar 

  64. Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S. & Redish, A. D. Hippocampal replay is not a simple function of experience. Neuron 65, 695–705 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Blumenthal, A., Steiner, A., Seeland, K. D. & Redish, A. D. Effects of pharmacological manipulations of NMDA-receptors on deliberation in the Multiple-T task. Neurobiol. Learn. Mem. 95, 376–384 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Steiner, A. & Redish, A. D. The road not taken: neural correlates of decision making in orbitofrontal cortex. Front. Decision Neurosci. 6, 131 (2012).

    Google Scholar 

  67. Steiner, A. & Redish, A. D. Behavioral and neurophysiological correlates of regret in rat decision-making on a neuroeconomic task. Nat. Neurosci. 17, 995–1002 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  68. Papale, A., Stott, J. J., Powell, N. J., Regier, P. S. & Redish, A. D. Interactions between deliberation and delay-discounting in rats. Cogn. Affect. Behav. Neurosci. 12, 513–526 (2012). On the spatial delay-discounting task, VTE occurs during the titration phase, when the rat is making flexible choices, not during the exploitation phase, when the rat has automated its behaviour, even though the values of the two options are equal in the exploitation phase.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Atance, C. M. & O'Neill, D. K. Episodic future thinking. Trends Cogn. Sci. 5, 533–539 (2001).

    Article  PubMed  Google Scholar 

  70. Seidenbecher, T., Laxmi, T. R., Stork, O. & Pape, H. C. Amygdalar and hippocampal theta rhythm synchronization during fear memory retrieval. Science 301, 846–850 (2003).

    Article  CAS  PubMed  Google Scholar 

  71. Schmidt, R. et al. Single-trial phase precession in the hippocampus. J. Neurosci. 29, 13232–13241 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  72. Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S. & Redish, A. D. Segmentation of spatial experience by hippocampal θ sequences. Nat. Neurosci. 15, 1032–1039 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  73. Hassabis, D. & Maguire, E. A. in Predictions in the Brain: Using our Past to Generate a Future (ed. Bar, M.) 70–82 (Oxford Univ. Press, 2011).

    Book  Google Scholar 

  74. Howard, L. R. et al. The hippocampus and entorhinal cortex encode the path and Euclidean distances to goals during navigation. Curr. Biol. 24, 1331–1340 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Wikenheiser, A. M. & Redish, A. D. Hippocampal theta sequences reflect current goals. Nat. Neurosci. 18, 289–294 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  76. Royer, S., Sirota, A., Patel, J. & Buzsaki, G. Distinct representations and theta dynamics in dorsal and ventral hippocampus. J. Neurosci. 30, 1777–1787 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  77. Stensola, H. et al. The entorhinal grid map is discretized. Nature 492, 72–78 (2012).

    Article  CAS  PubMed  Google Scholar 

  78. Dragoi, G. & Buzsaki, G. Temporal encoding of place sequences by hippocampal cell assemblies. Neuron 50, 145–157 (2006).

    Article  CAS  PubMed  Google Scholar 

  79. Lisman, J. & Redish, A. D. Prediction, sequences and the hippocampus. Phil. Trans. R. Soc. B 364, 1193–1201 (2009).

    Article  PubMed  Google Scholar 

  80. Buzsaki, G. Hippocampal sharp wave-ripple: a cognitive biomarker for episodic memory and planning. Hippocampus 25, 1073–1188 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  81. Diba, K. & Buzsaki, G. Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 10, 1241–1242 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  82. Yamamoto, J., Suh, J., Takeuchi, D. & Tonegawa, S. Successful execution of working memory linked to synchronized high-frequency gamma oscillations. Cell 157, 845–857 (2014).

    Article  CAS  PubMed  Google Scholar 

  83. Schmitzer-Torbert, N. C. & Redish, A. D. Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple-T task. J. Neurophysiol. 91, 2259–2272 (2004).

    Article  PubMed  Google Scholar 

  84. Tse, D. et al. Schemas and memory consolidation. Science 316, 76–82 (2007).

    Article  CAS  PubMed  Google Scholar 

  85. Hu, D. & Amsel, A. A simple test of the vicarious trial-and-error hypothesis of hippocampal function. Proc. Natl Acad. Sci. USA 92, 5506–5509 (1995).

    Article  CAS  PubMed  Google Scholar 

  86. Bett, D. et al. The neural substrates of deliberative decision making: contrasting effects of hippocampus lesions on performance and vicarious trial-and-error behavior in a spatial memory task and a visual discrimination task. Front. Behav. Neurosci. 6, 70 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  87. Bett, D., Murdoch, L. H., Wood, E. R. & Dudchenko, P. A. Hippocampus, delay discounting, and vicarious trial-and-error. Hippocampus 25, 643–654 (2015).

    Article  PubMed  Google Scholar 

  88. Barnes, C. A., Nadel, L. & Honig, W. K. Spatial memory deficit in senescent rats. Can. J. Psychol. 34, 29–39 (1980).

    Article  CAS  PubMed  Google Scholar 

  89. Breton, Y. A., Seeland K. D. & Redish, A. D. Aging impairs deliberation and behavioral flexibility in inter-temporal choice. Front. Aging Neurosci. 7, 41 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  90. Robbe, D. et al. Cannabinoids reveal importance of spike timing coordination in hippocampal function. Nat. Neurosci. 9, 1526–1533 (2006).

    Article  CAS  PubMed  Google Scholar 

  91. Robbe, D. & Buzsaki, G. Alteration of theta timescale dynamics of hippocampal place cells by a cannabinoid is associated with memory impairment. J. Neurosci. 29, 12597–12605 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  92. Papale, A. E. Hippocampal Representations on the Spatial Delay Discounting Task. Thesis, Univ. Minnesota (2015).

    Google Scholar 

  93. Amemiya, S., Noji, T., Kubota, N., Nishijima, T. & Kita, I. Noradrenergic modulation of vicarious trial-and-error behavior during a spatial decision-making task in rats. Neuroscience 265, 291–301 (2014).

    Article  CAS  PubMed  Google Scholar 

  94. Amemiya, S. & Redish, A. D. Manipulating decisiveness in decision making: effects of clonidine on hippocampal search strategies. J. Neurosci. 36, 814–827 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  95. Seifert, W. Neurobiology of the Hippocampus (Academic Press, 1983).

    Google Scholar 

  96. Teyler, T. J. and DiScenna, P. The hippocampal memory indexing theory. Behav. Neurosci. 100, 147–154 (1986).

    Article  CAS  PubMed  Google Scholar 

  97. Nadel, L., Samsonovich, A., Ryan, L. & Moscovitch, M. Multiple trace theory of human memory: computational, neuroimaging, and neuropsychological results. Hippocampus 10, 352–368 (2000).

    Article  CAS  PubMed  Google Scholar 

  98. Doll, B. B., Duncan, K. D., Simon, D. A., Shohamy, D. & Daw, N. D. Model-based choices involve prospective neural activity. Nat. Neurosci. 18, 767–772 (2015). In humans, prospection entails the representation of specific future outcomes.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  99. Killcross, S. & Coutureau, E. Coordination of actions and habits in the medial prefrontral cortex of rats. Cereb. Cortex 13, 400–408 (2003).

    Article  PubMed  Google Scholar 

  100. Sharpe, M. J. & Killcross, S. The prelimbic cortex directs attention toward predictive cues during fear learning. Learn. Mem. 22, 289–293 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  101. Spellman, T. et al. Hippocampal-prefrontal input supports spatial encoding in working memory. Nature 522, 309–314 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  102. Ito, H. T., Zhang, S. J., Witter, M. P., Moser, E. I. & Moser, M. B. A prefrontal–thalamo–hippocampal circuit for goal-directed spatial navigation. Nature 522, 50–55 (2015).

    Article  CAS  PubMed  Google Scholar 

  103. Kolb, B. in The Cerebral Cortex of the Rat (eds Kolb & Tees R. C.) 437–458 (MIT Press, 1990).

    Google Scholar 

  104. Dalley, J. W., Cardinal, R. N. & Robbins, T. W. Prefrontal executive and cognitive functions in rodents: neural and neurochemical substrates. Neurosci. Biobehav. Rev. 28, 771–784 (2004).

    Article  CAS  PubMed  Google Scholar 

  105. Wallis, J. D. Cross-species studies of orbitofrontal cortex and value-based decision-making. Nat. Neurosci. 15, 13–19 (2012).

    Article  CAS  Google Scholar 

  106. Jung, M. W., Qin, Y., McNaughton, B. L. & Barnes, C. A. Firing characteristics of deep layer neurons in prefrontal cortex in rats performing spatial working memory tasks. Cereb. Cortex 8, 437–450 (1998).

    Article  CAS  PubMed  Google Scholar 

  107. Powell, N. J. & Redish, A. D. Complex neural codes in rat prelimbic cortex are stable across days on a spatial decision task. Front. Behav. Neurosci. 8, (2014).

  108. Jones, M. W. & Wilson, M. A. Theta rhythms coordinate hippocampal–prefrontal interactions in a spatial memory task. PLoS Biol. 3, e402 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  109. Hyman, J. M., Zilli, E. A., Paley, A. M. & Hasselmo, M. E. Working memory performance correlates with prefrontal-hippocampal theta interactions but not with prefrontal neuron firing rates. Front. Integr.Neurosci. 4, 2 (2010).

    PubMed  PubMed Central  Google Scholar 

  110. Ragozzino, M. E., Detrick, S. & Kesner, R. P. Involvement of the prelimbic–infralimbic areas of the rodent prefrontal cortex in behavioral flexibility for place and response learning. J. Neurosci. 19, 4585–4594 (1999).

    Article  CAS  PubMed  Google Scholar 

  111. Rich, E. L. & Shapiro, M. L. Prelimbic/infralimbic inactivation impairs memory for multiple task switches, but not flexible selection of familiar tasks. J. Neurosci. 27, 4747–4755 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  112. Benchenane, K. et al. Coherent theta oscillations and reorganization of spike timing in the hippocampal–prefrontal network upon learning. Neuron 66, 921–936 (2010).

    Article  CAS  PubMed  Google Scholar 

  113. O'Neill, P. K., Gordon, J. A. & Sigurdsson, T. Theta oscillations in the medial prefrontal cortex are modulated by spatial working memory and synchronize with the hippocampus through its ventral subregion. J. Neurosci. 33, 14211–14224 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  114. Engel, A. K., Fries, P. & Singer, W. Dynamic predictions: oscillations and synchrony in top-down processing. Nat. Rev. Neurosci. 2, 704–716 (2001).

    Article  CAS  PubMed  Google Scholar 

  115. Kucewicz, M. T., Tricklebank, M. D., Bogacz, R. & Jones, M. W. Dysfunctional prefrontal cortical network activity and interactions following cannabinoid receptor activation. J. Neurosci. 31, 15560–15568 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  116. Mogenson, G. J., Jones, D. L. & Yim, C. Y. From motivation to action: functional interface between the limbic system and the motor system. Prog. Neurobiol. 14, 69–97 (1980).

    Article  CAS  PubMed  Google Scholar 

  117. McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y. & Schoenbaum, G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 31, 2700–2705 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  118. Jones, J. L. et al. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 338, 953–956 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  119. Padoa-Schioppa, C. & Assad, J. A. Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  120. Strait, C. E., Blanchard, T. C. & Hayden, B. Y. Reward value comparison via mutual inhibition in ventromedial prefrontal cortex. Neuron 82, 1357–1366 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  121. Strait, C. E., Sleezer, B. J. & Hayden, B. Y. Signatures of value comparison in ventral striatum neurons. PLoS Biol. 13, e1002173 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  122. Bechara, A., Tranel, D. & Damasio, H. Characterization of the decision-making deficit of patients with ventromedial prefrontal cortex lesions. Brain 123, 2189–2202 (2000).

    Article  PubMed  Google Scholar 

  123. Zald, D. H. & Rauch, S. L. (eds) The Orbitofrontal Cortex (Oxford Univ. Press, 2008).

    Google Scholar 

  124. Fellows, L. K. Deciding how to decide: ventromedial frontal lobe damage affects information acquisition in multi-attribute decision making. Brain 129, 944–952 (2006).

    Article  PubMed  Google Scholar 

  125. Bray, S., Shimojo, S. & O'Doherty, J. P. Human medial orbitofrontal cortex is recruited during experience of imagined and real rewards. J. Neurophysiol. 103, 2506–2512 (2010).

    Article  PubMed  Google Scholar 

  126. Tremblay, L. & Schultz, W. Relative reward preference in primate orbitofrontal cortex. Nature 398, 704–708 (1999).

    Article  CAS  PubMed  Google Scholar 

  127. Nicola, S. M., Yun, I. A., Wakabayashi, K. T. & Fields, H. L. Cue-evoked firing of nucleus accumbens neurons encodes motivational significance during a discriminative stimulus task. J. Neurophysiol. 91, 1840–1865 (2004).

    Article  PubMed  Google Scholar 

  128. Roitman, M. F., Wheeler, R. A. & Carelli, R. M. Nucleus accumbens neurons are innately tuned for rewarding and aversive taste stimuli, encode their predictors, and are linked to motor output. Neuron 45, 587–597 (2005).

    Article  CAS  PubMed  Google Scholar 

  129. Stott, J. J. & Redish, A. D. A functional difference in information processing between orbitofrontal cortex ventral striatum during decision-making behaviour. Phil. Trans. R. Soc. B 369, 20130472 (2014). In simultaneous recordings from the ventral striatum and the orbitofrontal cortex, ventral striatal representations show covert reward-related information before the turn-around during VTE, but orbitofrontal representations only show covert reward-related information after the turn-around point.

    Article  PubMed  Google Scholar 

  130. German, P. W. & Fields, H. L. Rat nucleus accumbens neurons persistently encode locations associated with morphine reward. J. Neurophysiol. 97, 2094–2106 (2007).

    Article  PubMed  Google Scholar 

  131. Moorman, D. E. & Aston-Jones, G. Orbitofrontal cortical neurons encode expectation-driving initiation of reward-seeking. J. Neurosci. 34, 10234–10246 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  132. van der Meer, M. A. A. & Redish, A. D. Covert expectation-of-reward in rat ventral striatum at decision points. Front. Integr. Neurosci. 3, 1–15 (2009).

    PubMed  PubMed Central  Google Scholar 

  133. Wilson, R. C., Takahashi, Y. K., Schoenbaum, G. & Niv, Y. Orbitofrontal cortex as a cognitive map of task space. Neuron 81, 267–279 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  134. Fenton, A. A. et al. Attention-like modulation of hippocampus place cell discharge. J. Neurosci. 30, 4613–4625 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  135. Wikenheiser, A. & Redish, A. D. Changes in reward contingency modulate the trial to trial variability of hippocampal place cells. J. Neurophysiol. 106, 589–598 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  136. Berke, J. D. & Eichenbaum, H. Striatal versus hippocampal representations during win-stay maze performance. J. Neurophysiol. 101, 1575–1587 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  137. Schmitzer-Torbert, N. C. & Redish, A. D. Task-dependent encoding of space and events by striatal neurons is dependent on neural subtype. Neuroscience 153, 349–360 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  138. Jog, M. S., Kubota, Y., Connolly, C. I., Hillegaart, V. & Graybiel, A. M. Building neural representations of habits. Science 286, 1746–1749 (1999).

    Article  Google Scholar 

  139. Jin, X. & Costa, R. M. Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature 466, 457–462 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  140. Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z. & Graybiel, A. M. Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 437, 1158–1161 (2005).

    Article  CAS  PubMed  Google Scholar 

  141. Barnes, T. D. et al. Advance cueing produces enhanced action-boundary patterns of spike activity in the sensorimotor striatum. J. Neurophysiol. 105, 1861–1878 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  142. Kubota, Y. et al. Stable encoding of task structure coexists with flexible coding of task events in sensorimotor striatum. J. Neurophysiol. 102, 2142–2160 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  143. Corbit, L. H. & Balleine, B. W. The role of the hippocampus in instrumental conditioning. J. Neurosci. 20, 4233–4239 (2000).

    Article  CAS  PubMed  Google Scholar 

  144. Ragozzino, M. E., Ragozzino, K. E., Mizumori, S. J. Y. & Kesner, R. P. The role of the dorsomedial striatum in behavioral flexibility for repsonse and visual cue discrimination learning. Behav. Neurosci. 116, 105–115 (2002).

    Article  PubMed  PubMed Central  Google Scholar 

  145. Ostlund, S. B., Winterbauer, N. E. & Balleine, B. W. Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. J. Neurosci. 29, 8280–8287 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  146. Corbit, L. H. & Janak, P. H. Posterior dorsomedial striatum is critical for both selective instrumental and Pavlovian reward learning. Eur. J. Neurosci. 31, 1312–1321 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  147. Bower, G. H. in Studies in Mathematical Learning Theory (eds Bush, R. R. & Estes, W. K.) 109–124 (Stanford Univ. Press, 1959).

    Google Scholar 

  148. Whishaw, I. Q. & Kolb, B. The Behavior of the Laboratory Rat: A Handbook with Tests (Oxford Univ. Press, 2004).

    Book  Google Scholar 

  149. Monaco, J. D., Rao, G., Roth, E. D. & Knierim, J. J. Attentive scanning behavior drives one-trial potentiation of hippocampal place fields. Nat. Neurosci. 17, 725–731 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  150. Baddeley, A. Working Memory, Thought and Action (Oxford Univ. Press, 2007).

    Book  Google Scholar 

  151. Fuster, J. The Prefrontal Cortex (Academic Press, 2008).

    Book  Google Scholar 

  152. Guthrie, E. R. The Psychology of Learning (Harpers, 1935).

    Google Scholar 

  153. Rodgers, R. J., Cao, B. J., Dalvi, A. & Holmes, A. Animals models of anxiety: an ethological perspective. Brazil. J. Med. Biol. Res. 30, 289–304 (1997).

    Article  CAS  Google Scholar 

  154. Jackson, J. C. & Redish, A. D. Detecting dynamical changes within a simulated neural ensemble using a measure of representational quality. Network 14, 629–645 (2003).

    Article  PubMed  Google Scholar 

  155. Adhikari, A. Distributed circuits underlying anxiety. Front. Behav. Neurosci. 8, 112 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  156. Allsop, S. A., Vander Weele, C. M., Wichmann, R. & Tye, K. M. Optogenetic insights on the relationship between anxiety-related behaviors and social deficits. Front. Behav. Neurosci. 8, 241 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  157. Adhikari, A., Topiwala, M. A. & Gordon, J. A. Synchronized activity between the ventral hippocampus and the medial prefrontal cortex during anxiety. Neuron 28, 257–269 (2010).

    Article  CAS  Google Scholar 

  158. Adhikari, A., Topwala, M. & Gordon, J. A. Single units in the medial prefrontal cortex with anxiety-related firing patterns are preferentially influenced by ventral hippocampal activity. Neuron 71, 898–910 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  159. Grewal, S. S., Shepherd, J. K., Bill, D. J., Fletcher, A. & Dourish, C. T. Behavioural and pharmacological characterisation of the canopy stretched attend posture test as a model of anxiety in mice and rats. Psychopharmacology 133, 29–38 (1997).

    Article  CAS  PubMed  Google Scholar 

  160. Choi, J. S. & Kim, J. J. Amygdala regulates risk of predation in rats foraging in a dynamic fear environment. Proc. Natl Acad. Sci. USA 107, 21773–21777 (2010).

    Article  CAS  PubMed  Google Scholar 

  161. Amir, A., Lee, S. C., Headley, D. B., Herzallah, M. M. & Pare, D. Amygdala signaling during foraging in a hazardous environment. J. Neurosci. 35, 12994–13005 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  162. Voss, J. L. et al. Spontaneous revisitation during visual exploration as a link among strategic behavior, learning, and the hippocampus. Proc. Natl Acad. Sci. USA 108, E402–E409 (2011).

    Article  CAS  PubMed  Google Scholar 

  163. Reutskaja, E., Nagel, R., Camerer, C. F. & Rangel, A. Search dynamics in consumer choice under time pressure: an eye-tracking study. Am. Econom. Rev. 101, 900–926 (2011).

    Article  Google Scholar 

  164. Treisman, A. M. & Gelade, G. A feature-integration theory of attention. Cogn. Psychol. 12, 97–136 (1980).

    Article  CAS  PubMed  Google Scholar 

  165. Kruse, J. M., Overmier, J. B., Konz, W. A. & Rokke, E. Pavlovian conditioned stimulus effects upon instrumental choice behavior are reinforcer specific. Learn. Motiv. 14, 165–181 (1983).

    Article  Google Scholar 

  166. Balleine, B. W. & Ostlund, S. B. Still at the choice-point: action selection and initiation in instrumental conditioning. Ann. NY Acad. Sci. 1104, 147–171 (2007).

    Article  PubMed  Google Scholar 

  167. Talmi, D., Seymour, B., Dayan, P. & Dolan, R. J. Human Pavlovian-instrumental transfer. J. Neurosci. 28, 360–368 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  168. Bray, S., Rangel, A., Shimojo, S., Balleine, B. & O'Doherty, J. P. The neural mechanisms underlying the influence of Pavlovian cues on human decision making. J. Neurosci. 28, 5861–5866 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  169. Damasio, A. Looking for Spinoza: Joy, Sorrow, and the Feeling Brain (Mariner, 2003).

    Google Scholar 

  170. Balleine, B. W. in The Behavior of the Laboratory Rat: A Handbook with Tests (eds Whishaw, I. Q. & Kolb, B.) 436–446 (Oxford Univ. Press, 2004).

    Book  Google Scholar 

  171. Gold, J. I. & Shadlen, M. N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 36, 299–308 (2002).

    Article  CAS  PubMed  Google Scholar 

  172. Hanks, T. D. et al. Distinct relationships of parietal and prefrontal cortices to evidence accumulation. Nature 520, 220–223 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The author thanks all of the students and colleagues who have worked with him over the years looking at different aspects of VTE, both for helpful discussions and for the experimental and theoretical work on these questions. The author also thanks A. Johnson, K. Smith, J. Stott, S. Amemiya and Y. Breton for comments on drafts of this manuscript, and J. Voss for helpful discussions. Time to work on this manuscript was funded by MH080318 and DA030672.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to A. David Redish.

Ethics declarations

Competing interests

The author declares no competing financial interests.

PowerPoint slides

Glossary

Neural ensembles

A set of cells recorded separably, but simultaneously, generally from a single brain structure during behaviour. Information represented within ensembles can be decoded from a sufficiently large ensemble.

Mental time travel

A process in which one imagines another time and place. Sometimes referred to as episodic future thinking.

Schema

An expertise-dependent representation of the structure of the world, identifying the important parameters over which the world varies.

Cognitive map

A world representation on which one can plan. Currently, the term is generally used in a spatial context, but Tolman's original use of the term was closer to the current use of the word 'schema'.

Deliberative decision making

A process in which one imagines potential future outcomes (serially and individually) and then selects an action to get to that specific future outcome.

Procedural decision making

A process in which one learns an action chain and the ability to recognize the situations in which to release it. Performance is rapid, but is usually learned slowly, and is inflexible once learned.

Local field potential

(LFP). Low-frequency voltage signals reflecting neural processing. In the hippocampus, the LFP is marked by two contrasting states: theta and large-amplitude irregular activity.

Hippocampal theta sequences

Sequences of firing of hippocampal place cells within a single theta cycle, generally proceeding from the location of the rat forward towards potential goals. Also called a 'hippocampal sweep'.

Hippocampal SWR sequences

A sequence of firing of hippocampal cells within a sharp wave ripple complex (SWR). Originally referred to as 'replay' (because early observations identified sequences repeated in order), but now known to include other sequences including backwards along the path of the animal or along unexplored shortcuts and novel paths.

Task bracketing

A phenomenon observed in dorsolateral striatal neural ensembles, in which the cells show increased activity at the beginning of an action chain sequence.

Covert reward signals

Signals reflecting imagined representation of reward, as detected from the patterns of activity of reward-associated neuronal ensembles during non-rewarded events.

Integration to threshold

A psychological theory of decision making whereby one accumulates evidence for one decision over another; when the evidence for one decision reaches a threshold, the decision is made.

Visuospatial scratchpad

A component hypothesized to underlie working memory processes, in which neural circuits normally used for perception are used to hold imagined information for processing.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Redish, A. Vicarious trial and error. Nat Rev Neurosci 17, 147–159 (2016). https://doi.org/10.1038/nrn.2015.30

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrn.2015.30

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing