Abstract
Prior experience is critical for decision-making. It enables explicit representation of potential outcomes and provides training to valuation mechanisms. However, we can also make choices in the absence of prior experience by merely imagining the consequences of a new experience. Using functional magnetic resonance imaging repetition suppression in humans, we examined how neuronal representations of novel rewards can be constructed and evaluated. A likely novel experience was constructed by invoking multiple independent memories in hippocampus and medial prefrontal cortex. This construction persisted for only a short time period, during which new associations were observed between the memories for component items. Together, these findings suggest that, in the absence of direct experience, coactivation of multiple relevant memories can provide a training signal to the valuation system that allows the consequences of new experiences to be imagined and acted on.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Wimmer, G.E. & Shohamy, D. Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338, 270–273 (2012).
Peters, J. & Büchel, C. Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 66, 138–148 (2010).
Kumaran, D., Summerfield, J.J., Hassabis, D. & Maguire, E.A. Tracking the emergence of conceptual knowledge during human decision making. Neuron 63, 889–901 (2009).
Zeithamova, D., Dominick, A.L. & Preston, A.R. Hippocampal and ventral medial prefrontal activation during retrieval-mediated learning supports novel inference. Neuron 75, 168–179 (2012).
Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).
Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).
Nicolle, A. et al. An agent independent axis for executed and modeled choice in medial prefrontal cortex. Neuron 75, 1114–1121 (2012).
Johnson, A. & Redish, A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).
Steiner, A.P. & Redish, A.D. The road not taken: neural correlates of decision making in orbitofrontal cortex. Front. Neurosci. 6, 131 (2012).
Grill-Spector, K., Henson, R. & Martin, A. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14–23 (2006).
Kohn, A. Visual adaptation: physiology, mechanisms and functional benefits. J. Neurophysiol. 97, 3155–3164 (2007).
Doeller, C.F., Barry, C. & Burgess, N. Evidence for grid cells in a human memory network. Nature 463, 657–661 (2010).
Kourtzi, Z. & Kanwisher, N. Representation of perceived object shape by the human lateral occipital complex. Science 293, 1506–1509 (2001).
Becker, G.M., DeGroot, M.H. & Marschak, J. Measuring utility by a single-response sequential method. Behav. Sci. 9, 226–232 (1964).
Kable, J.W. & Glimcher, P.W. The neurobiology of decision: consensus and controversy. Neuron 63, 733–745 (2009).
Boorman, E.D., Behrens, T.E.J., Woolrich, M.W. & Rushworth, M.F.S. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743 (2009).
Hunt, L.T. et al. Mechanisms underlying cortical activity during value-guided choice. Nat. Neurosci. 15, 470–476 (2012).
Jocham, G., Hunt, L.T., Near, J. & Behrens, T.E.J. A mechanism for value-guided choice based on the excitation-inhibition balance in prefrontal cortex. Nat. Neurosci. 15, 960–961 (2012).
Hassabis, D., Kumaran, D. & Maguire, E.A. Using imagination to understand the neural basis of episodic memory. J. Neurosci. 27, 14365–14374 (2007).
Schacter, D.L. & Addis, D.R. Constructive memory: the ghosts of past and future. Nature 445, 27 (2007).
Balleine, B.W. & Dickinson, A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–419 (1998).
Tolman, E.C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).
Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
Klein-Flügge, M.C., Barron, H.C., Brodersen, K.H., Dolan, R.J. & Behrens, T.E.J. Segregated encoding of reward-identity and stimulus-reward associations in human orbitofrontal cortex. J. Neurosci. 33, 3202–3211 (2013).
Wiggs, C.L. & Martin, A. Properties and mechanisms of perceptual priming. Curr. Opin. Neurobiol. 8, 227–233 (1998).
Desimone, R. Neural mechanisms for visual memory and their role in attention. Proc. Natl. Acad. Sci. USA 93, 13494–13499 (1996).
Summerfield, C., Trittschuh, E.H., Monti, J.M., Mesulam, M.-M. & Egner, T. Neural repetition suppression reflects fulfilled perceptual expectations. Nat. Neurosci. 11, 1004–1006 (2008).
Johnson, A., Van der Meer, M.A. & Redish, A.D. Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697 (2007).
Jones, J.L. et al. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 338, 953–956 (2012).
Sutton, R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proc. 7th Int. Conf. Mach. Learn. 216, 224 (1990).
Johnson, A. & Redish, A.D. Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw. 18, 1163–1171 (2005).
Gershman, S.J., Markman, A.B. & Otto, A.R. Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol. Gen. published online, doi:10.1037/a0030844 (10 December 2012) (2012).
Kable, J.W. & Glimcher, P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633 (2007).
FitzGerald, T.H.B., Seymour, B. & Dolan, R.J. The role of human orbitofrontal cortex in value comparison for incommensurable objects. J. Neurosci. 29, 8388–8395 (2009).
Plassmann, H., O'Doherty, J. & Rangel, A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 27, 9984–9988 (2007).
Levy, I., Lazzaro, S.C., Rutledge, R.B. & Glimcher, P.W. Choice from non-choice: predicting consumer preferences from blood oxygenation level–dependent signals obtained during passive viewing. J. Neurosci. 31, 118–125 (2011).
Behrens, T.E.J., Hunt, L.T., Woolrich, M.W. & Rushworth, M.F.S. Associative learning of social value. Nature 456, 245–249 (2008).
Janowski, V., Camerer, C. & Rangel, A. Empathic choice involves vmPFC value signals that are modulated by social processing implemented in IPL. Soc. Cogn. Affect. Neurosci. 8, 201–208 (2012).
Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).
McDannald, M.A. et al. Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 35, 991–996 (2012).
Dragoi, G. & Tonegawa, S. Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469, 397–401 (2011).
Doeller, C.F. & Burgess, N. Distinct error-correcting and incidental learning of location relative to landmarks and boundaries. Proc. Natl. Acad. Sci. USA 105, 5909–5914 (2008).
Hassabis, D. & Maguire, E.A. Deconstructing episodic memory with construction. Trends Cogn. Sci. 11, 299–306 (2007).
Klein, S.B., Loftus, J. & Kihlstrom, J.F. Memory and temporal experience: the effects of episodic memory loss on an amnesic patient's ability to remember the past and imagine the future. Soc. Cogn. 20, 353–379 (2002).
Buckner, R.L. The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol. 61, 27–48 (2010).
Davachi, L. Item, context and relational episodic encoding in humans. Curr. Opin. Neurobiol. 16, 693–700 (2006).
Rudy, J.W. & Sutherland, R.J. The hippocampal formation is necessary for rats to learn and remember configural discriminations. Behav. Brain Res. 34, 97–109 (1989).
Schacter, D.L. & Addis, D.R. The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Phil. Trans. R. Soc. Lond. B 362, 773–786 (2007).
Addis, D.R., Wong, A.T. & Schacter, D.L. Remembering the past and imagining the future: common and distinct neural substrates during event construction and elaboration. Neuropsychologia 45, 1363–1377 (2007).
Harris, A., Adolphs, R., Camerer, C. & Rangel, A. Dynamic construction of stimulus values in the ventromedial prefrontal cortex. PLoS ONE 6, e21074 (2011).
Acknowledgements
We thank P. Dayan, E.A. Maguire, N. Burgess, S.W. Kennerley, L.T. Hunt, M.C. Klein-Flügge and E.D. Boorman for helpful comments on an earlier draft of the manuscript, and H. Blumenthal's Fat Duck Cookbook for recipe inspiration. This study was supported by the Wellcome Trust (grant WT088312AIA to T.E.J.B. and a Senior Investigator Award to R.J.D., 098362/Z/12/Z), the Medical Research Council (grant G1000411 to H.C.B.). The Wellcome Trust Centre for Neuroimaging is supported by core funding from the Wellcome Trust Strategic Award Grant 091593/Z/10/Z.
Author information
Authors and Affiliations
Contributions
All of the authors contributed to the design of the study and preparation of the manuscript. H.C.B. acquired the data and analyzed it with T.E.J.B.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Partial correlations between novel and component values and adaptation between related components in the unfamiliar group.
(a)(b) As shown in Fig. 2d and 2e, after removing signal attributable to the average value of the component items, there was a significant correlation between the average value of the novel items and the extent to which individual participants showed adaptation between related components across all blocks in both mPFC, (r = 0.51, p = 0.015, a), and hippocampus (r = 0.60, p = 0.004, b). (c)(d) There was also a negative effect of the component values on the same signal after variance associated with the novel compounds had been removed, in both mPFC (trend, r = -0.32, p = 0.096, c), and hippocampus (r = -0.42, p = 0.042, d). Notably, this effect, c and d, cannot be simply due to the valuation of the currently displayed component item, as the contrast in question contains each item positively and negatively equally often. Instead, one possible implication of this finding is that the plasticity between component items does not only depend on the constructed value of the novel good, but that it may be particularly prominent if the value of the novel good is surprisingly high due to the low values of the components. If it is assumed that an initial prediction of the value of the novel good is the average value of the components, then the plasticity effect is best correlated with the error between the constructed value and this prediction.
Supplementary Figure 2 Consistency of choices on the decision-making task.
These logistic regressions show how the difference in value between the left and right option predicted choice during the decision phase of the experiment. The consistency of choices made by the unfamiliar group (a) was comparable to that of the familiar group (b), with no significant difference between the two groups (beta values: 1.42 for unfamiliars, 1.46 for familiars; one-tailed Mann-Whitney U-test: p = 0.182).
Supplementary Figure 3 Group comparison: average value assigned to the 13 novel food items.
The average difference in value assigned to each novel good by the familiar and unfamiliar groups. The stars on each bar indicate the number of participants assigned to each of the goods in the repetition suppression experiment, and out of those assigned there was no significant difference in valuation between the groups.
Supplementary Figure 4 Adaptation effects in visual regions of the unfamiliar group.
(a) Left side: Between early and late blocks, there was not a significant reduction in the adaptation effect size of visual regions to repeated stimulus presentation (one-tailed t-test: t(18) = 0.50, p = 0.312, parameter estimates extracted from unfamiliar group within the ROI shown in (b), see Methods for details). This suggests that across the duration of the experiment sensitivity to adaptation effects was maintained. Right side: Whilst there was significant adaptation to repeated stimulus presentation in visual regions (one-tailed t-test: t(18) = 3.22, p = 0.002), there was no evidence for adaptation between either related components (one-tailed t-test: t(18) = 1.10, p = 0.144) or between compounds and their related components (one-tailed t-test: t(18) = 0.60, p = 0.278) (block 1 only, ROI shown in (b)). Thus, repetition suppression in visual brain regions was not specific to the construction of a novel good.
Supplementary Figure 5 Partial correlations between the value of food items and adaptation between related component items in blocks 2 and 3 of the unfamiliar group.
Partial correlations are shown as in Supplementary Fig. 1, but now show effects in blocks 2 and 3 only, with the positive correlations between compound value and adaptation between related component items (mPFC: r = 0.64, p = 0.002, a; and hippocampus: r = 0.63, p = 0.003, b; as shown in Fig. 6b and 6d respectively), and negative correlations between component value and adaptation between related component items (mPFC: r = -0.63, p = 0.003, c; hippocampus: r = -0.47, p = 0.024, d).
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–5 and Supplementary Table 1 (PDF 1090 kb)
Rights and permissions
About this article
Cite this article
Barron, H., Dolan, R. & Behrens, T. Online evaluation of novel choices by simultaneous representation of multiple memories. Nat Neurosci 16, 1492–1498 (2013). https://doi.org/10.1038/nn.3515
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.3515
This article is cited by
-
How the evaluability bias shapes transformative decisions
Synthese (2024)
-
Neural mechanisms underlying the hierarchical construction of perceived aesthetic value
Nature Communications (2023)
-
Mnemonic construction and representation of temporal structure in the hippocampal formation
Nature Communications (2022)
-
Electrocorticographic evidence of a common neurocognitive sequence for mentalizing about the self and others
Nature Communications (2022)
-
Taking stock of value in the orbitofrontal cortex
Nature Reviews Neuroscience (2022)