Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Online evaluation of novel choices by simultaneous representation of multiple memories

Subjects

Abstract

Prior experience is critical for decision-making. It enables explicit representation of potential outcomes and provides training to valuation mechanisms. However, we can also make choices in the absence of prior experience by merely imagining the consequences of a new experience. Using functional magnetic resonance imaging repetition suppression in humans, we examined how neuronal representations of novel rewards can be constructed and evaluated. A likely novel experience was constructed by invoking multiple independent memories in hippocampus and medial prefrontal cortex. This construction persisted for only a short time period, during which new associations were observed between the memories for component items. Together, these findings suggest that, in the absence of direct experience, coactivation of multiple relevant memories can provide a training signal to the valuation system that allows the consequences of new experiences to be imagined and acted on.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Experimental design.
Figure 2: Neural correlates of constructing and evaluating a novel good.
Figure 3: Sensory exposure to a novel good: comparison between the unfamiliar and familiar groups during the decision-making task.
Figure 4: Sensory exposure to a novel good: comparison between the unfamiliar and familiar groups during construction of a novel good.
Figure 5: In the absence of sensory exposure, there was evidence for the construction mechanism only in early trials: block 1 compared with blocks 2 and 3 for unfamiliar subjects.
Figure 6: In the absence of sensory exposure, repetition suppression between related components was maintained across the duration of the experiment only if participants assigned high value to the compound goods.

References

  1. 1

    Wimmer, G.E. & Shohamy, D. Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338, 270–273 (2012).

    CAS  Article  Google Scholar 

  2. 2

    Peters, J. & Büchel, C. Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 66, 138–148 (2010).

    CAS  Article  Google Scholar 

  3. 3

    Kumaran, D., Summerfield, J.J., Hassabis, D. & Maguire, E.A. Tracking the emergence of conceptual knowledge during human decision making. Neuron 63, 889–901 (2009).

    CAS  Article  Google Scholar 

  4. 4

    Zeithamova, D., Dominick, A.L. & Preston, A.R. Hippocampal and ventral medial prefrontal activation during retrieval-mediated learning supports novel inference. Neuron 75, 168–179 (2012).

    CAS  Article  Google Scholar 

  5. 5

    Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).

    CAS  Article  Google Scholar 

  6. 6

    Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).

    Article  Google Scholar 

  7. 7

    Nicolle, A. et al. An agent independent axis for executed and modeled choice in medial prefrontal cortex. Neuron 75, 1114–1121 (2012).

    CAS  Article  Google Scholar 

  8. 8

    Johnson, A. & Redish, A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).

    CAS  Article  Google Scholar 

  9. 9

    Steiner, A.P. & Redish, A.D. The road not taken: neural correlates of decision making in orbitofrontal cortex. Front. Neurosci. 6, 131 (2012).

    Article  Google Scholar 

  10. 10

    Grill-Spector, K., Henson, R. & Martin, A. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14–23 (2006).

    Article  Google Scholar 

  11. 11

    Kohn, A. Visual adaptation: physiology, mechanisms and functional benefits. J. Neurophysiol. 97, 3155–3164 (2007).

    Article  Google Scholar 

  12. 12

    Doeller, C.F., Barry, C. & Burgess, N. Evidence for grid cells in a human memory network. Nature 463, 657–661 (2010).

    CAS  Article  Google Scholar 

  13. 13

    Kourtzi, Z. & Kanwisher, N. Representation of perceived object shape by the human lateral occipital complex. Science 293, 1506–1509 (2001).

    CAS  Article  Google Scholar 

  14. 14

    Becker, G.M., DeGroot, M.H. & Marschak, J. Measuring utility by a single-response sequential method. Behav. Sci. 9, 226–232 (1964).

    CAS  Article  Google Scholar 

  15. 15

    Kable, J.W. & Glimcher, P.W. The neurobiology of decision: consensus and controversy. Neuron 63, 733–745 (2009).

    CAS  Article  Google Scholar 

  16. 16

    Boorman, E.D., Behrens, T.E.J., Woolrich, M.W. & Rushworth, M.F.S. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743 (2009).

    CAS  Article  Google Scholar 

  17. 17

    Hunt, L.T. et al. Mechanisms underlying cortical activity during value-guided choice. Nat. Neurosci. 15, 470–476 (2012).

    CAS  Article  Google Scholar 

  18. 18

    Jocham, G., Hunt, L.T., Near, J. & Behrens, T.E.J. A mechanism for value-guided choice based on the excitation-inhibition balance in prefrontal cortex. Nat. Neurosci. 15, 960–961 (2012).

    CAS  Article  Google Scholar 

  19. 19

    Hassabis, D., Kumaran, D. & Maguire, E.A. Using imagination to understand the neural basis of episodic memory. J. Neurosci. 27, 14365–14374 (2007).

    CAS  Article  Google Scholar 

  20. 20

    Schacter, D.L. & Addis, D.R. Constructive memory: the ghosts of past and future. Nature 445, 27 (2007).

    CAS  Article  Google Scholar 

  21. 21

    Balleine, B.W. & Dickinson, A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–419 (1998).

    CAS  Article  Google Scholar 

  22. 22

    Tolman, E.C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).

    CAS  Article  Google Scholar 

  23. 23

    Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).

    CAS  Article  Google Scholar 

  24. 24

    Klein-Flügge, M.C., Barron, H.C., Brodersen, K.H., Dolan, R.J. & Behrens, T.E.J. Segregated encoding of reward-identity and stimulus-reward associations in human orbitofrontal cortex. J. Neurosci. 33, 3202–3211 (2013).

    Article  Google Scholar 

  25. 25

    Wiggs, C.L. & Martin, A. Properties and mechanisms of perceptual priming. Curr. Opin. Neurobiol. 8, 227–233 (1998).

    CAS  Article  Google Scholar 

  26. 26

    Desimone, R. Neural mechanisms for visual memory and their role in attention. Proc. Natl. Acad. Sci. USA 93, 13494–13499 (1996).

    CAS  Article  Google Scholar 

  27. 27

    Summerfield, C., Trittschuh, E.H., Monti, J.M., Mesulam, M.-M. & Egner, T. Neural repetition suppression reflects fulfilled perceptual expectations. Nat. Neurosci. 11, 1004–1006 (2008).

    CAS  Article  Google Scholar 

  28. 28

    Johnson, A., Van der Meer, M.A. & Redish, A.D. Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697 (2007).

    CAS  Article  Google Scholar 

  29. 29

    Jones, J.L. et al. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 338, 953–956 (2012).

    CAS  Article  Google Scholar 

  30. 30

    Sutton, R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proc. 7th Int. Conf. Mach. Learn. 216, 224 (1990).

    Google Scholar 

  31. 31

    Johnson, A. & Redish, A.D. Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw. 18, 1163–1171 (2005).

    Article  Google Scholar 

  32. 32

    Gershman, S.J., Markman, A.B. & Otto, A.R. Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol. Gen. published online, doi:10.1037/a0030844 (10 December 2012) (2012).

  33. 33

    Kable, J.W. & Glimcher, P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633 (2007).

    CAS  Article  Google Scholar 

  34. 34

    FitzGerald, T.H.B., Seymour, B. & Dolan, R.J. The role of human orbitofrontal cortex in value comparison for incommensurable objects. J. Neurosci. 29, 8388–8395 (2009).

    CAS  Article  Google Scholar 

  35. 35

    Plassmann, H., O'Doherty, J. & Rangel, A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 27, 9984–9988 (2007).

    CAS  Article  Google Scholar 

  36. 36

    Levy, I., Lazzaro, S.C., Rutledge, R.B. & Glimcher, P.W. Choice from non-choice: predicting consumer preferences from blood oxygenation level–dependent signals obtained during passive viewing. J. Neurosci. 31, 118–125 (2011).

    CAS  Article  Google Scholar 

  37. 37

    Behrens, T.E.J., Hunt, L.T., Woolrich, M.W. & Rushworth, M.F.S. Associative learning of social value. Nature 456, 245–249 (2008).

    CAS  Article  Google Scholar 

  38. 38

    Janowski, V., Camerer, C. & Rangel, A. Empathic choice involves vmPFC value signals that are modulated by social processing implemented in IPL. Soc. Cogn. Affect. Neurosci. 8, 201–208 (2012).

    Article  Google Scholar 

  39. 39

    Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    CAS  Article  Google Scholar 

  40. 40

    McDannald, M.A. et al. Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 35, 991–996 (2012).

    Article  Google Scholar 

  41. 41

    Dragoi, G. & Tonegawa, S. Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469, 397–401 (2011).

    CAS  Article  Google Scholar 

  42. 42

    Doeller, C.F. & Burgess, N. Distinct error-correcting and incidental learning of location relative to landmarks and boundaries. Proc. Natl. Acad. Sci. USA 105, 5909–5914 (2008).

    CAS  Article  Google Scholar 

  43. 43

    Hassabis, D. & Maguire, E.A. Deconstructing episodic memory with construction. Trends Cogn. Sci. 11, 299–306 (2007).

    Article  Google Scholar 

  44. 44

    Klein, S.B., Loftus, J. & Kihlstrom, J.F. Memory and temporal experience: the effects of episodic memory loss on an amnesic patient's ability to remember the past and imagine the future. Soc. Cogn. 20, 353–379 (2002).

    Article  Google Scholar 

  45. 45

    Buckner, R.L. The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol. 61, 27–48 (2010).

    Article  Google Scholar 

  46. 46

    Davachi, L. Item, context and relational episodic encoding in humans. Curr. Opin. Neurobiol. 16, 693–700 (2006).

    CAS  Article  Google Scholar 

  47. 47

    Rudy, J.W. & Sutherland, R.J. The hippocampal formation is necessary for rats to learn and remember configural discriminations. Behav. Brain Res. 34, 97–109 (1989).

    CAS  Article  Google Scholar 

  48. 48

    Schacter, D.L. & Addis, D.R. The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Phil. Trans. R. Soc. Lond. B 362, 773–786 (2007).

    Article  Google Scholar 

  49. 49

    Addis, D.R., Wong, A.T. & Schacter, D.L. Remembering the past and imagining the future: common and distinct neural substrates during event construction and elaboration. Neuropsychologia 45, 1363–1377 (2007).

    Article  Google Scholar 

  50. 50

    Harris, A., Adolphs, R., Camerer, C. & Rangel, A. Dynamic construction of stimulus values in the ventromedial prefrontal cortex. PLoS ONE 6, e21074 (2011).

    CAS  Article  Google Scholar 

Download references

Acknowledgements

We thank P. Dayan, E.A. Maguire, N. Burgess, S.W. Kennerley, L.T. Hunt, M.C. Klein-Flügge and E.D. Boorman for helpful comments on an earlier draft of the manuscript, and H. Blumenthal's Fat Duck Cookbook for recipe inspiration. This study was supported by the Wellcome Trust (grant WT088312AIA to T.E.J.B. and a Senior Investigator Award to R.J.D., 098362/Z/12/Z), the Medical Research Council (grant G1000411 to H.C.B.). The Wellcome Trust Centre for Neuroimaging is supported by core funding from the Wellcome Trust Strategic Award Grant 091593/Z/10/Z.

Author information

Affiliations

Authors

Contributions

All of the authors contributed to the design of the study and preparation of the manuscript. H.C.B. acquired the data and analyzed it with T.E.J.B.

Corresponding authors

Correspondence to Helen C Barron or Timothy E J Behrens.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Partial correlations between novel and component values and adaptation between related components in the unfamiliar group.

(a)(b) As shown in Fig. 2d and 2e, after removing signal attributable to the average value of the component items, there was a significant correlation between the average value of the novel items and the extent to which individual participants showed adaptation between related components across all blocks in both mPFC, (r = 0.51, p = 0.015, a), and hippocampus (r = 0.60, p = 0.004, b). (c)(d) There was also a negative effect of the component values on the same signal after variance associated with the novel compounds had been removed, in both mPFC (trend, r = -0.32, p = 0.096, c), and hippocampus (r = -0.42, p = 0.042, d). Notably, this effect, c and d, cannot be simply due to the valuation of the currently displayed component item, as the contrast in question contains each item positively and negatively equally often. Instead, one possible implication of this finding is that the plasticity between component items does not only depend on the constructed value of the novel good, but that it may be particularly prominent if the value of the novel good is surprisingly high due to the low values of the components. If it is assumed that an initial prediction of the value of the novel good is the average value of the components, then the plasticity effect is best correlated with the error between the constructed value and this prediction.

Supplementary Figure 2 Consistency of choices on the decision-making task.

These logistic regressions show how the difference in value between the left and right option predicted choice during the decision phase of the experiment. The consistency of choices made by the unfamiliar group (a) was comparable to that of the familiar group (b), with no significant difference between the two groups (beta values: 1.42 for unfamiliars, 1.46 for familiars; one-tailed Mann-Whitney U-test: p = 0.182).

Supplementary Figure 3 Group comparison: average value assigned to the 13 novel food items.

The average difference in value assigned to each novel good by the familiar and unfamiliar groups. The stars on each bar indicate the number of participants assigned to each of the goods in the repetition suppression experiment, and out of those assigned there was no significant difference in valuation between the groups.

Supplementary Figure 4 Adaptation effects in visual regions of the unfamiliar group.

(a) Left side: Between early and late blocks, there was not a significant reduction in the adaptation effect size of visual regions to repeated stimulus presentation (one-tailed t-test: t(18) = 0.50, p = 0.312, parameter estimates extracted from unfamiliar group within the ROI shown in (b), see Methods for details). This suggests that across the duration of the experiment sensitivity to adaptation effects was maintained. Right side: Whilst there was significant adaptation to repeated stimulus presentation in visual regions (one-tailed t-test: t(18) = 3.22, p = 0.002), there was no evidence for adaptation between either related components (one-tailed t-test: t(18) = 1.10, p = 0.144) or between compounds and their related components (one-tailed t-test: t(18) = 0.60, p = 0.278) (block 1 only, ROI shown in (b)). Thus, repetition suppression in visual brain regions was not specific to the construction of a novel good.

Supplementary Figure 5 Partial correlations between the value of food items and adaptation between related component items in blocks 2 and 3 of the unfamiliar group.

Partial correlations are shown as in Supplementary Fig. 1, but now show effects in blocks 2 and 3 only, with the positive correlations between compound value and adaptation between related component items (mPFC: r = 0.64, p = 0.002, a; and hippocampus: r = 0.63, p = 0.003, b; as shown in Fig. 6b and 6d respectively), and negative correlations between component value and adaptation between related component items (mPFC: r = -0.63, p = 0.003, c; hippocampus: r = -0.47, p = 0.024, d).

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–5 and Supplementary Table 1 (PDF 1090 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Barron, H., Dolan, R. & Behrens, T. Online evaluation of novel choices by simultaneous representation of multiple memories. Nat Neurosci 16, 1492–1498 (2013). https://doi.org/10.1038/nn.3515

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing