Abstract
Humans and other animals routinely make choices between goods of different values. Choices are often made within identifiable contexts, such that an efficient learner may represent values relative to their local context. However, if goods occur across multiple contexts, a relative value code can lead to irrational choices. In this case, an absolute context-independent value is preferable to a relative code. Here we test the hypothesis that value representation is not fixed but rationally adapted to context expectations. In two experiments, we manipulated participants’ expectations about whether item values learned within local contexts would need to be subsequently compared across contexts. Despite identical learning experiences, the group whose expectations included choices across local contexts went on to learn more absolute-like representation than the group whose expectations covered only fixed local contexts. Human value representation is thus neither relative nor absolute but efficiently and rationally tuned to task demands.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data are available online on the Open Science Framework (https://osf.io/h32u6/).
Code availability
The analysis code is available on the Open Science Framework (https://osf.io/h32u6/).
References
Morgenstern, O. & Von Neumann, J. Theory of Games and Economic Behavior (Princeton Univ. Press, 1953).
Stephens, D. W. & Krebs, J. R. Foraging Theory (Princeton Univ. Press, 1986).
Sutton, R. S. et al. Introduction to Reinforcement Learning (MIT Press, 1998).
Tversky, A. & Kahneman, D. Advances in prospect theory: cumulative representation of uncertainty. J. Risk Uncertain. 5, 297–323 (1992).
Olsen, S. R., Bhandawat, V. & Wilson, R. I. Divisive normalization in olfactory population codes. Neuron 66, 287–299 (2010).
Heeger, D. J. Normalization of cell responses in cat striate cortex. Vis. Neurosci. 9, 181–197 (1992).
Khaw, M. W., Glimcher, P. W. & Louie, K. Normalized value coding explains dynamic adaptation in the human valuation process. Proc. Natl Acad. Sci. USA 114, 12696–12701 (2017).
Karni, E., Schmeidler, D. & Vind, K. On state dependent preferences and subjective probabilities. Econometrica 51, 1021–1031 (1983).
Pompilio, L., Kacelnik, A. & Behmer, S. T. State-dependent learned valuation drives choice in an invertebrate. Science 311, 1613–1615 (2006).
Bavard, S., Rustichini, A. & Palminteri, S. Two sides of the same coin: beneficial and detrimental consequences of range adaptation in human reinforcement learning. Sci. Adv. 7, eabe0340 (2021).
Carandini, M. & Heeger, D. J. Normalization as a canonical neural computation. Nat. Rev. Neurosci. 13, 51–62 (2012).
Normann, R. A. & Werblin, F. S. Control of retinal sensitivity: I. Light and dark adaptation of vertebrate rods and cones. J. Gen. Physiol. 63, 37–61 (1974).
Stewart, N., Chater, N. & Brown, G. D. A. Decision by sampling. Cogn. Psychol. 53, 1–26 (2006).
Yamada, H., Louie, K., Tymula, A. & Glimcher, P. W. Free choice shapes normalized value signals in medial orbitofrontal cortex. Nat. Commun. 9, 162 (2018).
Klein, T. A., Ullsperger, M. & Jocham, G. Learning relative values in the striatum induces violations of normative decision making. Nat. Commun. 8, 16033 (2017).
Palminteri, S. & Lebreton, M. Context-dependent outcome encoding in human reinforcement learning. Curr. Opin. Behav. Sci. 41, 144–151 (2021).
Rigoli, F. Reference effects on decision-making elicited by previous rewards. Cognition 192, 104034 (2019).
Rigoli, F., Mathys, C., Friston, K. J. & Dolan, R. J. A unifying Bayesian account of contextual effects in value-based choice. PLoS Comput. Biol. 13, e1005769 (2017).
Ciranka, S. et al. Asymmetric reinforcement learning facilitates human inference of transitive relations. Nat. Hum. Behav. 6, 555–564 (2022).
Gluth, S., Kern, N., Kortmann, M. & Vitali, C. L. Value-based attention but not divisive normalization influences decisions with multiple alternatives. Nat. Hum. Behav. 4, 634–645 (2020).
Rustichini, A., Conen, K. E., Cai, X. & Padoa-Schioppa, C. Optimal coding and neuronal adaptation in economic decisions. Nat. Commun. 8, 1208 (2017).
Bhui, R. & Gershman, S. J. Decision by sampling implements efficient coding of psychoeconomic functions. Psychol. Rev. 125, 985–1001 (2018).
Polanía, R., Woodford, M. & Ruff, C. Efficient coding of subjective value. Nat. Neurosci. 22, 134–142 (2019).
Kool, W., Gershman, S. J. & Cushman, F. A. Cost–benefit arbitration between multiple reinforcement-learning systems. Psychol. Sci. 28, 1321–1333 (2017).
Griffiths, T. L. et al. Doing more with less: meta-reasoning and meta-learning in humans and machines. Curr. Opin. Behav. Sci. 29, 24–30 (2019).
James, W. The Principles of Psychology (Henry Holt, 1890).
Anderson, J. R. The Adaptive Character of Thought (Psychology Press, 2013).
Payne, J. W., Bettman, J. R. & Johnson, E. J. The Adaptive Decision Maker (Cambridge Univ. Press, 1993).
Anderson, J. The Adaptive Character of Thought (Erlbaum, 1990).
Kahneman, D. & Tversky, A. Prospect theory: an analysis of decision under risk. Econometrica 47, 263–292 (1979).
Kriegeskorte, N., Goebel, R. & Bandettini, P. Information-based functional brain mapping. Proc. Natl Acad. Sci. USA 103, 3863–3868 (2006).
Nili, H. et al. A toolbox for representational similarity analysis. PLoS Comput. Biol. 10, e1003553 (2014).
Luyckx, F., Nili, H., Spitzer, B. & Summerfield, C. Neural structure mapping in human probabilistic reward learning. eLife 8, e42816 (2019).
Sheahan, H., Luyckx, F., Nelli, S., Teupe, C. & Summerfield, C. Neural state space alignment for magnitude generalization in humans and recurrent networks. Neuron 109, 1214–1226.e8 (2021).
Hunt, L. T. et al. Triple dissociation of attention and decision computations across prefrontal cortex. Nat. Neurosci. 21, 1471–1481 (2018).
Hertwig, R., Barron, G., Weber, E. U. & Erev, I. Decisions from experience and the effect of rare events in risky choice. Psychol. Sci. 15, 534–539 (2004).
Bavard, S., Rustichini, A. & Palminteri, S. The construction and deconstruction of sub-optimal preferences through range-adapting reinforcement learning. Preprint at bioRxiv https://doi.org/10.1101/2020.07.28.224642 (2020).
Hotaling, J. M., Jarvstad, A., Donkin, C. & Newell, B. R. How to change the weight of rare events in decisions from experience. Psychol. Sci. 30, 1767–1779 (2019).
Shenhav, A. et al. Toward a rational and mechanistic account of mental effort. Annu. Rev. Neurosci. 40, 99–124 (2017).
Prat-Carrabin, A. & Woodford, M. Efficient coding of numbers explains decision bias and noise. Preprint at bioRxiv https://doi.org/10.1101/2020.02.18.942938 (2020).
Juechems, K., Balaguer, J., Spitzer, B. & Summerfield, C. Optimal utility and probability functions for agents with finite computational precision. Proc. Natl Acad. Sci. USA 118, e2002232118 (2021).
Spektor, M. S., Gluth, S., Fontanesi, L. & Rieskamp, J. How similarity between choice options affects decisions from experience: the accentuation-of-differences model. Psychol. Rev. 126, 52–88 (2019).
Collins, A. G. E. & Frank, M. J. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis: working memory in reinforcement learning. Eur. J. Neurosci. 35, 1024–1035 (2012).
Hayes, W. M. & Wedell, D. H. Regret in experience-based decisions: the effects of expected value differences and mixed gains and losses. Decision 8, 277–294 (2021).
Edwards, D. J., Pothos, E. M. & Perlman, A. Relational versus absolute representation in categorization. Am. J. Psychol. 125, 481–497 (2012).
Collins, A. G. E. & Cockburn, J. Beyond dichotomies in reinforcement learning. Nat. Rev. Neurosci. 21, 576–586 (2020).
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J. & Daw, N. D. Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput. Biol. 13, e1005768 (2017).
Koechlin, E. Prefrontal executive function and adaptive behavior in complex environments. Curr. Opin. Neurobiol. 37, 1–6 (2016).
Gershman, S. J., Horvitz, E. J. & Tenenbaum, J. B. Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science 349, 273–278 (2015).
Hunter, L. E. & Gershman, S. J. Reference-dependent preferences arise from structure learning. Preprint at bioRxiv https://doi.org/10.1101/252692 (2018).
Lieder, F., Shenhav, A., Musslick, S. & Griffiths, T. L. Rational metareasoning and the plasticity of cognitive control. PLoS Comput. Biol. 14, e1006043 (2018).
Vlaev, I., Chater, N., Stewart, N. & Brown, G. D. A. Does the brain calculate value? Trends Cogn. Sci. 15, 546–554 (2011).
Hayden, B. Y. & Niv, Y. The case against economic values in the orbitofrontal cortex (or anywhere else in the brain). Behav. Neurosci. 135, 192–201 (2021).
Kleiner, M., Brainard, D. & Pelli, D. What’s new in Psychtoolbox-3? Perception 36, 1–16 (2007).
Fox, C. R. & Hadar, L. “Decisions from experience” = sampling error + prospect theory: reconsidering Hertwig, Barron, Weber & Erev (2004). Judgm. Decis. Mak. 1, 159–161 (2006).
Acknowledgements
A.J. was supported by a British Academy Postdoctoral Fellowship in developing this research (D-MAD, PF150005). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank P. Barr for programming Experiment 2. We thank the Palminteri lab for helpful suggestions on these data.
Author information
Authors and Affiliations
Contributions
K.J. and A.J. designed the research. T.A. and R.H. conducted the research. A.J. analysed the data. K.J. and A.J. contributed materials and analysis tools and wrote the paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Human Behaviour thanks Charley Wu, Sebastian Gluth and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Methods I–III and Results I–IX.
Rights and permissions
About this article
Cite this article
Juechems, K., Altun, T., Hira, R. et al. Human value learning and representation reflect rational adaptation to task demands. Nat Hum Behav 6, 1268–1279 (2022). https://doi.org/10.1038/s41562-022-01360-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-022-01360-4
