Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

A framework for studying the neurobiology of value-based decision making

Key Points

  • Most behavioural and computational models of decision making assume that the following five processes are carried out at the time the decision is made: representation, action valuation, action selection, outcome valuation, and learning.

  • On the basis of a sizeable body of animal and human behavioural evidence, several groups have proposed the existence of three different types of valuation systems: Pavlovian, habitual and goal-directed systems.

  • Pavlovian systems assign value to only a small set of 'prepared' behaviours and thus have a limited behavioural repertoire. Nevertheless, they might be the driving force behind behaviours with important economic consequences (for example, overeating). Examples include preparatory behaviours, such as approaching a cue that predicts food, and consummatory behaviours, such as ingesting available food.

  • Habit valuation systems learn to assign values to stimulus–response associations on the basis of previous experience through a process of trial-and-error. Examples of habits include a smoker's desire to have a cigarette at particular times of day (for example, after a meal) and a rat's tendency to forage in a cue-dependent location after sufficient training.

  • Goal-directed systems assign values to actions by computing action–outcome associations and then evaluating the rewards that are associated with the different outcomes. An example of a goal-directed behaviour is the decision what to eat at a new restaurant.

  • An important difference between habitual and goal-directed systems has to do with how they respond to changes in the environment. The goal-directed system updates the value of an action as soon as the value of its outcome changes, whereas the habit system only learns with repeated experience.

  • The values computed by the three systems can be modulated by factors such as the risk that is associated with the decision, the time delay to the outcomes, and social considerations.

  • The quality of the decisions made by an animal depend on how its brain assigns control to the different valuation systems in situations in which it has to make a choice between several potential actions that are assigned conflicting values.

  • The learning properties of the habit system seem to be well-described by simple reinforcement algorithms, such as Q-learning. Some of the key computations that are predicted by these models are instantiated in the dopamine system.

Abstract

Neuroeconomics is the study of the neurobiological and computational basis of value-based decision making. Its goal is to provide a biologically based account of human behaviour that can be applied in both the natural and the social sciences. This Review proposes a framework to investigate different aspects of the neurobiology of decision making. The framework allows us to bring together recent findings in the field, highlight some of the most important outstanding problems, define a common lexicon that bridges the different disciplines that inform neuroeconomics, and point the way to future applications.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Basic computations involved in making a choice.
Figure 2: Conflict between the valuation systems.

References

  1. 1

    Busemeyer, J. R. & Johnson, J. G. in Handbook of Judgment and Decision Making (eds Koehler, D. & Narvey, N.) 133–154 (Blackwell Publishing Co., New York, 2004).

    Google Scholar 

  2. 2

    Mas-Colell, A., Whinston, M. & Green, J. Microeconomic Theory (Cambridge Univ. Press, Cambridge, 1995).

    Google Scholar 

  3. 3

    Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  4. 4

    Dickison, A. & Balleine, B. W. in Steven's Handbook of Experimental Psychology Vol. 3 Learning, Motivation & Emotion (ed. Gallistel, C.) 497–533 (Wiley & Sons, New York, 2002).

    Google Scholar 

  5. 5

    Dayan, P. in Better Than Conscious? Implications for Performance and Institutional Analysis (eds Engel, C. & Singer, W.) 51–70 (MIT Press, Cambridge, Massachusetts, 2008).

    Google Scholar 

  6. 6

    Balleine, B. W., Daw, N. & O'Doherty, J. in Neuroeconomics: Decision-Making and the Brain (eds Glimcher, P. W., Fehr, E., Camerer, C. & Poldrack, R. A.) 365–385 (Elsevier, New York, 2008).

    Google Scholar 

  7. 7

    Bouton, M. E. Learning and Behavior: A Contemporary Synthesis (Sinauer Associates, Inc., Sunderland, Massachusetts, 2007). This book reviews a large amount of evidence pointing to multiple valuation systems being active in value-based decision making.

    Google Scholar 

  8. 8

    Dayan, P. & Seymour, B. in Neuroeconomics: Decision Making and the Brain (eds Glimcher, P. W., Camerer, C. F., Fehr, E. & Poldrack, R. A.) 175–191 (Elsevier, New York, 2008).

    Google Scholar 

  9. 9

    Dayan, P., Niv, Y., Seymour, B. & Daw, N. D. The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153–1160 (2006). This paper provided several models of how “pathological behaviours” can arise from the competition process between Pavlovian, habitual and goal-directed valuation systems.

    PubMed  Google Scholar 

  10. 10

    Keay, K. A. & Bandler, R. Parallel circuits mediating distinct emotional coping reactions to different types of stress. Neurosci. Biobehav. Rev. 25, 669–678 (2001).

    CAS  PubMed  Google Scholar 

  11. 11

    Cardinal, R. N., Parkinson, J. A., Hall, J. & Everitt, B. J. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 26, 321–352 (2002).

    PubMed  Google Scholar 

  12. 12

    Holland, P. C. & Gallagher, M. Amygdala-frontal interactions and reward expectancy. Curr. Opin. Neurobiol. 14, 148–155 (2004).

    CAS  PubMed  Google Scholar 

  13. 13

    Fendt, M. & Fanselow, M. S. The neuroanatomical and neurochemical basis of conditioned fear. Neurosci. Biobehav. Rev. 23, 743–760 (1999).

    CAS  PubMed  Google Scholar 

  14. 14

    Adams, D. B. Brain mechanisms of aggressive behavior: an updated review. Neurosci. Biobehav. Rev. 30, 304–318 (2006).

    PubMed  Google Scholar 

  15. 15

    Niv, Y. in Neuroscience (Hebrew University, Jerusalem, 2007).

    Google Scholar 

  16. 16

    Dayan, P. & Abbott, L. R. Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems (MIT Press, Cambridge, Massachusetts, 1999).

    Google Scholar 

  17. 17

    Balleine, B. W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 86, 717–730 (2005). This important paper reviews a large amount of evidence pointing to multiple valuation systems being active in value-based decision making.

    CAS  PubMed  Google Scholar 

  18. 18

    Yin, H. H. & Knowlton, B. J. The role of the basal ganglia in habit formation. Nature Rev. Neurosci. 7, 464–476 (2006).

    CAS  Google Scholar 

  19. 19

    Killcross, S. & Coutureau, E. Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 13, 400–408 (2003).

    PubMed  Google Scholar 

  20. 20

    Coutureau, E. & Killcross, S. Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behav. Brain Res. 146, 167–174 (2003).

    PubMed  Google Scholar 

  21. 21

    Yin, H. H., Knowlton, B. J. & Balleine, B. W. Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci. 22, 505–512 (2005).

    PubMed  Google Scholar 

  22. 22

    Wallis, J. D. & Miller, E. K. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur. J. Neurosci. 18, 2069–2081 (2003).

    PubMed  Google Scholar 

  23. 23

    Padoa-Schioppa, C. & Assad, J. A. Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226 (2006). This paper showed that neurons in the monkey OFC encode the goal value of individual rewarding objects (for example, different liquids) irrespective of the action that needs to be taken to obtain them.

    CAS  PubMed  PubMed Central  Google Scholar 

  24. 24

    Wallis, J. D. Orbitofrontal cortex and its contribution to decision-making. Annu. Rev. Neurosci. 30, 31–56 (2007).

    CAS  PubMed  Google Scholar 

  25. 25

    Barraclough, D. J., Conroy, M. L. & Lee, D. Prefrontal cortex and decision making in a mixed-strategy game. Nature Neurosci. 7, 404–410 (2004).

    CAS  PubMed  Google Scholar 

  26. 26

    Schoenbaum, G. & Roesch, M. Orbitofrontal cortex, associative learning, and expectancies. Neuron 47, 633–636 (2005).

    CAS  PubMed  PubMed Central  Google Scholar 

  27. 27

    Tom, S. M., Fox, C. R., Trepel, C. & Poldrack, R. A. The neural basis of loss aversion in decision-making under risk. Science 315, 515–518 (2007). This fMRI study showed that the striatal-OFC network encodes a value signal at the time of the goal-directed choice that is consistent with the properties of PT. Furthermore, the study presented evidence that suggests that both the appetitive and the aversive aspects of goal-directed decisions might be encoded in a common valuation network.

    CAS  PubMed  Google Scholar 

  28. 28

    Plassmann, H., O'Doherty, J. & Rangel, A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 27, 9984–9988 (2007).

    CAS  PubMed  Google Scholar 

  29. 29

    Hare, T., O'Doherty, J., Camerer, C. F., Schultz, W. & Rangel, A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci. (in the press).

  30. 30

    Paulus, M. P. & Frank, L. R. Ventromedial prefrontal cortex activation is critical for preference judgments. Neuroreport 14, 1311–1315 (2003).

    PubMed  Google Scholar 

  31. 31

    Erk, S., Spitzer, M., Wunderlich, A. P., Galley, L. & Walter, H. Cultural objects modulate reward circuitry. Neuroreport 13, 2499–2503 (2002).

    PubMed  Google Scholar 

  32. 32

    Fellows, L. K. & Farah, M. J. The role of ventromedial prefrontal cortex in decision making: judgment under uncertainty or judgment per se? Cereb. Cortex 17, 2669–2674 (2007).

    PubMed  Google Scholar 

  33. 33

    Lengyel, M. & Dayan, P. Hippocampal contributions to control: the third way. NIPS [online] (2007).

    Google Scholar 

  34. 34

    Montague, P. R. Why Choose This Book? (Dutton, 2006).

    Google Scholar 

  35. 35

    Fehr, E. & Camerer, C. F. Social neuroeconomics: the neural circuitry of social preferences. Trends Cogn. Sci. 11, 419–427 (2007).

    PubMed  Google Scholar 

  36. 36

    Lee, D. Game theory and neural basis of social decision making. Nature Neurosci. 11, 404–409 (2008).

    CAS  PubMed  Google Scholar 

  37. 37

    Platt, M. L. & Huettel, S. A. Risky business: the neuroeconomics of decision making under uncertainty. Nature Neurosci. 11, 398–403 (2008).

    CAS  PubMed  Google Scholar 

  38. 38

    Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, J. S. & Stein, M. B. Increased activation in the right insula during risk-taking decision making is related to harm avoidance and neuroticism. Neuroimage 19, 1439–1448 (2003).

    PubMed  Google Scholar 

  39. 39

    Leland, D. S. & Paulus, M. P. Increased risk-taking decision-making but not altered response to punishment in stimulant-using young adults. Drug Alcohol Depend. 78, 83–90 (2005).

    PubMed  Google Scholar 

  40. 40

    Paulus, M. P. et al. Prefrontal, parietal, and temporal cortex networks underlie decision-making in the presence of uncertainty. Neuroimage 13, 91–100 (2001).

    CAS  PubMed  Google Scholar 

  41. 41

    Huettel, S. A., Song, A. W. & McCarthy, G. Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. J. Neurosci. 25, 3304–3311 (2005).

    CAS  PubMed  Google Scholar 

  42. 42

    Bossaerts, P. & Hsu, M. in Neuroeconomics: Decision Making and the Brain (eds Glimcher, P. W., Camerer, C. F., Fehr, E. & Poldrack, R. A.) 351–364 (Elsevier, New York, 2008).

    Google Scholar 

  43. 43

    Preuschoff, K. & Bossaerts, P. Adding prediction risk to the theory of reward learning. Ann. NY Acad. Sci. 1104, 135–146 (2007).

    PubMed  Google Scholar 

  44. 44

    Preuschoff, K., Bossaerts, P. & Quartz, S. R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381–390 (2006).

    CAS  Google Scholar 

  45. 45

    Tobler, P. N., O'Doherty, J. P., Dolan, R. J. & Schultz, W. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 97, 1621–1632 (2007).

    PubMed  Google Scholar 

  46. 46

    Rolls, E. T., McCabe, C. & Redoute, J. Expected value, reward outcome, and temporal difference error representations in a probabilistic decision task. Cereb. Cortex 18, 652–663 (2007).

    PubMed  Google Scholar 

  47. 47

    Dreher, J. C., Kohn, P. & Berman, K. F. Neural coding of distinct statistical properties of reward information in humans. Cereb. Cortex 16, 561–573 (2006).

    PubMed  Google Scholar 

  48. 48

    Preuschoff, K., Quartz, S. R. & Bossaerts, P. Human insula activation reflects prediction errors as well as risk. J. Neurosci. 28, 2745–2752 (2008). This fMRI study shows that the human insula encodes risk-prediction errors that could be used to learn the riskiness of different options and that are complementary to reward-prediction errors.

    CAS  PubMed  Google Scholar 

  49. 49

    Tobler, P. N., Fiorillo, C. D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).

    CAS  PubMed  Google Scholar 

  50. 50

    Platt, M. L. & Glimcher, P. W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).

    CAS  PubMed  Google Scholar 

  51. 51

    Camerer, C. F. & Weber, M. Recent developments in modelling preferences: uncertainty and ambiguity. J. Risk Uncertain. 5, 325–370 (1992).

    Google Scholar 

  52. 52

    Hsu, M., Bhatt, M., Adolphs, R., Tranel, D. & Camerer, C. F. Neural systems responding to degrees of uncertainty in human decision-making. Science 310, 1680–1683 (2005).

    CAS  PubMed  Google Scholar 

  53. 53

    Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T. & Platt, M. L. Neural signatures of economic preferences for risk and ambiguity. Neuron 49, 765–775 (2006).

    CAS  PubMed  Google Scholar 

  54. 54

    Hertwig, R., Barron, G., Weber, E. U. & Erev, I. Decisions from experience and the effect of rare events in risky choice. Psychol. Sci. 15, 534–539 (2004).

    PubMed  Google Scholar 

  55. 55

    Weller, J. A., Levin, I. P., Shiv, B. & Bechara, A. Neural correlates of adaptive decision making for risky gains and losses. Psychol. Sci. 18, 958–964 (2007).

    PubMed  Google Scholar 

  56. 56

    De Martino, B., Kumaran, D., Seymour, B. & Dolan, R. J. Frames, biases, and rational decision-making in the human brain. Science 313, 684–687 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  57. 57

    Frederick, S., Loewenstein, G. & O'Donoghue, T. Time discounting and time preference: a critical review. J. Econ. Lit. 40, 351–401 (2002).

    Google Scholar 

  58. 58

    McClure, S. M., Laibson, D. I., Loewenstein, G. & Cohen, J. D. Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507 (2004). This fMRI study argued that competing goal-directed valuation systems play a part in decisions that involve choosing between immediate small monetary payoffs and larger but delayed payoffs.

    CAS  PubMed  Google Scholar 

  59. 59

    McClure, S. M., Ericson, K. M., Laibson, D. I., Loewenstein, G. & Cohen, J. D. Time discounting for primary rewards. J. Neurosci. 27, 5796–5804 (2007).

    CAS  PubMed  Google Scholar 

  60. 60

    Berns, G. S., Laibson, D. & Loewenstein, G. Intertemporal choice - toward an integrative framework. Trends Cogn. Sci. 11, 482–488 (2007).

    PubMed  Google Scholar 

  61. 61

    Kable, J. W. & Glimcher, P. W. The neural correlates of subjective value during intertemporal choice. Nature Neurosci. 10, 1625–1633 (2007). This fMRI study argued that a single goal-directed valuation system plays a part in decisions that involve choosing between immediate small monetary payoffs and larger but delayed payoffs.

    CAS  PubMed  Google Scholar 

  62. 62

    Read, D., Frederick, S., Orsel, B. & Rahman, J. Four score and seven years ago from now: the “date/delay” effect in temporal discounting. Manage. Sci. 51, 1326–1335 (1997).

    Google Scholar 

  63. 63

    Mischel, W. & Underwood, B. Instrumental ideation in delay of gratification. Child Dev. 45, 1083–1088 (1974).

    CAS  PubMed  Google Scholar 

  64. 64

    Wilson, M. & Daly, M. Do pretty women inspire men to discount the future? Proc. Biol. Sci. 271 (Suppl 4), S177–S179 (2004).

    PubMed  PubMed Central  Google Scholar 

  65. 65

    Berns, G. S. et al. Neurobiological substrates of dread. Science 312, 754–758 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  66. 66

    Loewenstein, G. Anticipation and the valuation of delayed consumption. Econ. J. 97, 666–684 (1987).

    Google Scholar 

  67. 67

    Stevens, J. R., Hallinan, E. V. & Hauser, M. D. The ecology and evolution of patience in two New World monkeys. Biol. Lett. 1, 223–226 (2005).

    PubMed  PubMed Central  Google Scholar 

  68. 68

    Herrnstein, R. J. Relative and absolute strength of response as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267–272 (1961).

    CAS  PubMed  PubMed Central  Google Scholar 

  69. 69

    Mazur, J. E. Estimation of indifference points with an adjusting-delay procedure. J. Exp. Anal. Behav. 49, 37–47 (1988).

    CAS  PubMed  PubMed Central  Google Scholar 

  70. 70

    Corrado, G. S., Sugrue, L. P., Seung, H. S. & Newsome, W. T. Linear-nonlinear-poisson models of primate choice dynamics. J. Exp. Anal. Behav. 84, 581–617 (2005).

    PubMed  PubMed Central  Google Scholar 

  71. 71

    Newsome, W. T., Britten, K. H. & Movshon, J. A. Neuronal correlates of a perceptual decision. Nature 341, 52–54 (1989).

    CAS  PubMed  Google Scholar 

  72. 72

    Kim, J. N. & Shadlen, M. N. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nature Neurosci. 2, 176–185 (1999).

    PubMed  Google Scholar 

  73. 73

    Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  74. 74

    Gold, J. I. & Shadlen, M. N. Banburisms and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 36, 299–308 (2002).

    CAS  PubMed  Google Scholar 

  75. 75

    Gold, J. I. & Shadlen, M. N. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10–16 (2001).

    PubMed  Google Scholar 

  76. 76

    Heekeren, H. R., Marrett, S. & Ungerleider, L. G. The neural systems that mediate human perceptual decision making. Nature Rev. Neurosci. 9, 467–479 (2008).

    CAS  Google Scholar 

  77. 77

    Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neurosci. 8, 1704–1711 (2005). This paper proposed a theoretical model of how the brain might assign control to the different goal and habitual systems.

    CAS  PubMed  Google Scholar 

  78. 78

    Frank, M. J., Seeberger, L. C. & O'Reilly, R. C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306, 1940–1943 (2004).

    CAS  PubMed  Google Scholar 

  79. 79

    Frank, M. J. Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19, 1120–1136 (2006).

    PubMed  Google Scholar 

  80. 80

    de Araujo, I. E., Rolls, E. T., Kringelbach, M. L., McGlone, F. & Phillips, N. Taste-olfactory convergence, and the representation of the pleasantness of flavour, in the human brain. Eur. J. Neurosci. 18, 2059–2068 (2003).

    PubMed  Google Scholar 

  81. 81

    de Araujo, I. E., Kringelbach, M. L., Rolls, E. T. & McGlone, F. Human cortical responses to water in the mouth, and the effects of thirst. J. Neurophysiol. 90, 1865–1876 (2003).

    PubMed  PubMed Central  Google Scholar 

  82. 82

    Anderson, A. K. et al. Dissociated neural representations of intensity and valence in human olfaction. Nature Neurosci. 6, 196–202 (2003).

    CAS  PubMed  Google Scholar 

  83. 83

    de Araujo, I. E., Rolls, E. T., Velazco, M. I., Margot, C. & Cayeux, I. Cognitive modulation of olfactory processing. Neuron 46, 671–679 (2005).

    CAS  PubMed  Google Scholar 

  84. 84

    McClure, S. M. et al. Neural correlates of behavioral preference for culturally familiar drinks. Neuron 44, 379–387 (2004).

    CAS  PubMed  Google Scholar 

  85. 85

    Kringelbach, M. L., O'Doherty, J., Rolls, E. T. & Andrews, C. Activation of the human orbitofrontal cortex to a liquid food stimulus is correlated with its subjective pleasantness. Cereb. Cortex 13, 1064–1071 (2003).

    CAS  PubMed  Google Scholar 

  86. 86

    Small, D. M. et al. Dissociation of neural representation of intensity and affective valuation in human gustation. Neuron 39, 701–711 (2003).

    CAS  PubMed  Google Scholar 

  87. 87

    Blood, A. J. & Zatorre, R. J. Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proc. Natl Acad. Sci. USA 98, 11818–11823 (2001).

    CAS  PubMed  Google Scholar 

  88. 88

    O'Doherty, J. et al. Sensory-specific satiety-related olfactory activation of the human orbitofrontal cortex. Neuroreport 11, 399–403 (2000).

    CAS  PubMed  Google Scholar 

  89. 89

    Small, D. M., Zatorre, R. J., Dagher, A., Evans, A. C. & Jones-Gotman, M. Changes in brain activity related to eating chocolate: from pleasure to aversion. Brain 124, 1720–1733 (2001).

    CAS  PubMed  Google Scholar 

  90. 90

    Breiter, H. C., Aharon, I., Kahneman, D., Dale, A. & Shizgal, P. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30, 619–639 (2001).

    CAS  PubMed  PubMed Central  Google Scholar 

  91. 91

    Knutson, B., Fong, G. W., Adams, C. M., Varner, J. L. & Hommer, D. Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport 12, 3683–3687 (2001).

    CAS  PubMed  Google Scholar 

  92. 92

    Zink, C. F., Pagnoni, G., Martin-Skurski, M. E., Chappelow, J. C. & Berns, G. S. Human striatal responses to monetary reward depend on saliency. Neuron 42, 509–517 (2004).

    CAS  PubMed  Google Scholar 

  93. 93

    Peyron, R. et al. Haemodynamic brain responses to acute pain in humans: sensory and attentional networks. Brain 122, 1765–1780 (1999).

    PubMed  Google Scholar 

  94. 94

    Davis, K. D., Taylor, S. J., Crawley, A. P., Wood, M. L. & Mikulis, D. J. Functional MRI of pain- and attention-related activations in the human cingulate cortex. J. Neurophysiol. 77, 3370–3380 (1997).

    CAS  PubMed  Google Scholar 

  95. 95

    Seo, H. & Lee, D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 27, 8366–8377 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  96. 96

    Pecina, S., Smith, K. S. & Berridge, K. C. Hedonic hot spots in the brain. Neuroscientist 12, 500–511 (2006).

    PubMed  Google Scholar 

  97. 97

    Berridge, K. C. & Robinson, T. E. Parsing reward. Trends Neurosci. 26, 507–513 (2003).

    CAS  PubMed  Google Scholar 

  98. 98

    Berridge, K. C. Pleasures of the brain. Brain Cogn. 52, 106–128 (2003).

    PubMed  Google Scholar 

  99. 99

    Plassmann, H., O'Doherty, J., Shiv, B. & Rangel, A. Marketing actions can modulate neural representations of experienced pleasantness. Proc. Natl Acad. Sci. USA 105, 1050–1054 (2008). This paper showed that the level of “experienced pleasantness” encoded in the medial OFC at the time of consuming a wine is modulated by subjects' beliefs about the price of the wine that they are drinking.

    CAS  PubMed  Google Scholar 

  100. 100

    Montague, P. R., King-Casas, B. & Cohen, J. D. Imaging valuation models in human choice. Annu. Rev. Neurosci. 29, 417–448 (2006).

    CAS  PubMed  Google Scholar 

  101. 101

    Tremblay, L., Hollerman, J. R. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol. 80, 964–977 (1998).

    CAS  PubMed  Google Scholar 

  102. 102

    Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304–309 (1998).

    CAS  PubMed  Google Scholar 

  103. 103

    Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997). This seminal paper proposed the connection between the prediction-error component of reinforcement-learning models and the behaviour of dopamine cells.

    CAS  PubMed  Google Scholar 

  104. 104

    Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).

    CAS  PubMed  Google Scholar 

  105. 105

    Schultz, W. Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288 (2007).

    CAS  PubMed  Google Scholar 

  106. 106

    Schultz, W. Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr. Opin. Neurobiol. 14, 139–147 (2004).

    CAS  PubMed  Google Scholar 

  107. 107

    Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    CAS  PubMed  Google Scholar 

  108. 108

    Yacubian, J. et al. Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. J. Neurosci. 26, 9530–9537 (2006).

    CAS  PubMed  Google Scholar 

  109. 109

    Tanaka, S. C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neurosci. 7, 887–893 (2004).

    CAS  PubMed  Google Scholar 

  110. 110

    Pagnoni, G., Zink, C. F., Montague, P. R. & Berns, G. S. Activity in human ventral striatum locked to errors of reward prediction. Nature Neurosci. 5, 97–98 (2002).

    CAS  PubMed  Google Scholar 

  111. 111

    O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H. & Dolan, R. J. Temporal difference models and reward-related learning in the human brain. Neuron 38, 329–337 (2003).

    CAS  PubMed  Google Scholar 

  112. 112

    Knutson, B., Westdorp, A., Kaiser, E. & Hommer, D. fMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12, 20–27 (2000).

    CAS  PubMed  Google Scholar 

  113. 113

    Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C. & Fiez, J. A. Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol. 84, 3072–3077 (2000).

    CAS  PubMed  Google Scholar 

  114. 114

    Bayer, H. M. & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

    CAS  PubMed  PubMed Central  Google Scholar 

  115. 115

    Bayer, H. M., Lau, B. & Glimcher, P. W. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 98, 1428–1439 (2007).

    PubMed  Google Scholar 

  116. 116

    Seymour, B., Daw, N., Dayan, P., Singer, T. & Dolan, R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 27, 4826–4831 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  117. 117

    Daw, N. D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

    PubMed  Google Scholar 

  118. 118

    Lohrenz, T., McCabe, K., Camerer, C. F. & Montague, P. R. Neural signature of fictive learning signals in a sequential investment task. Proc. Natl Acad. Sci. USA 104, 9493–9498 (2007).

    CAS  PubMed  Google Scholar 

  119. 119

    Camerer, C. F. & Chong, J. K. Self-tuning experience weighted attraction learning in games. J. Econ. Theory 133, 177–198 (2007).

    Google Scholar 

  120. 120

    Olsson, A. & Phelps, E. A. Social learning of fear. Nature Neurosci. 10, 1095–1102 (2007).

    CAS  PubMed  Google Scholar 

  121. 121

    Montague, P. R. et al. Dynamic gain control of dopamine delivery in freely moving animals. J. Neurosci. 24, 1754–1759 (2004).

    CAS  PubMed  Google Scholar 

  122. 122

    Tversky, A. & Kahneman, D. Advances in prospect theory cumulative representation of uncertainty. J. Risk Uncertain. 5, 297–323 (1992).

    Google Scholar 

  123. 123

    Kahneman, D. & Tversky, A. Prospect Theory: an analysis of decision under risk. Econometrica 4, 263–291 (1979). This seminal paper proposed the PT model for goal-directed valuation in the presence of risk and provided some supporting evidence. It is one of the most cited papers in economics.

    Google Scholar 

  124. 124

    Chen, K., Lakshminarayanan, V. & Santos, L. How basic are behavioral biases? Evidence from capuchin-monkey trading behavior. J. Polit. Econ. 114, 517–537 (2006).

    Google Scholar 

  125. 125

    Camerer, C. F. in Choice, Values, and Frames (eds Kahneman, D. & Tversky, A.) (Cambridge Univ. Press, Cambridge, 2000).

    Google Scholar 

  126. 126

    Gilboa, I. & Schmeidler, D. Maxmin expected utility with non-unique prior. J. Math. Econ. 28, 141–153 (1989).

    Google Scholar 

  127. 127

    Ghirardato, P., Maccheroni, F. & Marinacci, M. Differentiating ambiguity and ambiguity attitude. J. Econ. Theory 118, 133–173 (2004).

    Google Scholar 

  128. 128

    Nestler, E. J. & Charney, D. S. The Neurobiology of Mental Illness (Oxford Univ. Press, Oxford, 2004).

    Google Scholar 

  129. 129

    Kauer, J. A. & Malenka, R. C. Synaptic plasticity and addiction. Nature Rev. Neurosci. 8, 844–858 (2007).

    CAS  Google Scholar 

  130. 130

    Hyman, S. E., Malenka, R. C. & Nestler, E. J. Neural mechanisms of addiction: the role of reward-related learning and memory. Annu. Rev. Neurosci. 29, 565–598 (2006).

    CAS  PubMed  Google Scholar 

  131. 131

    Redish, A. D. & Johnson, A. A computational model of craving and obsession. Ann. NY Acad. Sci. 1104, 324–339 (2007).

    PubMed  Google Scholar 

  132. 132

    Redish, A. D. Addiction as a computational process gone awry. Science 306, 1944–1947 (2004). This paper showed how addiction can be conceptualized as a disease of the habit valuation system, using a simple modification of the reinforcement-learning model.

    CAS  PubMed  Google Scholar 

  133. 133

    Paulus, M. P. Decision-making dysfunctions in psychiatry—altered homeostatic processing? Science 318, 602–606 (2007).

    CAS  PubMed  Google Scholar 

  134. 134

    Miller, E. K. & Cohen, J. D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202 (2001).

    CAS  PubMed  Google Scholar 

  135. 135

    Hazy, T. E., Frank, M. J. & O'Reilly, R. C. Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 1601–1613 (2007).

    PubMed  PubMed Central  Google Scholar 

  136. 136

    Niv, Y., Joel, D. & Dayan, P. A normative perspective on motivation. Trends Cogn. Sci. 10, 375–381 (2006).

    PubMed  Google Scholar 

Download references

Acknowledgements

Financial support from the National Science Foundation (SES-0134618, A.R.) and the Human Frontier Science Program (C.F.C.) is gratefully acknowledged. This work was also supported by a grant from the Gordon and Betty Moore Foundation to the Caltech Brain Imaging Center (A.R., C.F.C.). R.M. acknowledges support from the National Institute on Drug Abuse, the National Institute of Neurological Disorders and Stroke, The Kane Family Foundation, The Angel Williamson Imaging Center and The Dana Foundation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Antonio Rangel.

Related links

Related links

FURTHER INFORMATION

Author's homepage

Glossary

Valence

The appetitive or aversive nature of a stimulus.

Propositional logic system

A cognitive system that makes predictions about the world on the basis of known pieces of information.

Statistical moments

Properties of a distribution, such as mean and variance.

Expected-utility theory

A theory that states that the value of a prospect (or of random rewards) equals the sum of the value of the potential outcomes weighted by their probability.

Prospect theory

An alternative to the expected utility theory that also pertains to how to evaluate prospects.

Dual-process models

A class of psychological models in which two processes with different properties compete to determine the outcome of a computation.

Race-to-barrier diffusion process

A stochastic process that terminates when the variable of interest reaches a certain threshold value.

Credit-assignment problem

The problem of crediting rewards to particular actions in complex environments.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Rangel, A., Camerer, C. & Montague, P. A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci 9, 545–556 (2008). https://doi.org/10.1038/nrn2357

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing