Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Modulators of decision making

Abstract

Human and animal decisions are modulated by a variety of environmental and intrinsic contexts. Here I consider computational factors that can affect decision making and review anatomical structures and neurochemical systems that are related to contextual modulation of decision making. Expectation of a high reward can motivate a subject to go for an action despite a large cost, a decision that is influenced by dopamine in the anterior cingulate cortex. Uncertainty of action outcomes can promote risk taking and exploratory choices, in which norepinephrine and the orbitofrontal cortex appear to be involved. Predictable environments should facilitate consideration of longer-delayed rewards, which depends on serotonin in the dorsal striatum and dorsal prefrontal cortex. This article aims to sort out factors that affect the process of decision making from the viewpoint of reinforcement learning theory and to bridge between such computational needs and their neurophysiological substrates.

Your institute does not have access to this article

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Figure 1: Standard models of evaluation of amount, delay and probability of reward4.
Figure 2: A hypothetical model of realization of reinforcement learning in the cortex–basal ganglia network2.
Figure 3: Possible links between computational factors and parameters of decision making and learning, and their neurobiological substrates.

References

  1. Doya, K., Ishii, S., Pouget, A. & Rao, R. Bayesian Brain: Probabilistic Approach to Neural Coding and Learning (MIT Press, Cambridge, Massachusetts, USA, (2007).

    Google Scholar 

  2. Doya, K. Reinforcement learning: computational theory and biological mechanisms. HFSP J. 1, 30–40 (2007).

    PubMed  PubMed Central  Article  Google Scholar 

  3. Sutton, R.S. & Barto, A.G. Reinforcement Learning (MIT Press, Cambridge, Massachusetts, USA, (1998).

    Google Scholar 

  4. Ho, M.Y., Mobini, S., Chiang, T.J., Bradshaw, C.M. & Szabadi, E. Theory and method in the quantitative analysis of “impulsive choice” behaviour: implications for psychopharmacology. Psychopharmacology (Berl.) 146, 362–372 (1999).

    CAS  Article  Google Scholar 

  5. Berns, G.S., Laibson, D. & Loewenstein, G. Intertemporal choice–toward an integrative framework. Trends Cogn. Sci. 11, 482–488 (2007).

    PubMed  Article  Google Scholar 

  6. Laibson, D.I. Golden eggs and hyperbolic discounting. Q. J. Econ. 62, 443–477 (1997).

    Article  Google Scholar 

  7. von Neumann, J. & Morgenstern, O. Theory of Games and Economic Behavior (Princeton Univ. Press, Princeton, New Jersey, USA, (1944).

    Google Scholar 

  8. Kahneman, D. & Tversky, A. Prospect theory: an analysis of decision under risk. Econometrica 47, 263–291 (1979).

    Article  Google Scholar 

  9. Ishii, S., Yoshida, W. & Yoshimoto, J. Control of exploitation-exploration meta-parameter in reinforcement learning. Neural Netw. 15, 665–687 (2002).

    PubMed  Article  Google Scholar 

  10. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).

    CAS  PubMed  Article  Google Scholar 

  11. Herrnstein, R.J. Relative and absolute strength of response as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267–272 (1961).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  12. Baum, W.M. Optimization and the matching law as accounts of instrumental behavior. J. Exp. Anal. Behav. 36, 387–403 (1981).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  13. Puterman, M.L. Markov Decision Processes: Discrete Dynamic Stochastic Programming (Wiley, New York, (1994).

    Book  Google Scholar 

  14. Balleine, B.W., Delgado, M.R. & Hikosaka, O. The role of the dorsal striatum in reward and decision-making. J. Neurosci. 27, 8161–8165 (2007).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  15. Balleine, B.W. & Killcross, S. Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 29, 272–279 (2006).

    CAS  PubMed  Article  Google Scholar 

  16. Kakade, S. & Dayan, P. Acquisition and extinction in autoshaping. Psychol. Rev. 109, 533–544 (2002).

    PubMed  Article  Google Scholar 

  17. Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    CAS  PubMed  Article  Google Scholar 

  18. Niv, Y., Daw, N.D., Joel, D. & Dayan, P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 191, 507–520 (2007).

    CAS  Article  Google Scholar 

  19. Schweighofer, N. et al. Humans can adopt optimal discounting strategy under real-time constraints. PLOS Comput. Biol. 2, e152 (2006).

    Article  Google Scholar 

  20. Doya, K. Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732–739 (2000).

    CAS  PubMed  Article  Google Scholar 

  21. Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).

    CAS  PubMed  Article  Google Scholar 

  22. Platt, M.L. & Glimcher, P.W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).

    CAS  PubMed  Article  Google Scholar 

  23. Schultz, W., Tremblay, L. & Hollerman, J.R. Reward processing in primate orbitofrontal cortex and basal ganglia. Cereb. Cortex 10, 272–284 (2000).

    CAS  PubMed  Article  Google Scholar 

  24. Watanabe, M. Reward expectancy in primate prefrontal neurons. Nature 382, 629–632 (1996).

    CAS  PubMed  Article  Google Scholar 

  25. Kawagoe, R., Takikawa, Y. & Hikosaka, O. Expectation of reward modulates cognitive signals in the basal ganglia. Nat. Neurosci. 1, 411–416 (1998).

    CAS  PubMed  Article  Google Scholar 

  26. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

    CAS  PubMed  Article  Google Scholar 

  27. Pasquereau, B. et al. Shaping of motor responses by incentive values through the basal ganglia. J. Neurosci. 27, 1176–1183 (2007).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  28. Komura, Y., Tamura, R., Uwano, T., Nishijo, H. & Ono, T. Auditory thalamus integrates visual inputs into behavioral gains. Nat. Neurosci. 8, 1203–1209 (2005).

    CAS  PubMed  Article  Google Scholar 

  29. Minamimoto, T., Hori, Y. & Kimura, M. Complementary process to response bias in the centromedian nucleus of the thalamus. Science 308, 1798–1801 (2005).

    CAS  PubMed  Article  Google Scholar 

  30. Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  31. Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    CAS  PubMed  Article  Google Scholar 

  32. McClure, S.M., Berns, G.S. & Montague, P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339–346 (2003).

  33. O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

    CAS  PubMed  Article  Google Scholar 

  34. O'Doherty, J.P., Dayan, P., Friston, K., Critchley, H. & Dolan, R.J. Temporal difference models and reward-related learning in the human brain. Neuron 38, 329–337 (2003).

    CAS  PubMed  Article  Google Scholar 

  35. Seymour, B. et al. Temporal difference models describe higher-order learning in humans. Nature 429, 664–667 (2004).

    CAS  PubMed  Article  Google Scholar 

  36. Reynolds, J.N. & Wickens, J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507–521 (2002).

    PubMed  Article  Google Scholar 

  37. Wickens, J.R., Begg, A.J. & Arbuthnott, G.W. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70, 1–5 (1996).

    CAS  PubMed  Article  Google Scholar 

  38. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).

    PubMed  Article  Google Scholar 

  39. Yacubian, J. et al. Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. J. Neurosci. 26, 9530–9537 (2006).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  40. Belova, M.A., Paton, J.J., Morrison, S.E. & Salzman, C.D. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55, 970–984 (2007).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  41. Seymour, B., Daw, N., Dayan, P., Singer, T. & Dolan, R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 27, 4826–4831 (2007).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  42. Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  43. Daw, N.D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

    PubMed  Article  Google Scholar 

  44. Matsumoto, M. & Hikosaka, O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111–1115 (2007).

    CAS  PubMed  Article  Google Scholar 

  45. Denk, F. et al. Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort. Psychopharmacology (Berl.) 179, 587–596 (2005).

    CAS  Article  Google Scholar 

  46. Kuhnen, C.M. & Knutson, B. The neural basis of financial risk taking. Neuron 47, 763–770 (2005).

    CAS  PubMed  Article  Google Scholar 

  47. Tobler, P.N., O'Doherty, J.P., Dolan, R.J. & Schultz, W. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 97, 1621–1632 (2007).

    PubMed  Article  Google Scholar 

  48. Cardinal, R.N. Neural systems implicated in delayed and probabilistic reinforcement. Neural Netw. 19, 1277–1301 (2006).

    PubMed  Article  Google Scholar 

  49. Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).

    CAS  PubMed  Article  Google Scholar 

  50. Tanaka, S.C. et al. Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw. 19, 1233–1241 (2006).

    PubMed  Article  Google Scholar 

  51. McClure, S.M., Laibson, D.I., Loewenstein, G. & Cohen, J.D. Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507 (2004).

    CAS  PubMed  Article  Google Scholar 

  52. McClure, S.M., Ericson, K.M., Laibson, D.I., Loewenstein, G. & Cohen, J.D. Time discounting for primary rewards. J. Neurosci. 27, 5796–5804 (2007).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  53. Kable, J.W. & Glimcher, P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633 (2007).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  54. Rudebeck, P.H., Walton, M.E., Smyth, A.N., Bannerman, D.M. & Rushworth, M.F. Separate neural pathways process different decision costs. Nat. Neurosci. 9, 1161–1168 (2006).

    CAS  PubMed  Article  Google Scholar 

  55. Winstanley, C.A., Theobald, D.E., Dalley, J.W., Cardinal, R.N. & Robbins, T.W. Double dissociation between serotonergic and dopaminergic modulation of medial prefrontal and orbitofrontal cortex during a test of impulsive choice. Cereb. Cortex 16, 106–114 (2006).

    PubMed  Article  Google Scholar 

  56. Tanaka, S.C. et al. Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE 2, e1333 (2007).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  57. Behrens, T.E., Woolrich, M.W., Walton, M.E. & Rushworth, M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).

    CAS  PubMed  Article  Google Scholar 

  58. Bouret, S. & Sara, S.J. Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci. 28, 574–582 (2005).

    CAS  PubMed  Article  Google Scholar 

  59. Dayan, P. & Yu, A.J. Phasic norepinephrine: A neural interrupt signal for unexpected events. Network 17, 335–350 (2006).

    PubMed  Article  Google Scholar 

  60. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    CAS  PubMed  Article  Google Scholar 

  61. Clarke, H.F., Dalley, J.W., Crofts, H.S., Robbins, T.W. & Roberts, A.C. Cognitive inflexibility after prefrontal serotonin depletion. Science 304, 878–880 (2004).

    CAS  PubMed  Article  Google Scholar 

  62. Floresco, S.B., Tse, M.T. & Ghods-Sharifi, S. Dopaminergic and glutamatergic regulation of effort- and delay-based decision making. Neuropsychopharmacology, published online 5 September 2007 (doi:10.1038/sj.npp.1301565).

    PubMed  Article  CAS  Google Scholar 

  63. Walton, M.E., Bannerman, D.M. & Rushworth, M.F. The role of rat medial frontal cortex in effort-based decision making. J. Neurosci. 22, 10996–11003 (2002).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  64. Rogers, R.D., Lancaster, M., Wakeley, J. & Bhagwagar, Z. Effects of beta-adrenoceptor blockade on components of human decision-making. Psychopharmacology (Berl.) 172, 157–164 (2004).

    CAS  Article  Google Scholar 

  65. Cardinal, R.N. & Howes, N.J. Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats. BMC Neurosci. 6, 37 (2005).

  66. Winstanley, C.A., Dalley, J.W., Theobald, D.E. & Robbins, T.W. Fractionating impulsivity: contrasting effects of central 5-HT depletion on different measures of impulsive behavior. Neuropsychopharmacology 29, 1331–1343 (2004).

    CAS  Article  PubMed  Google Scholar 

  67. van Gaalen, M.M., van Koten, R., Schoffelmeer, A.N. & Vanderschuren, L.J. Critical involvement of dopaminergic neurotransmission in impulsive decision making. Biol. Psychiatry 60, 66–73 (2006).

    CAS  PubMed  Article  Google Scholar 

  68. Kheramin, S. et al. Effects of orbital prefrontal cortex dopamine depletion on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl.) 175, 206–214 (2004).

    CAS  Article  Google Scholar 

  69. Robinson, E.S. et al. Similar effects of the selective noradrenaline reuptake inhibitor atomoxetine on three distinct forms of impulsivity in the rat. Neuropsychopharmacology, published online 18 July 2007 (doi:10.1038/sj.npp.1301487).

    PubMed  Article  CAS  Google Scholar 

  70. Rudebeck, P.H., Buckley, M.J., Walton, M.E. & Rushworth, M.F. A role for the macaque anterior cingulate gyrus in social valuation. Science 313, 1310–1312 (2006).

    CAS  PubMed  Article  Google Scholar 

  71. Mobini, S. et al. Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement. Psychopharmacology (Berl.) 160, 290–298 (2002).

    CAS  Article  Google Scholar 

  72. Kheramin, S. et al. Effects of quinolinic acid-induced lesions of the orbital prefrontal cortex on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl.) 165, 9–17 (2002).

    CAS  Article  Google Scholar 

  73. Kheramin, S. et al. The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: implications for models of inter-temporal choice. Behav. Brain Res. 156, 145–152 (2005).

    CAS  PubMed  Article  Google Scholar 

  74. Winstanley, C.A., Theobald, D.E., Cardinal, R.N. & Robbins, T.W. Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J. Neurosci. 24, 4718–4722 (2004).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  75. Cardinal, R.N., Pennicott, D.R., Sugathapala, C.L., Robbins, T.W. & Everitt, B.J. Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292, 2499–2501 (2001).

    CAS  Article  PubMed  Google Scholar 

  76. Pothuizen, H.H., Jongen-Relo, A.L., Feldon, J. & Yee, B.K. Double dissociation of the effects of selective nucleus accumbens core and shell lesions on impulsive-choice behaviour and salience learning in rats. Eur. J. Neurosci. 22, 2605–2616 (2005).

    PubMed  Article  Google Scholar 

  77. Hariri, A.R. et al. Preference for immediate over delayed rewards is associated with magnitude of ventral striatal activity. J. Neurosci. 26, 13213–13217 (2006).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  78. Kennerley, S.W., Walton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).

    CAS  PubMed  Article  Google Scholar 

  79. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation, and attention. Neuron 46, 681–692 (2005).

    CAS  PubMed  Article  Google Scholar 

  80. Rogers, R.D. et al. Tryptophan depletion impairs stimulus-reward learning while methylphenidate disrupts attentional control in healthy young adults: implications for the monoaminergic basis of impulsive behaviour. Psychopharmacology (Berl.) 146, 482–491 (1999).

    CAS  Article  Google Scholar 

  81. Rogers, R.D. et al. Dissociable deficits in the decision-making cognition of chronic amphetamine abusers, opiate abusers, patients with focal damage to prefrontal cortex, and tryptophan-depleted normal volunteers: evidence for monoaminergic mechanisms. Neuropsychopharmacology 20, 322–339 (1999).

    CAS  PubMed  Article  Google Scholar 

  82. Chamberlain, S.R. et al. Neurochemical modulation of response inhibition and probabilistic learning in humans. Science 311, 861–863 (2006).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  83. Clarke, H.F., Walker, S.C., Dalley, J.W., Robbins, T.W. & Roberts, A.C. Cognitive inflexibility after prefrontal serotonin depletion is behaviorally and neurochemically specific. Cereb. Cortex 17, 18–27 (2007).

    CAS  Article  Google Scholar 

  84. van der Plasse, G. et al. Medial prefrontal serotonin in the rat is involved in goal-directed behaviour when affect guides decision making. Psychopharmacology (Berl.) 195, 435–449 (2007).

    Article  CAS  Google Scholar 

  85. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  86. Corbit, L.H. & Balleine, B.W. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 146, 145–157 (2003).

    PubMed  Article  Google Scholar 

  87. Balleine, B.W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 86, 717–730 (2005).

    CAS  PubMed  Article  Google Scholar 

  88. Rachlin, H., Raineri, A. & Cross, D. Subjective probability and delay. J. Exp. Anal. Behav. 55, 233–244 (1991).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

Download references

Acknowledgements

The author thanks P. Dayan, N. Daw and N. Schweighofer for discussions on temporal discounting. This research was supported by a Grant-in-Aid for Scientific Research on Priority Areas, Ministry of Education, Culture, Sports, Science and Technology of Japan.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kenji Doya.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Doya, K. Modulators of decision making. Nat Neurosci 11, 410–416 (2008). https://doi.org/10.1038/nn2077

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn2077

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing