Abstract
Reinforcement learning models that focus on the striatum and dopamine can predict the choices of animals and people. Representations of reward expectation and of reward prediction errors that are pertinent to decision making, however, are not confined to these regions but are also found in prefrontal and cingulate cortex. Moreover, decisions are not guided solely by the magnitude of the reward that is expected. Uncertainty in the estimate of the reward expectation, the value of information that might be gained by taking a course of action and the cost of an action all influence the manner in which decisions are made through prefrontal and cingulate cortex.
This is a preview of subscription content, access via your institution
Relevant articles
Open Access articles citing this article.
-
The effect of prediction error on episodic memory encoding is modulated by the outcome of the predictions
npj Science of Learning Open Access 29 May 2023
-
Neuro-computational mechanisms and individual biases in action-outcome learning under moral conflict
Nature Communications Open Access 06 March 2023
-
Regulation of social hierarchy learning by serotonin transporter availability
Neuropsychopharmacology Open Access 09 August 2022
Access options
Subscribe to this journal
Receive 12 print issues and online access
$189.00 per year
only $15.75 per issue
Rent or buy this article
Get just this article for as long as you need it
$39.95
Prices may be subject to local taxes which are calculated during checkout







References
Rescorla, R. & Wagner, A. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton-Century-Crofts, New York, 1972).
Sutton, R. & Barto, A.G. Reinforcement Learning (MIT Press, Cambridge, Massachusetts, 1998).
Dayan, P., Kakade, S. & Montague, P.R. Learning and selective attention. Nat. Neurosci. 3 (suppl.): 1218–1223 (2000).
Courville, A.C., Daw, N.D. & Touretzky, D.S. Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294–300 (2006).
Behrens, T.E., Woolrich, M.W., Walton, M.E. & Rushworth, M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).
Schultz, W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115 (2006).
Roesch, M.R., Calu, D.J. & Schoenbaum, G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615–1624 (2007).
Tobler, P.N., Fiorillo, C.D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).
Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).
Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).
O'Doherty, J.P., Deichmann, R., Critchley, H.D. & Dolan, R.J. Neural responses during anticipation of a primary taste reward. Neuron 33, 815–826 (2002).
Haruno, M. et al. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci. 24, 1660–1665 (2004).
Haruno, M. & Kawato, M. Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J. Neurophysiol. 95, 948–959 (2006).
O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).
Haruno, M. & Kawato, M. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw. 19, 1242–1254 (2006).
Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).
Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
Cohen, M.X. & Ranganath, C. Reinforcement learning signals predict future decisions. J. Neurosci. 27, 371–378 (2007).
Williams, S.M. & Goldman-Rakic, P.S. Widespread origin of the primate mesofrontal dopamine system. Cereb. Cortex 8, 321–345 (1998).
Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).
Amiez, C., Joseph, J.P. & Procyk, E. Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex 16, 1040–1055 (2006).
Seo, H. & Lee, D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 27, 8366–8377 (2007).
Amiez, C., Joseph, J.P. & Procyk, E. Anterior cingulate error-related activity is modulated by predicted reward. Eur. J. Neurosci. 21, 3447–3452 (2005).
Matsumoto, M., Matsumoto, K., Abe, H. & Tanaka, K. Medial prefrontal cell activity signaling prediction errors of action values. Nat. Neurosci. 10, 647–656 (2007).
Mars, R.B. et al. Neural dynamics of error processing in medial frontal cortex. Neuroimage 28, 1007–1013 (2005).
Ullsperger, M., Nittono, H. & von Cramon, D.Y. When goals are missed: dealing with self-generated and externally induced failure. Neuroimage 35, 1356–1364 (2007).
Klein, T.A. et al. Genetically determined differences in learning from errors. Science 318, 1642–1645 (2007).
Walton, M.E., Devlin, J.T. & Rushworth, M.F.S. Interactions between decision making and performance monitoring within prefrontal cortex. Nat. Neurosci. 7, 1259–1265 (2004).
Debener, S. et al. Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J. Neurosci. 25, 11730–11737 (2005).
Holroyd, C.B. & Coles, M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709 (2002).
Oliveira, F.T., McDonald, J.J. & Goodman, D. Performance monitoring in the anterior cingulate is not all error related: expectancy deviation and the representation of action-outcome associations. J. Cogn. Neurosci. 19, 1994–2004 (2007).
Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).
Tobler, P.N., Dickinson, A. & Schultz, W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci. 23, 10402–10410 (2003).
Ljunberg, T., Apicella, P. & Schultz, W. Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67, 145–163 (1992).
Dum, R.P. & Strick, P.L. Spinal cord terminations of the medial wall motor areas in macaque monkeys. J. Neurosci. 16, 6513–6525 (1996).
Van Hoesen, G.W., Morecraft, R.J. & Vogt, B.A. Connections of the monkey cingulate cortex. in Neurobiology of Cingulate Cortex and Limbic Thalamus (eds. Vogt, B.A. & Gabriel, M.) 249–284 (Birkhauser, Boston, 1993).
Akkal, D., Bioulac, B., Audin, J. & Burbaud, P. Comparison of neuronal activity in the rostral supplementary and cingulate motor areas during a task with cognitive and motor demands. Eur. J. Neurosci. 15, 887–904 (2002).
Hoshi, E., Sawamura, H. & Tanji, J. Neurons in the rostral cingulate motor area monitor multiple phases of visuomotor behavior with modest parametric selectivity. J. Neurophysiol. 94, 640–656 (2005).
Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).
Kennerley, S.W., Walton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).
Corrado, G.S., Sugrue, L.P., Seung, H.S. & Newsome, W.T. Linear-nonlinear-Poisson models of primate choice dynamics. J. Exp. Anal. Behav. 84, 581–617 (2005).
Yoshida, W. & Ishii, S. Resolution of uncertainty in prefrontal cortex. Neuron 50, 781–789 (2006).
Deiber, M.-P. et al. Cortical areas and the selection of movement: a study with positron emission tomography. Exp. Brain Res. 84, 393–402 (1991).
Forstmann, B.U., Brass, M., Koch, I. & von Cramon, D.Y. Voluntary selection of task sets revealed by functional magnetic resonance imaging. J. Cogn. Neurosci. 18, 388–398 (2006).
Procyk, E., Tanaka, Y.L. & Joseph, J.P. Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).
Deaner, R.O., Khera, A.V. & Platt, M.L. Monkeys pay per view: adaptive valuation of social images by rhesus macaques. Curr. Biol. 15, 543–548 (2005).
Shepherd, S.V., Deaner, R.O. & Platt, M.L. Social status gates social attention in monkeys. Curr. Biol. 16, R119–R120 (2006).
Rudebeck, P.H., Buckley, M.J., Walton, M.E. & Rushworth, M.F. A role for the macaque anterior cingulate gyrus in social valuation. Science 313, 1310–1312 (2006).
Rudebeck, P.H. et al. Distinct contributions of frontal areas to emotion and social behavior in the rat. Eur. J. Neurosci. 26, 2315–2326 (2007).
Tomlin, D. et al. Agent-specific responses in the cingulate cortex during economic exchanges. Science 312, 1047–1050 (2006).
Delgado, M.R., Frank, R.H. & Phelps, E.A. Perceptions of moral character modulate the neural systems of reward during the trust game. Nat. Neurosci. 8, 1611–1618 (2005).
King-Casas, B. et al. Getting to know you: reputation and trust in a two-person economic exchange. Science 308, 78–83 (2005).
Amodio, D.M. & Frith, C.D. Meeting of minds: the medial frontal cortex and social cognition. Nat. Rev. Neurosci. 7, 268–277 (2006).
Schoenbaum, G., Chiba, A.A. & Gallagher, M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat. Neurosci. 1, 155–159 (1998).
Tremblay, L. & Schultz, W. Relative reward preference in primate orbitofrontal cortex. Nature 398, 704–708 (1999).
Rushworth, M.F., Behrens, T.E., Rudebeck, P.H. & Walton, M.E. Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behavior. Trends Cogn. Sci. 11, 168–176 (2007).
Wallis, J.D. & Miller, E.K. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur. J. Neurosci. 18, 2069–2081 (2003).
Rolls, E.T. & Baylis, L.L. Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex. J. Neurosci. 14, 5437–5452 (1994).
Rolls, E.T., Critchley, H.D., Browning, A.S., Hernadi, I. & Lenard, L. Responses to the sensory properties of fat of neurons in the primate orbitofrontal cortex. J. Neurosci. 19, 1532–1540 (1999).
Padoa-Schioppa, C. & Assad, J.A. The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nat. Neurosci. 11, 95–102 (2008).
Rolls, E.T., Sienkiewicz, Z.J. & Yaxley, S. Hunger modulates the responses to gustatory stimuli of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey. Eur. J. Neurosci. 1, 53–60 (1989).
Gallagher, M., McMahan, R.W. & Schoenbaum, G. Orbitofrontal cortex and representation of incentive value in associative learning. J. Neurosci. 19, 6610–6614 (1999).
Baxter, M.G., Parker, A., Lindner, C.C.C., Izquierdo, A.D. & Murray, E.A. Control of response selection by reinforcer value requires interaction of amygdala and orbital frontal cortex. J. Neurosci. 20, 4311–4319 (2000).
Izquierdo, A., Suda, R.K. & Murray, E.A. Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J. Neurosci. 24, 7540–7548 (2004).
Ostlund, S. & Balleine, B.W. The contribution of orbitofrontal cortex to action selection. Ann. N Y Acad. Sci. 1121, 174–192 (2007).
Padoa-Schioppa, C. & Assad, J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226 (2006).
Platt, M.L. & Glimcher, P.W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).
Knutson, B., Taylor, J., Kaufman, M., Peterson, R. & Glover, G. Distributed neural representation of expected value. J. Neurosci. 25, 4806–4812 (2005).
McClure, S.M., Laibson, D.I., Loewenstein, G. & Cohen, J.D. Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507 (2004).
Kable, J.W. & Glimcher, P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633 (2007).
Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).
Plassmann, H., O'Doherty, J. & Rangel, A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 27, 9984–9988 (2007).
Knutson, B., Rick, S., Wimmer, G.E., Prelec, D. & Loewenstein, G. Neural predictors of purchases. Neuron 53, 147–156 (2007).
Vogt, B.A., Vogt, L., Farber, N.B. & Bush, G. Architecture and neurocytology of monkey cingulate gyrus. J. Comp. Neurol. 485, 218–239 (2005).
Ongur, D., Ferry, A.T. & Price, J.L. Architectonic subdivision of the human orbital and medial prefrontal cortex. J. Comp. Neurol. 460, 425–449 (2003).
Carmichael, S.T. & Price, J.L. Connectional networks within the orbital and medial prefrontal cortex of macaque monkeys. J. Comp. Neurol. 371, 179–207 (1996).
Walton, M.E., Kennerley, S.W., Bannerman, D.M., Phillips, P.E. & Rushworth, M.F. Weighing up the benefits of work: behavioral and neural analyses of effort-related decision making. Neural Netw. 19, 1302–1314 (2006).
Roesch, M.R. & Olson, C.R. Neuronal activity in primate orbitofrontal cortex reflects the value of time. J. Neurophysiol. 94, 2457–2471 (2005).
Roesch, M.R., Taylor, A.R. & Schoenbaum, G. Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51, 509–520 (2006).
Feierstein, C.E., Quirk, M.C., Uchida, N., Sosulski, D.L. & Mainen, Z.F. Representation of spatial goals in rat orbitofrontal cortex. Neuron 51, 495–507 (2006).
Kheramin, S. et al. Effects of quinolinic acid-induced lesions of the orbital prefrontal cortex on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl.) 165, 9–17 (2002).
Mobini, S. et al. Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement. Psychopharmacology (Berl.) 160, 290–298 (2002).
Rudebeck, P.H., Walton, M.E., Smyth, A.N., Bannerman, D.M. & Rushworth, M.F. Separate neural pathways process different decision costs. Nat. Neurosci. 9, 1161–1168 (2006).
Walton, M.E., Bannerman, D.M. & Rushworth, M.F.S. The role of rat medial frontal cortex in effort-based decision making. J. Neurosci. 22, 10996–11003 (2002).
Walton, M.E., Bannerman, D.M., Alterescu, K. & Rushworth, M.F.S. Functional specialization within medial frontal cortex of the anterior cingulate for evaluating effort-related decisions. J. Neurosci. 23, 6475–6479 (2003).
Shidara, M. & Richmond, B.J. Anterior cingulate: single neuronal signals related to degree of reward expectancy. Science 296, 1709–1711 (2002).
Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).
Preuschoff, K., Bossaerts, P. & Quartz, S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381–390 (2006).
Tobler, P.N., O'Doherty, J.P., Dolan, R.J. & Schultz, W. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 97, 1621–1632 (2007).
McClure, S.M., Ericson, K.M., Laibson, D.I., Loewenstein, G. & Cohen, J.D. Time discounting for primary rewards. J. Neurosci. 27, 5796–5804 (2007).
Watanabe, M. Reward expectancy in primate prefrontal neurons. Nature 382, 629–632 (1996).
Kobayashi, S. et al. Influences of rewarding and aversive outcomes on activity in macaque lateral prefrontal cortex. Neuron 51, 861–870 (2006).
Watanabe, M. & Sakagami, M. Integration of cognitive and motivational context information in the primate prefrontal cortex. Cereb. Cortex 17 (suppl. 1), i101–i109 (2007).
Wise, S.P. & Murray, E.A. Arbitrary associations between antecedents and actions. Trends Neurosci. 23, 271–276 (2000).
Genovesio, A., Brasted, P.J. & Wise, S.P. Representation of future and previous spatial goals by separate neural populations in prefrontal cortex. J. Neurosci. 26, 7305–7316 (2006).
Mushiake, H., Saito, N., Sakamoto, K., Itoyama, Y. & Tanji, J. Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 50, 631–641 (2006).
Saito, N., Mushiake, H., Sakamoto, K., Itoyama, Y. & Tanji, J. Representation of immediate and final behavioral goals in the monkey prefrontal cortex during an instructed delay period. Cereb. Cortex 15, 1535–1546 (2005).
Averbeck, B.B., Sohn, J.W. & Lee, D. Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 9, 276–282 (2006).
Acknowledgements
Funded by the UK Medical Research Council.
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Rushworth, M., Behrens, T. Choice, uncertainty and value in prefrontal and cingulate cortex. Nat Neurosci 11, 389–397 (2008). https://doi.org/10.1038/nn2066
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn2066
This article is cited by
-
A reservoir of foraging decision variables in the mouse brain
Nature Neuroscience (2023)
-
The effect of prediction error on episodic memory encoding is modulated by the outcome of the predictions
npj Science of Learning (2023)
-
A neural substrate of sex-dependent modulation of motivation
Nature Neuroscience (2023)
-
Neuro-computational mechanisms and individual biases in action-outcome learning under moral conflict
Nature Communications (2023)
-
Uncertainty aversion predicts the neural expansion of semantic representations
Nature Human Behaviour (2023)