Optimal decision making and the anterior cingulate cortex

Abstract

Learning the value of options in an uncertain environment is central to optimal decision making. The anterior cingulate cortex (ACC) has been implicated in using reinforcement information to control behavior. Here we demonstrate that the ACC's critical role in reinforcement-guided behavior is neither in detecting nor in correcting errors, but in guiding voluntary choices based on the history of actions and outcomes. ACC lesions did not impair the performance of monkeys (Macaca mulatta) immediately after errors, but made them unable to sustain rewarded responses in a reinforcement-guided choice task and to integrate risk and payoff in a dynamic foraging task. These data suggest that the ACC is essential for learning the value of actions.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Diagram of the macaque brain and overview of experiments 1 and 2.
Figure 2: Performance in experiment 1A.
Figure 3: Performance for sustaining rewarded behavior following an error in experiment 1.
Figure 4: Estimates of the influence of previous reward history on current choice, in experiment 1.
Figure 5: Postoperative performance in the matching task in experiment 2.
Figure 6: Schematic of a sample choice trial in experiment 3.
Figure 7: Percentage of high reward options selected for each of the three choice pairs across three preoperative (unfilled bars) and one postoperative (hatched bars) sessions.
Figure 8: Coronal sections showing the cingulate lesion in all three monkeys that received surgery in experiments 1–3.

References

  1. 1

    McCoy, A.N. & Platt, M.L. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220–1227 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  2. 2

    Platt, M.L. & Glimcher, P.W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  3. 3

    Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  4. 4

    Murray, E.A., Bussey, T.J. & Wise, S.P. Role of prefrontal cortex in a network for arbitrary visuomotor mapping. Exp. Brain Res. 133, 114–129 (2000).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  5. 5

    Sutton, R.S. & Barto, A.G. Reinforcement Learning (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  6. 6

    Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  7. 7

    Schultz, W. Getting formal with dopamine and reward. Neuron 36, 241–263 (2002).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  8. 8

    Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  9. 9

    Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  10. 10

    Paus, T. Primate anterior cingulate cortex: where motor control, drive and cognition interface. Nat. Rev. Neurosci. 2, 417–424 (2001).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  11. 11

    Rushworth, M.F., Walton, M.E., Kennerley, S.W. & Bannerman, D.M. Action sets and decisions in the medial frontal cortex. Trends Cogn. Sci. 8, 410–417 (2004).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12

    Botvinick, M.M., Braver, T.S., Barch, D.M., Carter, C.S. & Cohen, J.D. Conflict monitoring and cognitive control. Psychol. Rev. 108, 624–652 (2001).

    CAS  Article  Google Scholar 

  13. 13

    Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  14. 14

    Walton, M.E., Devlin, J.T. & Rushworth, M.F. Interactions between decision making and performance monitoring within prefrontal cortex. Nat. Neurosci. 7, 1259–1265 (2004).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  15. 15

    Hadland, K.A., Rushworth, M.F., Gaffan, D. & Passingham, R.E. The anterior cingulate and reward-guided selection of actions. J. Neurophysiol. 89, 1161–1164 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  16. 16

    Morecraft, R.J. & Van Hoesen, G.W. Convergence of limbic input to the cingulate motor cortex in the rhesus monkey. Brain Res. Bull. 45, 209–232 (1998).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  17. 17

    Amiez, C., Joseph, J.P. & Procyk, E. Anterior cingulate error-related activity is modulated by predicted reward. Eur. J. Neurosci. 21, 3447–3452 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  18. 18

    Shima, K. & Tanji, J. Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282, 1335–1338 (1998).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  19. 19

    Ito, S., Stuphorn, V., Brown, J.W. & Schall, J.D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302, 120–122 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  20. 20

    Ullsperger, M. & von Cramon, D.Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  21. 21

    Gehring, W.J., Goss, B., Coles, M.G., Meyer, D.E & Donchin, E. A neural system for error detection and compensation. Psychol. Sci. 4, 385–390 (1993).

    Article  Google Scholar 

  22. 22

    Holroyd, C.B. & Coles, M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709 (2002).

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23

    Brown, J.W. & Braver, T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 307, 1118–1121 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  24. 24

    Fellows, L.K. & Farah, M.J. Is anterior cingulate cortex necessary for cognitive control? Brain 128, 788–796 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  25. 25

    Swick, D. & Turken, A.U. Dissociation between conflict detection and error monitoring in the human anterior cingulate cortex. Proc. Natl Acad. Sci. USA 99, 16354–16359 (2002).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  26. 26

    Rushworth, M.F., Hadland, K.A., Gaffan, D. & Passingham, R.E. The effect of cingulate cortex lesions on task switching and working memory. J. Cogn. Neurosci. 15, 338–353 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  27. 27

    Amiez, C., Joseph, J.P. & Procyk, E. Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex, published online 5 October 2005 (doi:10.1093/cercor/bhj046).

  28. 28

    Lau, B. & Glimcher, P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav. 84, 555–579 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  29. 29

    Lee, D., Conroy, M.L., McGreevy, B.P. & Barraclough, D.J. Reinforcement learning and decision making in monkeys during a competitive game. Brain Res. Cogn. Brain Res. 22, 45–58 (2004).

    Article  PubMed  PubMed Central  Google Scholar 

  30. 30

    Herrnstein, R.J. The Matching Law: Papers in Psychology and Economics (eds. Rachlin, H. & Laibson, D.I.) (Harvard Univ. Press, Cambridge, Massachusetts, 1997).

    Google Scholar 

  31. 31

    Bush, G. et al. Dorsal anterior cingulate cortex: a role in reward-based decision making. Proc. Natl Acad. Sci. USA 99, 523–528 (2002).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  32. 32

    Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171, 213–224 (1979).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  33. 33

    Genovesio, A., Brasted, P.J., Mitz, A.R. & Wise, S.P. Prefrontal cortex activity related to abstract response strategies. Neuron 47, 307–320 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  34. 34

    Williams, S.M. & Goldman-Rakic, P.S. Widespread origin of the primate mesofrontal dopamine system. Cereb. Cortex 8, 321–345 (1998).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  35. 35

    Murray, E.A., Davidson, M., Gaffan, D., Olton, D.S. & Suomi, S. Effects of fornix transection and cingulate cortical ablation on spatial memory in rhesus monkeys. Exp. Brain Res. 74, 173–186 (1989).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  36. 36

    Morris, R., Pandya, D.N. & Petrides, M. Fiber system linking the mid-dorsolateral frontal cortex with the retrosplenial/presubicular region in the rhesus monkey. J. Comp. Neurol. 407, 183–192 (1999).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  37. 37

    Aron, A.R., Robbins, T.W. & Poldrack, R.A. Inhibition and the right inferior frontal cortex. Trends Cogn. Sci. 8, 170–177 (2004).

    Article  PubMed  PubMed Central  Google Scholar 

  38. 38

    Roberts, A.C. & Wallis, J.D. Inhibitory control and affective processing in the prefrontal cortex: neuropsychological studies in the common marmoset. Cereb. Cortex 10, 252–262 (2000).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  39. 39

    Knutson, B., Taylor, J., Kaufman, M., Peterson, R. & Glover, G. Distributed neural representation of expected value. J. Neurosci. 25, 4806–4812 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  40. 40

    Shidara, M. & Richmond, B.J. Anterior cingulate: single neuronal signals related to degree of reward expectancy. Science 296, 1709–1711 (2002).

    Article  PubMed  PubMed Central  Google Scholar 

  41. 41

    Wallis, J.D. & Miller, E.K. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur. J. Neurosci. 18, 2069–2081 (2003).

    Article  PubMed  PubMed Central  Google Scholar 

  42. 42

    Roesch, M.R. & Olson, C.R. Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304, 307–310 (2004).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  43. 43

    Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  44. 44

    Barraclough, D.J., Conroy, M.L. & Lee, D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 7, 404–410 (2004).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  45. 45

    Nakamura, K., Roesch, M.R. & Olson, C.R. Neuronal activity in macaque SEF and ACC during performance of tasks involving conflict. J. Neurophysiol. 93, 884–908 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  46. 46

    Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  47. 47

    Yu, A.J. & Dayan, P. Uncertainty, neuromodulation, and attention. Neuron 46, 681–692 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  48. 48

    He, S.Q., Dum, R.P. & Strick, P.L. Topographic organization of corticospinal projections from the frontal lobe: motor areas on the medial surface of the hemisphere. J. Neurosci. 15, 3284–3306 (1995).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  49. 49

    Bates, J.F. & Goldman-Rakic, P.S. Prefrontal connections of medial motor areas in the rhesus monkey. J. Comp. Neurol. 336, 211–228 (1993).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  50. 50

    Paxinos, G., Huang, X.F. & Toga, A.W. The Rhesus Monkey Brain in Stereotaxic Coordinates (Academic Press, San Diego, 1999).

    Google Scholar 

Download references

Acknowledgements

We are grateful to D. Gaffan for advice and encouragement. Supported by the Medical Research Council. Additional support from the Clarendon Foundation (S.W.K.), the Wellcome Trust (M.E.W.) and the Royal Society (M.J.B and M.F.S.R.).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Steven W Kennerley.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Fig. 1

Number of trials to update to the other response after a task-imposed response switch in Experiment 1. (PDF 57 kb)

Supplementary Fig. 2

Inter-response time performance in Experiment 1A. (PDF 54 kb)

Supplementary Fig. 3

Performance for sustaining rewarded behavior following errors that occurred at a task-imposed switch trial or all other errors. (PDF 59 kb)

Supplementary Fig. 4

Relative contributions of the high and low reward-probability responses in determining ropt for the 0.4:0.1 action-reward ratio pairing in Experiment 2. (PDF 84 kb)

Supplementary Fig. 5

The average expected reward per response for each action-reward ratio pair in Experiment 2. (PDF 64 kb)

Supplementary Fig. 6

Group response ratio plot for the action-reward ratio 0.4:0.1. (PDF 79 kb)

Supplementary Methods (PDF 189 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kennerley, S., Walton, M., Behrens, T. et al. Optimal decision making and the anterior cingulate cortex. Nat Neurosci 9, 940–947 (2006). https://doi.org/10.1038/nn1724

Download citation

Further reading