Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Learning the value of information in an uncertain world

Abstract

Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Figure 1: Probability-tracking task.
Figure 2: Behavior of Bayesian learner and human subjects.
Figure 3: Experiment II, cingulate activity reflecting estimated volatility.
Figure 4: Region-of-interest analysis and potential confounding factors.
Figure 5: Estimated volatility and variance on r.
Figure 6: VTA correlate of reward prediction.

References

  1. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    CAS  Article  PubMed  Google Scholar 

  2. Kording, K.P. & Wolpert, D.M. Bayesian integration in sensorimotor learning. Nature 427, 244–247 (2004).

    PubMed  Article  Google Scholar 

  3. Kahneman, D. & Tversky, A. Choices, Values and Frames (Cambridge University Press, Cambridge, 2000).

    Book  Google Scholar 

  4. Montague, P.R., Dayan, P., Person, C. & Sejnowski, T.J. Bee foraging in uncertain environments using predictive hebbian learning. Nature 377, 725–728 (1995).

    CAS  PubMed  Article  Google Scholar 

  5. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

    CAS  PubMed  Article  Google Scholar 

  6. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  7. Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  8. Rescorla, R.A. & Wagner, A.R. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton-Century Crofts, New York, 1972).

    Google Scholar 

  9. Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  10. Dayan, P., Kakade, S. & Montague, P.R. Learning and selective attention. Nat. Neurosci. 3 Suppl, 1218–1223 (2000).

    CAS  PubMed  Article  Google Scholar 

  11. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).

    PubMed  Article  Google Scholar 

  12. Pearce, J.M. & Hall, G. A model for Pavlovian learning: variations in the effectiveness of conditioned, but not of unconditioned, stimuli. Psychol. Rev. 87, 532–552 (1980).

    CAS  PubMed  Article  Google Scholar 

  13. Dickinson, A. & Mackintosh, N.J. Classical conditioning in animals. Annu. Rev. Psychol. 29, 587–612 (1978).

    CAS  PubMed  Article  Google Scholar 

  14. Cox, R.T. Probability, frequency and reasonable expectaion. Am. J. Phys. 14, 1–13 (1946).

    Article  Google Scholar 

  15. Kakade, S. & Dayan, P. Acquisition and extinction in autoshaping. Psychol. Rev. 109, 533–544 (2002).

    PubMed  Article  Google Scholar 

  16. Courville, A.C., Daw, N.D. & Touretzky, D.S. Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294–300 (2006).

    PubMed  Article  Google Scholar 

  17. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation and attention. Neuron 46, 681–692 (2005).

    CAS  Article  PubMed  Google Scholar 

  18. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).

    CAS  PubMed  Article  Google Scholar 

  19. Kennerley, S.W., Walton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).

    CAS  PubMed  Article  Google Scholar 

  20. Gallistel, C.R., Mark, T.A., King, A.P. & Latham, P.E. The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. J. Exp. Psychol. Anim. Behav. Process. 27, 354–372 (2001).

    CAS  PubMed  Article  Google Scholar 

  21. Procyk, E., Tanaka, Y.L. & Joseph, J.P. Anterior cingulate activity during routine and nonroutine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).

    CAS  PubMed  Article  Google Scholar 

  22. Walton, M.E., Devlin, J.T. & Rushworth, M.F. Interactions between decision making and performance monitoring within prefrontal cortex. Nat. Neurosci. 7, 1259–1265 (2004).

    CAS  PubMed  Article  Google Scholar 

  23. Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171, 213–224 (1979).

    CAS  PubMed  Article  Google Scholar 

  24. Ullsperger, M. & von Cramon, D.Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  25. Brown, J.W. & Braver, T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 307, 1118–1121 (2005).

    CAS  Article  PubMed  Google Scholar 

  26. Ito, S., Stuphorn, V., Brown, J.W. & Schall, J.D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302, 120–122 (2003).

    CAS  PubMed  Article  Google Scholar 

  27. Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).

    CAS  PubMed  Article  Google Scholar 

  28. Smith, S.M. et al. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23 Suppl 1, S208–S219 (2004).

    Article  PubMed  Google Scholar 

  29. Koechlin, E., Ody, C. & Kouneiher, F. The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).

    CAS  PubMed  Article  Google Scholar 

  30. Strick, P.L., Dum, R.P. & Picard, N. Motor areas on the medial wall of the hemisphere. Novartis Found Symp. 218, 64–75; discussion 75–80, 104–8 (1998).

    CAS  PubMed  Google Scholar 

  31. Van Hoesen, G.W., Morecraft, R.J. & Vogt, B.A. in Neurobiology of Cingulate Cortex and Limbic Thalamus (eds. Vogt, B.A. & Gabriel, M.) (Birkhauser, Boston, 1993).

    Google Scholar 

  32. McCoy, A.N., Crowley, J.C., Haghighian, G., Dean, H.L. & Platt, M.L. Saccade reward signals in posterior cingulate cortex. Neuron 40, 1031–1040 (2003).

    CAS  PubMed  Article  Google Scholar 

  33. McCoy, A.N. & Platt, M.L. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220–1227 (2005).

    CAS  PubMed  Article  Google Scholar 

  34. Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).

    CAS  Article  PubMed  Google Scholar 

  35. Preuschoff, K., Bossaerts, P. & Quartz, S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381–390 (2006).

    CAS  PubMed  Article  Google Scholar 

  36. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    CAS  PubMed  Article  Google Scholar 

  37. Engle, R.F. Autoregressive conditional Heteroscedasticity with estimates of the variance of UK inflation. Econometrica 50, 987–1008 (1982).

    Article  Google Scholar 

  38. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).

    CAS  Article  PubMed  Google Scholar 

  39. O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

    CAS  PubMed  Article  Google Scholar 

  40. Haruno, M. et al. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci. 24, 1660–1665 (2004).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  41. Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).

    CAS  PubMed  Article  Google Scholar 

  42. Kunishio, K. & Haber, S.N. Primate cingulostriatal projection: limbic striatal versus sensorimotor striatal input. J. Comp. Neurol. 350, 337–356 (1994).

    CAS  PubMed  Article  Google Scholar 

  43. Amiez, C., Joseph, J.P. & Procyk, E. Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex 16, 1040–1055 (2006).

    CAS  PubMed  Article  Google Scholar 

  44. Yoshida, W. & Ishii, S. Resolution of uncertainty in prefrontal cortex. Neuron 50, 781–789 (2006).

    CAS  PubMed  Article  Google Scholar 

  45. Fitzgerald, K.D. et al. Error-related hyperactivity of the anterior cingulate cortex in obsessive-compulsive disorder. Biol. Psychiatry 57, 287–294 (2005).

    PubMed  Article  Google Scholar 

  46. Critchley, H.D., Mathias, C.J. & Dolan, R.J. Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron 29, 537–545 (2001).

    CAS  Article  PubMed  Google Scholar 

  47. Botvinick, M.M., Cohen, J.D. & Carter, C.S. Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–546 (2004).

    PubMed  Article  Google Scholar 

  48. Rushworth, M.F., Buckley, M.J., Behrens, T.E., Walton, M.E. & Bannerman, D.M. Functional organization of the medial frontal cortex. Curr. Opin. Neurobiol. 17, 220–227 (2007).

    CAS  PubMed  Article  Google Scholar 

  49. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  50. Preuschoff, K. & Bossaerts, P. Adding prediction risk to the theory of reward learning. Ann. N Y Acad. Sci. 1104, 135–146 (2007).

    PubMed  Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank K. Watkins for advice with the study and the manuscript. This work was supported by the UK Medical Research Council (T.B.), the Engineering and Physical Sciences Research Council (M.W.W.), the Wellcome trust (M.E.W.) and the Royal Society (M.F.S.R.).

Author information

Authors and Affiliations

Authors

Contributions

All four authors were involved in generating the hypothesis, designing the experiment and writing the manuscript. Where specific roles can be assigned: T.E.J.B. and M.W.W. built the model. T.E.J.B. acquired and analyzed the data. M.E.W. supplied the necessary incisive wit. M.F.S.R. supervised the project.

Corresponding author

Correspondence to Timothy E J Behrens.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1 and 2, Table 1, Supplementary Information (PDF 1986 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Behrens, T., Woolrich, M., Walton, M. et al. Learning the value of information in an uncertain world. Nat Neurosci 10, 1214–1221 (2007). https://doi.org/10.1038/nn1954

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn1954

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing