Article | Published:

Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex

Nature Neuroscience volume 14, pages 15901597 (2011) | Download Citation

Abstract

The orbitofrontal cortex has been hypothesized to carry information regarding the value of expected rewards. Such information is essential for associative learning, which relies on comparisons between expected and obtained reward for generating instructive error signals. These error signals are thought to be conveyed by dopamine neurons. To test whether orbitofrontal cortex contributes to these error signals, we recorded from dopamine neurons in orbitofrontal-lesioned rats performing a reward learning task. Lesions caused marked changes in dopaminergic error signaling. However, the effect of lesions was not consistent with a simple loss of information regarding expected value. Instead, without orbitofrontal input, dopaminergic error signals failed to reflect internal information about the impending response that distinguished externally similar states leading to differently valued future rewards. These results are consistent with current conceptualizations of orbitofrontal cortex as supporting model-based behavior and suggest an unexpected role for this information in dopaminergic error signaling.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    & Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).

  2. 2.

    , , & Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).

  3. 3.

    & Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

  4. 4.

    & Reinforcement Learning: An Introduction. (MIT Press, 1998).

  5. 5.

    & A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532–552 (1980).

  6. 6.

    & in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton-Century-Crofts, 1972).

  7. 7.

    et al. Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task. J. Neurosci. 23, 11078–11084 (2003).

  8. 8.

    , & Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J. Neurosci. 24, 7540–7548 (2004).

  9. 9.

    , , & Neural responses during anticipation of a primary taste reward. Neuron 33, 815–826 (2002).

  10. 10.

    , & Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 301, 1104–1107 (2003).

  11. 11.

    & Neurons in orbitofrontal cortex encode economic value. Nature 441, 223–226 (2006).

  12. 12.

    , & Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51, 509–520 (2006).

  13. 13.

    et al. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62, 269–280 (2009).

  14. 14.

    , & Pyramidal neurons in rat prefrontal cortex projecting to ventral tegmental area and dorsal raphe nucleus express 5–HT2A receptors. Cereb. Cortex 19, 1678–1686 (2009).

  15. 15.

    , & Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615–1624 (2007).

  16. 16.

    , , & The ventral tegmental area revisited: is there an electrophysiological marker for dopaminergic neurons? J. Physiol. (Lond.) 577, 907–924 (2006).

  17. 17.

    & Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature 466, 457–462 (2010).

  18. 18.

    & The control of firing pattern in nigral dopamine neurons: burst firing. J. Neurosci. 4, 2877–2890 (1984).

  19. 19.

    , & A neural substrate for prediction and reward. Science 275, 1593–1599 (1997).

  20. 20.

    , , & A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat. Rev. Neurosci. 10, 885–892 (2009).

  21. 21.

    , , , & Distinct roles of rodent orbitofrontal and medial prefrontal cortex in decision making. Neuron 66, 449–460 (2010).

  22. 22.

    & Dissociable contributions of the orbitofrontal and infralimbic cortex to pavlovian autoshaping and discrimination reversal learning: further evidence for the functional heterogeneity of the rodent frontal cortex. J. Neurosci. 23, 8771–8780 (2003).

  23. 23.

    & Ventromedial frontal cortex mediates affective shifting in humans: evidence from a reversal learning paradigm. Brain 126, 1830–1837 (2003).

  24. 24.

    & Learning latent structure: carving nature at its joints. Curr. Opin. Neurobiol. 20, 251–256 (2010).

  25. 25.

    , , & Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol. Rev. 114, 784–805 (2007).

  26. 26.

    , & Time, context and extinction. Psychol. Rev. 117, 197–209 (2010).

  27. 27.

    & Neural correlates of olfactory recognition memory in the rat orbitofrontal cortex. J. Neurosci. 20, 8199–8208 (2000).

  28. 28.

    , & Population coding of reward magnitude in the orbitofrontal cortex of the rat. J. Neurosci. 28, 8590–8603 (2008).

  29. 29.

    et al. Single-cell and population coding of expected reward probability in the orbitofrontal cortex of the rat. J. Neurosci. 29, 8965–8976 (2009).

  30. 30.

    , , , & Separable learning systems in the macaque brain and the role of the orbitofrontal cortex in contingent learning. Neuron 65, 927–939 (2010).

  31. 31.

    , & Beyond reversal: a critical role for human orbitofrontal cortex in flexible learning from probabilistic feedback. J. Neurosci. 30, 16868–16875 (2010).

  32. 32.

    , & Monkey orbitofrontal cortex encodes response choices near feedback time. J. Neurosci. 29, 2569–2574 (2009).

  33. 33.

    , , , & Representation of spatial goals in rat orbitofrontal cortex. Neuron 51, 495–507 (2006).

  34. 34.

    , & Rat orbitofrontal cortex separately encodes response and outcome information during performance of goal-directed behavior. J. Neurosci. 28, 5127–5138 (2008).

  35. 35.

    & Distributed coding of actual and hypothetical outcomes in the orbital and dorsolateral prefrontal cortex. Neuron 70, 731–741 (2011).

  36. 36.

    & Dynamic coding of goal-directed paths by orbital prefrontal cortex. J. Neurosci. 31, 5989–6000 (2011).

  37. 37.

    et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

  38. 38.

    , , Groenewegen, H.J., Robbins, T.W. & Pennartz, C.M.A. Putting a spin on the dorsal-ventral divide of the striatum. Trends Neurosci. 27, 468–474 (2004).

  39. 39.

    The medial prefrontal and orbitofrontal cortices differentially regulate dopamine system function. Neuropsychopharmacology 36, 1227–1236 (2011).

  40. 40.

    , & Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

  41. 41.

    , , & The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 454, 340–344 (2008).

  42. 42.

    & Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental learning. J. Neurosci. 27, 4819–4825 (2007).

  43. 43.

    , , , & Model-based influences on humans' choices and striatal prediction errors. Neuron (in the press).

  44. 44.

    & Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci. (in the press).

  45. 45.

    , , & A pallidus-habenula-dopamine pathway signals inferred stimulus values. J. Neurophysiol. 104, 1068–1076 (2010).

  46. 46.

    , , , & Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).

  47. 47.

    , & the role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).

  48. 48.

    , , & Prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).

  49. 49.

    , , , & Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 31, 2700–2705 (2011).

  50. 50.

    , & Choice values. Nat. Neurosci. 9, 987–988 (2006).

Download references

Acknowledgements

This work was supported by grants from the US National Institute on Drug Abuse to G.S. and M.R. and from the US National Institute on Mental Health to Y.K.T., a Sloan Research Fellowship to Y.N. and a Binational United States-Israel Science Foundation grant to Y.N. and R.C.W.

Author information

Author notes

    • Yael Niv
    •  & Geoffrey Schoenbaum

    These authors contributed equally to this work.

Affiliations

  1. Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland, USA.

    • Yuji K Takahashi
    • , Kathy Toreson
    • , Patricio O'Donnell
    •  & Geoffrey Schoenbaum
  2. Department of Psychology, University of Maryland College Park, College Park, Maryland, USA.

    • Matthew R Roesch
  3. Program in Neuroscience and Cognitive Science, University of Maryland College Park, College Park, Maryland, USA.

    • Matthew R Roesch
  4. Department of Psychology, Princeton University, Princeton, New Jersey, USA.

    • Robert C Wilson
    •  & Yael Niv
  5. Neuroscience Institute, Princeton University, Princeton, New Jersey, USA.

    • Robert C Wilson
    •  & Yael Niv
  6. Department of Psychiatry, University of Maryland School of Medicine, Baltimore, Maryland, USA.

    • Patricio O'Donnell
    •  & Geoffrey Schoenbaum
  7. National Institute on Drug Abuse Intramural Research Program, Baltimore, Maryland, USA.

    • Geoffrey Schoenbaum

Authors

  1. Search for Yuji K Takahashi in:

  2. Search for Matthew R Roesch in:

  3. Search for Robert C Wilson in:

  4. Search for Kathy Toreson in:

  5. Search for Patricio O'Donnell in:

  6. Search for Yael Niv in:

  7. Search for Geoffrey Schoenbaum in:

Contributions

G.S., M.R.R. and Y.K.T. conceived the initial unit recording study in awake rats; Y.K.T. and M.R.R. carried it out, and Y.K.T. and G.S. analyzed the data. Subsequently, G.S. approached P.O. and Y.N. regarding in vivo recording and computational modeling, respectively. K.T. conducted the in vivo experiments, and K.T. and P.O. analyzed the data. R.C.W. and Y.N. conceived the alternative computational models, R.C.W. carried out the modeling, and Y.N. and R.C.W. interpreted the experimental data in light of simulation results. Y.N., G.S. and Y.K.T. collaborated in writing the manuscript with assistance from the other team members.

Competing interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to Yuji K Takahashi or Geoffrey Schoenbaum.

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–4 and Supplementary Results

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn.2957

Further reading