Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A causal link between prediction errors, dopamine neurons and learning

Abstract

Situations in which rewards are unexpectedly obtained or withheld represent opportunities for new learning. Often, this learning includes identifying cues that predict reward availability. Unexpected rewards strongly activate midbrain dopamine neurons. This phasic signal is proposed to support learning about antecedent cues by signaling discrepancies between actual and expected outcomes, termed a reward prediction error. However, it is unknown whether dopamine neuron prediction error signaling and cue-reward learning are causally linked. To test this hypothesis, we manipulated dopamine neuron activity in rats in two behavioral procedures, associative blocking and extinction, that illustrate the essential function of prediction errors in learning. We observed that optogenetic activation of dopamine neurons concurrent with reward delivery, mimicking a prediction error, was sufficient to cause long-lasting increases in cue-elicited reward-seeking behavior. Our findings establish a causal role for temporally precise dopamine neuron signaling in cue-reward learning, bridging a critical gap between experimental evidence and influential theoretical frameworks.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Behavioral demonstration of the blocking effect.
Figure 2: Dopamine neuron stimulation drives new learning.
Figure 3: Dopamine neuron stimulation attenuates behavioral decrements associated with a downshift in reward value.
Figure 4: Dopamine neuron stimulation attenuates behavioral decrements associated with reward omission.

Similar content being viewed by others

References

  1. Rescorla, R.A. & Wagner, A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton Century Crofts, New York, 1972).

  2. Glimcher, P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. USA 108 (suppl. 3): 15647–15654 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  PubMed  Google Scholar 

  5. Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473–500 (2000).

    Article  CAS  PubMed  Google Scholar 

  6. Sutton, R.S. & Barto, A.G. Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170 (1981).

    Article  CAS  PubMed  Google Scholar 

  7. Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B. & Uchida, N. Neuron type–specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Roesch, M.R., Calu, D.J. & Schoenbaum, G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615–1624 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).

    Article  CAS  PubMed  Google Scholar 

  11. Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).

    Article  CAS  PubMed  Google Scholar 

  12. Day, J.J., Roitman, M.F., Wightman, R.M. & Carelli, R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).

    Article  CAS  PubMed  Google Scholar 

  13. Takahashi, Y.K. et al. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62, 269–280 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Iordanova, M.D., Westbrook, R.F. & Killcross, A.S. Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning. Eur. J. Neurosci. 24, 3265–3270 (2006).

    Article  PubMed  Google Scholar 

  15. O'Tuathaigh, C.M. et al. The effect of amphetamine on Kamin blocking and overshadowing. Behav. Pharmacol. 14, 315–322 (2003).

    Article  CAS  PubMed  Google Scholar 

  16. Parker, J.G. et al. Absence of NMDA receptors in dopamine neurons attenuates dopamine release, but not conditioned approach, during Pavlovian conditioning. Proc. Natl. Acad. Sci. USA 107, 13491–13496 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Zweifel, L.S. et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc. Natl. Acad. Sci. USA 106, 7281–7288 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Adamantidis, A.R. et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 31, 10829–10835 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Domingos, A.I. et al. Leptin regulates the reward value of nutrient. Nat. Neurosci. 14, 1562–1568 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Tsai, H.C. et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080–1084 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Witten, I.B. et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721–733 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Boyden, E.S., Zhang, F., Bamberg, E., Nagel, G. & Deisseroth, K. Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 8, 1263–1268 (2005).

    Article  CAS  PubMed  Google Scholar 

  23. Zhang, F., Wang, L.P., Boyden, E.S. & Deisseroth, K. Channelrhodopsin-2 and optical control of excitable cells. Nat. Methods 3, 785–792 (2006).

    Article  CAS  PubMed  Google Scholar 

  24. Kamin, L.J. “Attention-like” processes in classical conditioning. in Miami Symposium on the Prediction of Behavior, 1967: Aversive Stimulation (ed. Jones, M.R.) 9–31 (University of Miami Press, 1968).

  25. Kamin, L.J. Selective association and conditioning. in Fundamental Issues in Associative Learning (eds. Mackintosh, N.J. & Honig, F.W.K.) 42–64 (Dalhousie University Press, 1969).

  26. Kamin, L.J. Predictability, surprise, attention and conditioning. in Punishment and Aversive Behavior (eds. Campbell, B.A. & Church, R.M.) 279–296 (Appleton-Century-Crofts, New York, NY, 1969).

  27. Holland, P.C. Unblocking in Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 10, 476–497 (1984).

    Article  CAS  PubMed  Google Scholar 

  28. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).

    Article  CAS  PubMed  Google Scholar 

  29. Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).

    Article  CAS  PubMed  Google Scholar 

  30. Burke, K.A., Franz, T.M., Miller, D.N. & Schoenbaum, G. The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 454, 340–344 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Daw, N.D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

    Article  PubMed  Google Scholar 

  32. Peters, J., Kalivas, P.W. & Quirk, G.J. Extinction circuits for fear and addiction overlap in prefrontal cortex. Learn. Mem. 16, 279–288 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  33. Becker, J.B. Gender differences in dopaminergic function in striatum and nucleus accumbens. Pharmacol. Biochem. Behav. 64, 803–812 (1999).

    Article  CAS  PubMed  Google Scholar 

  34. Berridge, K.C. & Robinson, T.E. What is the role of dopamine in reward: hedonic impact, reward learning or incentive salience? Brain Res. Rev. 28, 309–369 (1998).

    Article  CAS  PubMed  Google Scholar 

  35. Wassum, K.M., Ostlund, S.B., Balleine, B.W. & Maidment, N.T. Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling. Learn. Mem. 18, 475–483 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Beckstead, R.M., Domesick, V.B. & Nauta, W.J. Efferent connections of the substantia nigra and ventral tegmental area in the rat. Brain Res. 175, 191–217 (1979).

    Article  CAS  PubMed  Google Scholar 

  37. Fields, H.L., Hjelmstad, G.O., Margolis, E.B. & Nicola, S.M. Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289–316 (2007).

    Article  CAS  PubMed  Google Scholar 

  38. Swanson, L.W. The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res. Bull. 9, 321–353 (1982).

    Article  CAS  PubMed  Google Scholar 

  39. Reynolds, J.N. & Wickens, J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507–521 (2002).

    Article  PubMed  Google Scholar 

  40. Wickens, J.R., Horvitz, J.C., Costa, R.M. & Killcross, S. Dopaminergic mechanisms in actions and habits. J. Neurosci. 27, 8181–8183 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  41. Gerfen, C.R. & Surmeier, D.J. Modulation of striatal projection systems by dopamine. Annu. Rev. Neurosci. 34, 441–466 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Reynolds, J.N., Hyland, B.I. & Wickens, J.R. A cellular mechanism of reward-related learning. Nature 413, 67–70 (2001).

    Article  CAS  PubMed  Google Scholar 

  43. Tye, K.M. et al. Methylphenidate facilitates learning-induced amygdala plasticity. Nat. Neurosci. 13, 475–481 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Stuber, G.D. et al. Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science 321, 1690–1692 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Brown, M.T. et al. Drug-driven AMPA receptor redistribution mimicked by selective dopamine neuron stimulation. PLoS ONE 5, e15870 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Suri, R.E. TD models of reward predictive responses in dopamine neurons. Neural Netw. 15, 523–533 (2002).

    Article  PubMed  Google Scholar 

  47. Flagel, S.B. et al. A selective role for dopamine in stimulus-reward learning. Nature 469, 53–57 (2011).

    Article  CAS  PubMed  Google Scholar 

  48. Karim, B.O. et al. Estrous cycle and ovarian changes in a rat mammary carcinogenesis model after irradiation, tamoxifen chemoprevention and aging. Comp. Med. 53, 532–538 (2003).

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We are grateful for the technical assistance of M. Olsman, L. Sahuque and R. Reese, and for critical feedback from H. Fields during manuscript preparation. This research was supported by the US National Institutes of Health (grants DA015096 and AA17072 to P.H.J. and a New Innovator award to I.B.W.), funds from the State of California for medical research on alcohol and substance abuse through the University of California, San Francisco (P.H.J.), a National Science Foundation Graduate Research Fellowship (E.E.S.), and grants from the National Institute of Mental Health, the National Institute on Drug Abuse, the Michael J Fox Foundation, the Howard Hughes Medical Institute, and the Defense Advanced Research Projects Agency Reorganization and Plasticity to Accelerate Injury Recovery Program (K.D.).

Author information

Authors and Affiliations

Authors

Contributions

E.E.S., R.K. and P.H.J. designed the experiments. E.E.S., R.K. and J.R.B. performed the experiments. I.B.W. and K.D. contributed reagents. E.E.S., R.K. and P.H.J. wrote the paper with comments from all of the authors.

Corresponding author

Correspondence to Patricia H Janak.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–5 (PDF 5485 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Steinberg, E., Keiflin, R., Boivin, J. et al. A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16, 966–973 (2013). https://doi.org/10.1038/nn.3413

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.3413

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing