Article

A causal link between prediction errors, dopamine neurons and learning

  • Nature Neuroscience volume 16, pages 966973 (2013)
  • doi:10.1038/nn.3413
  • Download Citation
Received:
Accepted:
Published:

Abstract

Situations in which rewards are unexpectedly obtained or withheld represent opportunities for new learning. Often, this learning includes identifying cues that predict reward availability. Unexpected rewards strongly activate midbrain dopamine neurons. This phasic signal is proposed to support learning about antecedent cues by signaling discrepancies between actual and expected outcomes, termed a reward prediction error. However, it is unknown whether dopamine neuron prediction error signaling and cue-reward learning are causally linked. To test this hypothesis, we manipulated dopamine neuron activity in rats in two behavioral procedures, associative blocking and extinction, that illustrate the essential function of prediction errors in learning. We observed that optogenetic activation of dopamine neurons concurrent with reward delivery, mimicking a prediction error, was sufficient to cause long-lasting increases in cue-elicited reward-seeking behavior. Our findings establish a causal role for temporally precise dopamine neuron signaling in cue-reward learning, bridging a critical gap between experimental evidence and influential theoretical frameworks.

  • Subscribe to Nature Neuroscience for full access:

    $59

    Subscribe

Additional access options:

Already a subscriber?  Log in  now or  Register  for online access.

References

  1. 1.

    & A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton Century Crofts, New York, 1972).

  2. 2.

    Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. USA 108 (suppl. 3): 15647–15654 (2011).

  3. 3.

    , & A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

  4. 4.

    , & A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

  5. 5.

    & Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473–500 (2000).

  6. 6.

    & Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170 (1981).

  7. 7.

    , & Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).

  8. 8.

    , , , & Neuron type–specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012).

  9. 9.

    , & Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615–1624 (2007).

  10. 10.

    & Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).

  11. 11.

    & Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).

  12. 12.

    , , & Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).

  13. 13.

    et al. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62, 269–280 (2009).

  14. 14.

    , & Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning. Eur. J. Neurosci. 24, 3265–3270 (2006).

  15. 15.

    et al. The effect of amphetamine on Kamin blocking and overshadowing. Behav. Pharmacol. 14, 315–322 (2003).

  16. 16.

    et al. Absence of NMDA receptors in dopamine neurons attenuates dopamine release, but not conditioned approach, during Pavlovian conditioning. Proc. Natl. Acad. Sci. USA 107, 13491–13496 (2010).

  17. 17.

    et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc. Natl. Acad. Sci. USA 106, 7281–7288 (2009).

  18. 18.

    et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 31, 10829–10835 (2011).

  19. 19.

    et al. Leptin regulates the reward value of nutrient. Nat. Neurosci. 14, 1562–1568 (2011).

  20. 20.

    et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080–1084 (2009).

  21. 21.

    et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721–733 (2011).

  22. 22.

    , , , & Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 8, 1263–1268 (2005).

  23. 23.

    , , & Channelrhodopsin-2 and optical control of excitable cells. Nat. Methods 3, 785–792 (2006).

  24. 24.

    “Attention-like” processes in classical conditioning. in Miami Symposium on the Prediction of Behavior, 1967: Aversive Stimulation (ed. Jones, M.R.) 9–31 (University of Miami Press, 1968).

  25. 25.

    Selective association and conditioning. in Fundamental Issues in Associative Learning (eds. Mackintosh, N.J. & Honig, F.W.K.) 42–64 (Dalhousie University Press, 1969).

  26. 26.

    Predictability, surprise, attention and conditioning. in Punishment and Aversive Behavior (eds. Campbell, B.A. & Church, R.M.) 279–296 (Appleton-Century-Crofts, New York, NY, 1969).

  27. 27.

    Unblocking in Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 10, 476–497 (1984).

  28. 28.

    , & Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).

  29. 29.

    Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).

  30. 30.

    , , & The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 454, 340–344 (2008).

  31. 31.

    , & Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

  32. 32.

    , & Extinction circuits for fear and addiction overlap in prefrontal cortex. Learn. Mem. 16, 279–288 (2009).

  33. 33.

    Gender differences in dopaminergic function in striatum and nucleus accumbens. Pharmacol. Biochem. Behav. 64, 803–812 (1999).

  34. 34.

    & What is the role of dopamine in reward: hedonic impact, reward learning or incentive salience? Brain Res. Rev. 28, 309–369 (1998).

  35. 35.

    , , & Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling. Learn. Mem. 18, 475–483 (2011).

  36. 36.

    , & Efferent connections of the substantia nigra and ventral tegmental area in the rat. Brain Res. 175, 191–217 (1979).

  37. 37.

    , , & Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289–316 (2007).

  38. 38.

    The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res. Bull. 9, 321–353 (1982).

  39. 39.

    & Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507–521 (2002).

  40. 40.

    , , & Dopaminergic mechanisms in actions and habits. J. Neurosci. 27, 8181–8183 (2007).

  41. 41.

    & Modulation of striatal projection systems by dopamine. Annu. Rev. Neurosci. 34, 441–466 (2011).

  42. 42.

    , & A cellular mechanism of reward-related learning. Nature 413, 67–70 (2001).

  43. 43.

    et al. Methylphenidate facilitates learning-induced amygdala plasticity. Nat. Neurosci. 13, 475–481 (2010).

  44. 44.

    et al. Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science 321, 1690–1692 (2008).

  45. 45.

    et al. Drug-driven AMPA receptor redistribution mimicked by selective dopamine neuron stimulation. PLoS ONE 5, e15870 (2010).

  46. 46.

    TD models of reward predictive responses in dopamine neurons. Neural Netw. 15, 523–533 (2002).

  47. 47.

    et al. A selective role for dopamine in stimulus-reward learning. Nature 469, 53–57 (2011).

  48. 48.

    et al. Estrous cycle and ovarian changes in a rat mammary carcinogenesis model after irradiation, tamoxifen chemoprevention and aging. Comp. Med. 53, 532–538 (2003).

Download references

Acknowledgements

We are grateful for the technical assistance of M. Olsman, L. Sahuque and R. Reese, and for critical feedback from H. Fields during manuscript preparation. This research was supported by the US National Institutes of Health (grants DA015096 and AA17072 to P.H.J. and a New Innovator award to I.B.W.), funds from the State of California for medical research on alcohol and substance abuse through the University of California, San Francisco (P.H.J.), a National Science Foundation Graduate Research Fellowship (E.E.S.), and grants from the National Institute of Mental Health, the National Institute on Drug Abuse, the Michael J Fox Foundation, the Howard Hughes Medical Institute, and the Defense Advanced Research Projects Agency Reorganization and Plasticity to Accelerate Injury Recovery Program (K.D.).

Author information

Author notes

    • Elizabeth E Steinberg
    •  & Ronald Keiflin

    These authors contributed equally to this work.

Affiliations

  1. Ernest Gallo Clinic and Research Center, University of California, San Francisco, California, USA.

    • Elizabeth E Steinberg
    • , Ronald Keiflin
    • , Josiah R Boivin
    •  & Patricia H Janak
  2. Graduate program in Neuroscience, University of California, San Francisco, California, USA.

    • Elizabeth E Steinberg
    • , Josiah R Boivin
    •  & Patricia H Janak
  3. Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, USA.

    • Ilana B Witten
  4. Department of Psychology, Princeton University, Princeton, New Jersey, USA.

    • Ilana B Witten
  5. Department of Bioengineering, Stanford University, Stanford, California, USA.

    • Karl Deisseroth
  6. Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, California, USA.

    • Karl Deisseroth
  7. Howard Hughes Medical Institute, Stanford University, Stanford, California, USA.

    • Karl Deisseroth
  8. Crack the Neural Code Program, Stanford University, Stanford, California, USA

    • Karl Deisseroth
  9. Department of Neurology, University of California, San Francisco, California, USA.

    • Patricia H Janak
  10. Wheeler Center for the Neurobiology of Addiction, University of California, San Francisco, California, USA.

    • Patricia H Janak

Authors

  1. Search for Elizabeth E Steinberg in:

  2. Search for Ronald Keiflin in:

  3. Search for Josiah R Boivin in:

  4. Search for Ilana B Witten in:

  5. Search for Karl Deisseroth in:

  6. Search for Patricia H Janak in:

Contributions

E.E.S., R.K. and P.H.J. designed the experiments. E.E.S., R.K. and J.R.B. performed the experiments. I.B.W. and K.D. contributed reagents. E.E.S., R.K. and P.H.J. wrote the paper with comments from all of the authors.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Patricia H Janak.

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–5