Abstract
Situations in which rewards are unexpectedly obtained or withheld represent opportunities for new learning. Often, this learning includes identifying cues that predict reward availability. Unexpected rewards strongly activate midbrain dopamine neurons. This phasic signal is proposed to support learning about antecedent cues by signaling discrepancies between actual and expected outcomes, termed a reward prediction error. However, it is unknown whether dopamine neuron prediction error signaling and cue-reward learning are causally linked. To test this hypothesis, we manipulated dopamine neuron activity in rats in two behavioral procedures, associative blocking and extinction, that illustrate the essential function of prediction errors in learning. We observed that optogenetic activation of dopamine neurons concurrent with reward delivery, mimicking a prediction error, was sufficient to cause long-lasting increases in cue-elicited reward-seeking behavior. Our findings establish a causal role for temporally precise dopamine neuron signaling in cue-reward learning, bridging a critical gap between experimental evidence and influential theoretical frameworks.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Rescorla, R.A. & Wagner, A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton Century Crofts, New York, 1972).
Glimcher, P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. USA 108 (suppl. 3): 15647–15654 (2011).
Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473–500 (2000).
Sutton, R.S. & Barto, A.G. Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170 (1981).
Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).
Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B. & Uchida, N. Neuron type–specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012).
Roesch, M.R., Calu, D.J. & Schoenbaum, G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615–1624 (2007).
Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).
Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).
Day, J.J., Roitman, M.F., Wightman, R.M. & Carelli, R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).
Takahashi, Y.K. et al. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62, 269–280 (2009).
Iordanova, M.D., Westbrook, R.F. & Killcross, A.S. Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning. Eur. J. Neurosci. 24, 3265–3270 (2006).
O'Tuathaigh, C.M. et al. The effect of amphetamine on Kamin blocking and overshadowing. Behav. Pharmacol. 14, 315–322 (2003).
Parker, J.G. et al. Absence of NMDA receptors in dopamine neurons attenuates dopamine release, but not conditioned approach, during Pavlovian conditioning. Proc. Natl. Acad. Sci. USA 107, 13491–13496 (2010).
Zweifel, L.S. et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc. Natl. Acad. Sci. USA 106, 7281–7288 (2009).
Adamantidis, A.R. et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 31, 10829–10835 (2011).
Domingos, A.I. et al. Leptin regulates the reward value of nutrient. Nat. Neurosci. 14, 1562–1568 (2011).
Tsai, H.C. et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080–1084 (2009).
Witten, I.B. et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721–733 (2011).
Boyden, E.S., Zhang, F., Bamberg, E., Nagel, G. & Deisseroth, K. Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 8, 1263–1268 (2005).
Zhang, F., Wang, L.P., Boyden, E.S. & Deisseroth, K. Channelrhodopsin-2 and optical control of excitable cells. Nat. Methods 3, 785–792 (2006).
Kamin, L.J. “Attention-like” processes in classical conditioning. in Miami Symposium on the Prediction of Behavior, 1967: Aversive Stimulation (ed. Jones, M.R.) 9–31 (University of Miami Press, 1968).
Kamin, L.J. Selective association and conditioning. in Fundamental Issues in Associative Learning (eds. Mackintosh, N.J. & Honig, F.W.K.) 42–64 (Dalhousie University Press, 1969).
Kamin, L.J. Predictability, surprise, attention and conditioning. in Punishment and Aversive Behavior (eds. Campbell, B.A. & Church, R.M.) 279–296 (Appleton-Century-Crofts, New York, NY, 1969).
Holland, P.C. Unblocking in Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 10, 476–497 (1984).
Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).
Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).
Burke, K.A., Franz, T.M., Miller, D.N. & Schoenbaum, G. The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 454, 340–344 (2008).
Daw, N.D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).
Peters, J., Kalivas, P.W. & Quirk, G.J. Extinction circuits for fear and addiction overlap in prefrontal cortex. Learn. Mem. 16, 279–288 (2009).
Becker, J.B. Gender differences in dopaminergic function in striatum and nucleus accumbens. Pharmacol. Biochem. Behav. 64, 803–812 (1999).
Berridge, K.C. & Robinson, T.E. What is the role of dopamine in reward: hedonic impact, reward learning or incentive salience? Brain Res. Rev. 28, 309–369 (1998).
Wassum, K.M., Ostlund, S.B., Balleine, B.W. & Maidment, N.T. Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling. Learn. Mem. 18, 475–483 (2011).
Beckstead, R.M., Domesick, V.B. & Nauta, W.J. Efferent connections of the substantia nigra and ventral tegmental area in the rat. Brain Res. 175, 191–217 (1979).
Fields, H.L., Hjelmstad, G.O., Margolis, E.B. & Nicola, S.M. Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289–316 (2007).
Swanson, L.W. The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res. Bull. 9, 321–353 (1982).
Reynolds, J.N. & Wickens, J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507–521 (2002).
Wickens, J.R., Horvitz, J.C., Costa, R.M. & Killcross, S. Dopaminergic mechanisms in actions and habits. J. Neurosci. 27, 8181–8183 (2007).
Gerfen, C.R. & Surmeier, D.J. Modulation of striatal projection systems by dopamine. Annu. Rev. Neurosci. 34, 441–466 (2011).
Reynolds, J.N., Hyland, B.I. & Wickens, J.R. A cellular mechanism of reward-related learning. Nature 413, 67–70 (2001).
Tye, K.M. et al. Methylphenidate facilitates learning-induced amygdala plasticity. Nat. Neurosci. 13, 475–481 (2010).
Stuber, G.D. et al. Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science 321, 1690–1692 (2008).
Brown, M.T. et al. Drug-driven AMPA receptor redistribution mimicked by selective dopamine neuron stimulation. PLoS ONE 5, e15870 (2010).
Suri, R.E. TD models of reward predictive responses in dopamine neurons. Neural Netw. 15, 523–533 (2002).
Flagel, S.B. et al. A selective role for dopamine in stimulus-reward learning. Nature 469, 53–57 (2011).
Karim, B.O. et al. Estrous cycle and ovarian changes in a rat mammary carcinogenesis model after irradiation, tamoxifen chemoprevention and aging. Comp. Med. 53, 532–538 (2003).
Acknowledgements
We are grateful for the technical assistance of M. Olsman, L. Sahuque and R. Reese, and for critical feedback from H. Fields during manuscript preparation. This research was supported by the US National Institutes of Health (grants DA015096 and AA17072 to P.H.J. and a New Innovator award to I.B.W.), funds from the State of California for medical research on alcohol and substance abuse through the University of California, San Francisco (P.H.J.), a National Science Foundation Graduate Research Fellowship (E.E.S.), and grants from the National Institute of Mental Health, the National Institute on Drug Abuse, the Michael J Fox Foundation, the Howard Hughes Medical Institute, and the Defense Advanced Research Projects Agency Reorganization and Plasticity to Accelerate Injury Recovery Program (K.D.).
Author information
Authors and Affiliations
Contributions
E.E.S., R.K. and P.H.J. designed the experiments. E.E.S., R.K. and J.R.B. performed the experiments. I.B.W. and K.D. contributed reagents. E.E.S., R.K. and P.H.J. wrote the paper with comments from all of the authors.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–5 (PDF 5485 kb)
Rights and permissions
About this article
Cite this article
Steinberg, E., Keiflin, R., Boivin, J. et al. A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16, 966–973 (2013). https://doi.org/10.1038/nn.3413
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.3413
This article is cited by
-
Goal-directed learning in adolescence: neurocognitive development and contextual influences
Nature Reviews Neuroscience (2024)
-
Downstream network transformations dissociate neural activity from causal functional contributions
Scientific Reports (2024)
-
Dopamine-independent effect of rewards on choices through hidden-state inference
Nature Neuroscience (2024)
-
Role of dopamine D1 receptor in the modulation of memory consolidation by passive and self-administered heroin and associated conditioned stimuli
Scientific Reports (2023)
-
The effect of prediction error on episodic memory encoding is modulated by the outcome of the predictions
npj Science of Learning (2023)