Abstract

Dopamine cell firing can encode errors in reward prediction, providing a learning signal to guide future behavior. Yet dopamine is also a key modulator of motivation, invigorating current behavior. Existing theories propose that fast (phasic) dopamine fluctuations support learning, whereas much slower (tonic) dopamine changes are involved in motivation. We examined dopamine release in the nucleus accumbens across multiple time scales, using complementary microdialysis and voltammetric methods during adaptive decision-making. We found that minute-by-minute dopamine levels covaried with reward rate and motivational vigor. Second-by-second dopamine release encoded an estimate of temporally discounted future reward (a value function). Changing dopamine immediately altered willingness to work and reinforced preceding action choices by encoding temporal-difference reward prediction errors. Our results indicate that dopamine conveys a single, rapidly evolving decision variable, the available reward for investment of effort, which is employed for both learning and motivational functions.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    , & A cellular mechanism of reward-related learning. Nature 413, 67–70 (2001).

  2. 2.

    , & A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

  3. 3.

    , , & Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci 10, 1020–8 (2007).

  4. 4.

    , , & Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J. Neurosci. 34, 698–704 (2014).

  5. 5.

    et al. Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement. PLoS ONE 7, e33612 (2012).

  6. 6.

    et al. A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966–973 (2013).

  7. 7.

    The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl.) 191, 391–431 (2007).

  8. 8.

    et al. Dopamine modulates reward-related vigor. Neuropsychopharmacology 38, 1495–1503 (2013).

  9. 9.

    & Regional brain dopamine metabolism: a marker for the speed, direction, and posture of moving animals. Science 229, 62–65 (1985).

  10. 10.

    , & How fast to work: response vigor, motivation and tonic dopamine. Adv. Neural Inf. Process. Syst. 18, 1019 (2006).

  11. 11.

    , , & Mice with chronically elevated dopamine exhibit enhanced motivation, but not learning, for a food reward. Neuropsychopharmacology 31, 1362–1370 (2006).

  12. 12.

    & The mysterious motivational functions of mesolimbic dopamine. Neuron 76, 470–485 (2012).

  13. 13.

    , , & Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).

  14. 14.

    , , , & Subsecond dopamine release promotes cocaine seeking. Nature 422, 614–618 (2003).

  15. 15.

    , , , & Dopamine operates as a subsecond modulator of food seeking. J. Neurosci. 24, 1265–1271 (2004).

  16. 16.

    , , , & Prolonged dopamine signaling in striatum signals proximity and value of distant rewards. Nature 500, 575–579 (2013).

  17. 17.

    , , & Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

  18. 18.

    , , , & Vigor in the face of fluctuating rates of reward: an experimental examination. J. Cogn. Neurosci. 23, 3933–3938 (2011).

  19. 19.

    , & The dorsomedial striatum encodes net expected return, critical for energizing performance vigor. Nat. Neurosci. 16, 639–647 (2013).

  20. 20.

    Foraging theory (Princeton University Press, 1986).

  21. 21.

    & Reinforcement Learning: an Introduction (MIT Press, 1998).

  22. 22.

    The goal-gradient hypothesis and maze learning. Psychol. Rev. 39, 25 (1932).

  23. 23.

    , & Response Times of carbon fiber microelectrodes to dynamic changes in catecholamine concentration. Anal. Chem. 74, 539–546 (2002).

  24. 24.

    et al. Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. Proc. Natl. Acad. Sci. USA 102, 10023–10028 (2005).

  25. 25.

    The flexible approach hypothesis: unification of effort and cue-responding hypotheses for the role of nucleus accumbens dopamine in the activation of reward-seeking behavior. J. Neurosci. 30, 16585–16600 (2010).

  26. 26.

    & The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res. Brain Res. Rev. 31, 6–41 (1999).

  27. 27.

    , & Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat. Neurosci. 13, 25–27 (2010).

  28. 28.

    & The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633 (2007).

  29. 29.

    et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 31, 10829–10835 (2011).

  30. 30.

    et al. Optimal indolence: a normative microscopic approach to work and leisure. J. R. Soc. Interface 11, 20130969 (2014).

  31. 31.

    , & What 50 years of research tell us about pausing under ratio schedules of reinforcement. Behav. Anal. 31, 39 (2008).

  32. 32.

    & Dopamine invigorates reward seeking by promoting cue-evoked excitation in the nucleus accumbens. J. Neurosci. 34, 14349–14364 (2014).

  33. 33.

    & Reward without dopamine. J. Neurosci. 23, 10827–10831 (2003).

  34. 34.

    , , , & Accumbens dopamine and the regulation of effort in food-seeking behavior: modulation of work output by different ratio or force requirements. Behav. Brain Res. 151, 83–91 (2004).

  35. 35.

    et al. Dextroamphetamine. Its cognitive and behavioral effects in normal and hyperactive boys and normal men. Arch. Gen. Psychiatry 37, 933–943 (1980).

  36. 36.

    , , , & Amping up effort: effects of d-amphetamine on human effort-based decision-making. J. Neurosci. 31, 16597–16602 (2011).

  37. 37.

    et al. From anticipation to action, the role of dopamine in perceptual decision making: an fMRI-tyrosine depletion study. J. Neurophysiol. 108, 501–512 (2012).

  38. 38.

    et al. Dissociable effects of dopamine on learning and performance within sensorimotor striatum. Basal Ganglia 4, 43–54 (2014).

  39. 39.

    & Basal ganglia contributions to motor control: a vigorous tutor. Curr. Opin. Neurobiol. 20, 704–716 (2010).

  40. 40.

    , & Evidence for hyperbolic temporal discounting of reward in control of movements. J. Neurosci. 32, 11727–11736 (2012).

  41. 41.

    , , , & Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).

  42. 42.

    & Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459, 837–41 (2009).

  43. 43.

    , , & Influence of phasic and tonic dopamine release on receptor activation. J. Neurosci. 30, 14273–14283 (2010).

  44. 44.

    , & A computational substrate for incentive salience. Trends Neurosci. 26, 423–428 (2003).

  45. 45.

    & The primate ventral pallidum encodes expected reward value and regulates motor action. Neuron 76, 826–837 (2012).

  46. 46.

    & Ventral striatum: a critical look at models of learning and evaluation. Curr. Opin. Neurobiol. 21, 387–392 (2011).

  47. 47.

    , & Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–902 (2003).

  48. 48.

    Dopamine ramps are a consequence of reward prediction errors. Neural Comput. 26, 467–471 (2014).

  49. 49.

    & Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits. Front Neural Circuits 8, 36 (2014).

  50. 50.

    et al. Striatal dopamine release is triggered by synchronized activity in cholinergic interneurons. Neuron 75, 58–64 (2012).

  51. 51.

    et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721–733 (2011).

  52. 52.

    , , & Selective activation of striatal fast-spiking interneurons during choice execution. Neuron 67, 466–79 (2010).

  53. 53.

    et al. Basal ganglia beta oscillations accompany cue utilization. Neuron 73, 523–536 (2012).

  54. 54.

    , , , & Canceling actions involves a race between basal ganglia pathways. Nat. Neurosci. 16, 1118–1124 (2013).

  55. 55.

    & Dynamic response-by-response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav. 84, 555–579 (2005).

  56. 56.

    , & Rapid decision threshold modulation by reward rate in a neural network. Neural Netw. 19, 1013–1026 (2006).

  57. 57.

    , & Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

  58. 58.

    , & Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).

  59. 59.

    , & Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Front Neurosci 6, 9 (2012).

  60. 60.

    , & Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources. Front Integr Neurosci 6, 49 (2012).

  61. 61.

    , , & In vivo neurochemical monitoring using benzoyl chloride derivatization and liquid chromatography–mass spectrometry. Anal. Chem. 84, 412–419 (2012).

  62. 62.

    , , , & Mass spectrometry “sensor” for in vivo acetylcholine monitoring. Anal. Chem. 84, 4659–4664 (2012).

  63. 63.

    et al. Regional specificity in the real-time development of phasic dopamine transmission patterns during acquisition of a cue–cocaine association in rats. Eur. J. Neurosci. 30, 1889–1899 (2009).

  64. 64.

    , & Representation and timing in theories of the dopamine system. Neural Comput. 18, 1637–1677 (2006).

  65. 65.

    et al. Optimizing the temporal resolution of fast-scan cyclic voltammetry. ACS Chem. Neurosci. 3, 285–292 (2012).

  66. 66.

    Tests of an equivalence rule for fixed and variable reinforcer delays. J. Exp. Psychol. Anim. Behav. Process. 10, 426 (1984).

  67. 67.

    Précis of breakdown of will. Behav. Brain Sci. 28, 635–650 (2005).

  68. 68.

    & Influence of reward delays on responses of dopamine neurons. J. Neurosci. 28, 7837–7846 (2008).

  69. 69.

    Normative and descriptive models of decision making: time discounting and risk sensitivity. Ciba Found. Symp. 208, 51–67 (1997).

Download references

Acknowledgements

We thank K. Berridge, T. Robinson, R. Wise, P. Redgrave, P. Dayan, D. Weissman, A. Kreitzer, N. Sanderson, D. Leventhal, S. Singh, J. Beeler, M. Walton, S. Nicola and members of the Berke laboratory for critical reading of various manuscript drafts, N. Mallet for initial assistance with viral injections, and K. Porter-Stransky for initial assistance with microdialysis procedures. Th-Cre+ rats were developed by K. Deisseroth and I. Witten and made available for distribution through RRRC (http://www.rrrc.us). This work was supported by the National Institute on Drug Abuse (DA032259, training grant DA007281), the National Institute of Mental Health (MH093888, MH101697), the National Institute on Neurological Disorders and Stroke (NS078435, training grant NS076401), and the National Institute of Biomedical Imaging and Bioengineering (EB003320). R.S. was supported by the BrainLinks-BrainTools Cluster of Excellence funded by the German Research Foundation (DFG grant number EXC1086).

Author information

Author notes

    • Robert Schmidt
    •  & Caitlin M Vander Weele

    Present address: BrainLinks-BrainTools Cluster of Excellence and Bernstein Center, University of Freiburg, Germany (R.S.), Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA (C.M.V.W.).

    • Arif A Hamid
    •  & Jeffrey R Pettibone

    These authors contributed equally to this work.

Affiliations

  1. Department of Psychology, University of Michigan, Ann Arbor, Michigan, USA.

    • Arif A Hamid
    • , Jeffrey R Pettibone
    • , Vaughn L Hetrick
    • , Robert Schmidt
    • , Caitlin M Vander Weele
    • , Brandon J Aragona
    •  & Joshua D Berke
  2. Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, USA.

    • Arif A Hamid
    • , Brandon J Aragona
    •  & Joshua D Berke
  3. Department of Chemistry, University of Michigan, Ann Arbor, Michigan, USA.

    • Omar S Mabrouk
    •  & Robert T Kennedy
  4. Department of Pharmacology, University of Michigan, Ann Arbor, Michigan, USA.

    • Omar S Mabrouk
    •  & Robert T Kennedy
  5. Department of Biomedical Engineering, University of Michigan, Ann Arbor, Michigan, USA.

    • Joshua D Berke

Authors

  1. Search for Arif A Hamid in:

  2. Search for Jeffrey R Pettibone in:

  3. Search for Omar S Mabrouk in:

  4. Search for Vaughn L Hetrick in:

  5. Search for Robert Schmidt in:

  6. Search for Caitlin M Vander Weele in:

  7. Search for Robert T Kennedy in:

  8. Search for Brandon J Aragona in:

  9. Search for Joshua D Berke in:

Contributions

A.A.H. performed and analyzed both FSCV and optogenetic experiments, and J.R.P. performed and analyzed the microdialysis experiments. O.S.M. assisted with microdialysis, C.M.V.W. assisted with FSCV, V.L.H. assisted with optogenetics and R.S. assisted with reinforcement learning models. B.J.A. helped supervise the FSCV experiments and data analysis, and R.T.K. helped supervise microdialysis experiments. J.D.B. designed and supervised the study, performed the computational modeling, developed the theoretical interpretation, and wrote the manuscript.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Joshua D Berke.

Integrated supplementary information

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–10

  2. 2.

    Supplementary Methods Checklist

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn.4173

Further reading