Letter | Published:

Arithmetic and local circuitry underlying dopamine prediction errors

Nature volume 525, pages 243246 (10 September 2015) | Download Citation

  • An Erratum to this article was published on 07 October 2015

Abstract

Dopamine neurons are thought to facilitate learning by comparing actual and expected reward1,2. Despite two decades of investigation, little is known about how this comparison is made. To determine how dopamine neurons calculate prediction error, we combined optogenetic manipulations with extracellular recordings in the ventral tegmental area while mice engaged in classical conditioning. Here we demonstrate, by manipulating the temporal expectation of reward, that dopamine neurons perform subtraction, a computation that is ideal for reinforcement learning but rarely observed in the brain. Furthermore, selectively exciting and inhibiting neighbouring GABA (γ-aminobutyric acid) neurons in the ventral tegmental area reveals that these neurons are a source of subtraction: they inhibit dopamine neurons when reward is expected, causally contributing to prediction-error calculations. Finally, bilaterally stimulating ventral tegmental area GABA neurons dramatically reduces anticipatory licking to conditioned odours, consistent with an important role for these neurons in reinforcement learning. Together, our results uncover the arithmetic and local circuitry underlying dopamine prediction errors.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    , & A neural substrate of prediction and reward. Science 275, 1593–1599 (1997)

  2. 2.

    & Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005)

  3. 3.

    & A mathematical model for simple learning. Psychol. Rev. 58, 313–323 (1951)

  4. 4.

    & in Classical Conditioning II: Current Research and Theory (eds & ) 64–99 (Appleton-Century-Crofts, 1972)

  5. 5.

    & Normalization as a canonical neural computation. Nature Rev. Neurosci. 13, 51–62 (2012)

  6. 6.

    , , , & Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012)

  7. 7.

    , & Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005)

  8. 8.

    Neuronal arithmetic. Nature Rev. Neurosci. 11, 474–489 (2010)

  9. 9.

    , & in Models of Information Processing in the Basal Ganglia (eds , & ) 249–270 (MIT Press, 1995)

  10. 10.

    & Efficient reinforcement learning: computational theories, neuroscience and robotics. Curr. Opin. Neurobiol. 17, 205–212 (2007)

  11. 11.

    & Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111–1115 (2007)

  12. 12.

    , , , & Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. J. Neurosci. 31, 11457–11471 (2011)

  13. 13.

    & Ultrastructural analysis of local collaterals of rat ventral tegmental area neurons: GABA phenotype and synapses onto dopamine and GABA cells. Synapse 63, 895–906 (2009)

  14. 14.

    , , & Activation of VTA GABA neurons disrupts reward consumption. Neuron 73, 1184–1194 (2012)

  15. 15.

    et al. GABA neurons of the VTA drive conditioned place aversion. Neuron 73, 1173–1183 (2012)

  16. 16.

    , & Neural mechanisms of acquired phasic dopamine responses in learning. Neurosci. Biobehav. Rev. 34, 701–720 (2010)

  17. 17.

    , & Conditioning and time representation in long short-term memory networks. Biol. Cybern. 108, 23–48 (2014)

  18. 18.

    & Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area. Front. Neurorobot. 8, 4 (2014)

  19. 19.

    , & Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput. 20, 3034–3054 (2008)

  20. 20.

    & A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward. J. Neurosci. 28, 10062–10074 (2008)

  21. 21.

    et al. A high-light sensitivity optical neural silencer: development and application to optogenetic control of non-human primate cortex. Front. Syst. Neurosci. 5, 18 (2011)

  22. 22.

    , & Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli. J. Neurosci. 33, 4710–4725 (2013)

  23. 23.

    et al. Cortical interneurons that specialize in disinhibitory control. Nature 503, 521–524 (2013)

  24. 24.

    , , & Division and subtraction by distinct cortical inhibitory networks in vivo. Nature 488, 343–348 (2012)

  25. 25.

    , , & Parvalbumin-expressing interneurons linearly transform cortical responses to visual stimuli. Neuron 73, 159–170 (2012)

  26. 26.

    & Multiplicative gain changes are induced by excitation or inhibition alone. J. Neurosci. 23, 10040–10051 (2003)

  27. 27.

    & Gain Modulation of neuronal responses by subtractive and divisive mechanisms of inhibition. J. Neurophysiol. 101, 958–968 (2009)

  28. 28.

    & Shunting inhibition does not have a divisive effect on firing rates. Neural Comput. 9, 1001–1013 (1997)

  29. 29.

    & Dissociating self-generated from passively applied head motion: neural mechanisms in the vestibular nuclei. J. Neurosci. 24, 2102–2111 (2004)

  30. 30.

    , , & Spatiotemporal elements of macaque V1 receptive fields. Neuron 46, 945–956 (2005)

  31. 31.

    et al. Characterization of a mouse strain expressing Cre recombinase from the 3′ untranslated region of the dopamine transporter locus. Genesis 44, 383–390 (2006)

  32. 32.

    et al. Leptin action on GABAergic neurons prevents obesity and reduces inhibitory tone to POMC neurons. Neuron 71, 142–154 (2011)

  33. 33.

    , , , & Millisecond-timescale, genetically targeted optical control of neural activity. Nature Neurosci. 8, 1263–1268 (2005)

  34. 34.

    , , & A FLEX switch targets Channelrhodopsin-2 to multiple cell types for imaging and long-range circuit mapping. J. Neurosci. 28, 7025–7030 (2008)

  35. 35.

    & Speed and accuracy of olfactory discrimination in the rat. Nature Neurosci. 6, 1224–1229 (2003)

  36. 36.

    & Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple T task. J. Neurophysiol. 91, 2259–2272 (2004)

  37. 37.

    , , & PINP: a new method of tagging neuronal populations for identification during in vivo electrophysiological recording. PLoS One 4, e6099 (2009)

  38. 38.

    et al. Distinct behavioural and network correlates of two interneuron types in prefrontal cortex. Nature 498, 363–366 (2013)

  39. 39.

    , & Divisive normalization in olfactory population codes. Neuron 66, 287–299 (2010)

Download references

Acknowledgements

We thank M. Andermann, J. Assad, R. Born, J. Buckholtz, P. Glimcher, J. Maunsell, B. Sabatini, W. Schultz, R. Wilson, and members of the Uchida laboratory for comments on the manuscript; S. Haesler for technical expertise and discussions on experimental design; E. Molnar for histology assistance; C. Dulac for sharing resources; K. Deisseroth for the AAV–FLEX–ChR2 construct; and E. Boyden for the AAV–FLEX–ArchT construct. This work was supported by a Sackler Fellowship in Psychobiology (N.E.) and National Institutes of Health grants T32GM007753 (to N.E.), F30MH100729 (to N.E.), R01MH095953 (to N.U.), and R01MH101207 (to N.U.).

Author information

Affiliations

  1. Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, Massachusetts 02138, USA

    • Neir Eshel
    • , Michael Bukwich
    • , Vinod Rao
    • , Vivian Hemmelder
    • , Ju Tian
    •  & Naoshige Uchida

Authors

  1. Search for Neir Eshel in:

  2. Search for Michael Bukwich in:

  3. Search for Vinod Rao in:

  4. Search for Vivian Hemmelder in:

  5. Search for Ju Tian in:

  6. Search for Naoshige Uchida in:

Contributions

N.E. and N.U. designed the recording experiments. N.E., V.R., and N.U. designed the behaviour experiment. N.E., M.B., V.R., V.H., and J.T. collected data. N.E., M.B., and V.R. analysed data. N.E. wrote the manuscript with comments from N.U.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Naoshige Uchida.

Extended data

Supplementary information

PDF files

  1. 1.

    Supplementary Information

    This file contains Supplementary Text.

Excel files

  1. 1.

    Supplementary Information

    This file contains Supplementary Table 1.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nature14855

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.