Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

The temporal precision of reward prediction in dopamine neurons

Nature Neuroscience volume 11, pages 966–973 (2008)Cite this article

Abstract

Midbrain dopamine neurons are activated when reward is greater than predicted, and this error signal could teach target neurons both the value of reward and when it will occur. We used the dopamine error signal to measure how the expectation of reward was distributed over time. Animals were trained with fixed-duration intervals of 1–16 s between conditioned stimulus onset and reward. In contrast to the weak responses that have been observed after short intervals (1–2 s), activations to reward increased steeply and linearly with the logarithm of the interval. Results with varied stimulus-reward intervals suggest that the neural expectation was substantial after just half an interval had elapsed. Thus, the neural expectation of reward in these experiments was not highly precise and the precision declined sharply with interval duration. The neural precision of expectation appeared to be at least qualitatively similar to the precision of anticipatory licking behavior.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Timing of anticipatory licking behavior (Experiment 1).
Figure 2: Dopamine neurons are sensitive to interval duration (Experiment 1).
Figure 3: Response of dopamine neurons to juice delivered earlier or later than usual (Experiment 2).
Figure 4: Response of dopamine neurons to juice delivered following a stimulus-reward interval that varied across trials (Experiment 3).
Figure 5: Responses of dopamine neurons as a function of a stimulus-reward interval that varies from trial to trial (Experiment 3).
Figure 6: A model of interval timing and the dopamine error signal that could account for the data of Figure 2d.

Similar content being viewed by others

References

  1. Buhusi, C.V. & Meck, W.H. What makes us tick? Functional and neural mechanisms of interval timing. Nat. Rev. Neurosci. 6, 755–765 (2005).

    Article  CAS  PubMed  Google Scholar 

  2. Mauk, M.D. & Buonomano, D.V. The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004).

    Article  CAS  PubMed  Google Scholar 

  3. Rao, S.M., Mayer, A.R. & Harrington, D.L. The evolution of brain activation during temporal processing. Nat. Neurosci. 4, 317–323 (2001).

    Article  CAS  PubMed  Google Scholar 

  4. Gibbon, J., Malapani, C., Dale, C.L. & Gallistel, C.R. Toward a neurobiology of temporal cognition: advances and challenges. Curr. Opin. Neurobiol. 7, 170–184 (1997).

    Article  CAS  PubMed  Google Scholar 

  5. Meck, W.H. Neuropharmacology of timing and time perception. Brain Res. Cogn. Brain Res. 3, 227–242 (1996).

    Article  CAS  PubMed  Google Scholar 

  6. Matell, M.S. & Meck, W.H. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Brain Res. Cogn. Brain Res. 21, 139–170 (2004).

    Article  PubMed  Google Scholar 

  7. Houk, J., Adams, J. & Barto, A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia (eds Houk, J., Davis, J. & Beiser, D.) 249–270 (MIT Press, Cambridge, Massachusetts, 1995).

    Google Scholar 

  8. Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    Article  CAS  PubMed  Google Scholar 

  9. Schultz, W., Dayan, P. & Montague, R.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  PubMed  Google Scholar 

  10. Berns, G.S. & Sejnowski, T.J. A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 10, 108–121 (1998).

    Article  CAS  PubMed  Google Scholar 

  11. Brown, J., Bullock, D. & Grossberg, S. How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci. 19, 10502–10511 (1999).

    Article  CAS  PubMed  Google Scholar 

  12. Contreras-Vidal, J.L. & Schultz, W. A predictive reinforcement model of dopamine neurons for learning approach behavior. J. Comput. Neurosci. 6, 191–214 (1999).

    Article  CAS  PubMed  Google Scholar 

  13. Suri, R.E. & Schultz, W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890 (1999).

    Article  CAS  PubMed  Google Scholar 

  14. Daw, N.D., Courville, A.C. & Touretzky, D.S. Representation and timing in theories of the dopamine system. Neural Comput. 18, 1637–1677 (2006).

    Article  PubMed  Google Scholar 

  15. Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).

    Article  CAS  PubMed  Google Scholar 

  16. Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).

    Article  CAS  PubMed  Google Scholar 

  17. Schultz, W. The predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).

    Article  CAS  PubMed  Google Scholar 

  18. Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).

    Article  CAS  PubMed  Google Scholar 

  19. Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).

    Article  CAS  PubMed  Google Scholar 

  20. Nakahara, H., Itho, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004).

    Article  CAS  PubMed  Google Scholar 

  21. Morris, G., Arkadir, D., Nevet, A., Vaadia, E. & Bergman, H. Coincident, but distinct, messages of midbrain dopamine and striatal tonically active neurons. Neuron 43, 133–143 (2004).

    Article  CAS  PubMed  Google Scholar 

  22. Pan, W.X., Schmidt, R., Wickens, J.R. & Hyland, B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).

    Article  CAS  PubMed  Google Scholar 

  23. Tobler, P.N., Fiorillo, C.D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).

    Article  CAS  PubMed  Google Scholar 

  24. Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Rakitin, B.C. et al. Scalar expectancy theory and peak-interval timing in humans. J. Exp. Psychol. Anim. Behav. Process. 24, 15–33 (1998).

    Article  CAS  PubMed  Google Scholar 

  26. Kacelnik, A. & Brito e Abreu, F. Risky choice and Weber's Law. J. Theor. Biol. 194, 289–298 (1998).

    Article  CAS  PubMed  Google Scholar 

  27. Janssen, P. & Shadlen, M.N. A representation of the hazard rate of elapsed time in macaque area LIP. Nat. Neurosci. 8, 234–241 (2005).

    Article  CAS  PubMed  Google Scholar 

  28. Ghose, G.M. & Maunsell, J.H.R. Attentional modulation in visual cortex depends on task-timing. Nature 419, 616–620 (2002).

    Article  CAS  PubMed  Google Scholar 

  29. Bateson, M. & Kacelnik, A. Accuracy of memory for amount in the foraging starling, Sturnus vulgaris. Anim. Behav. 50, 431–443 (1995).

    Article  Google Scholar 

  30. Komura, Y. et al. Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412, 546–549 (2001).

    Article  CAS  PubMed  Google Scholar 

  31. Brody, C.D., Hernandez, A., Zanos, A. & Romo, R. Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb. Cortex 13, 1196–1207 (2003).

    Article  Google Scholar 

  32. Leon, M.I. & Shadlen, M.N. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron 38, 317–327 (2003).

    Article  CAS  PubMed  Google Scholar 

  33. Renoult, L.Roux. S. & Riehle, A. Time is a rubberband: neuronal activity in monkey motor cortex in relation to time estimation. Eur. J. Neurosci. 23, 3098–3108 (2006).

    Article  PubMed  Google Scholar 

  34. O'Reilly, R.C., Frank, M.J., Hazy, T.E. & Watz, B. PVLV: the primary value and learned value Pavlovian learning algorithm. Behav. Neurosci. 121, 31–49 (2007).

    Article  PubMed  Google Scholar 

  35. Grossberg, S. & Schmajuk, N.A. Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Netw. 2, 79–102 (1989).

    Article  Google Scholar 

  36. Medina, J.F., Nores, W.L. & Mauk, M.D. Inhibition of climbing fibers is a signal for the extinction of conditioned eyelid responses. Nature 416, 330–333 (2002).

    Article  CAS  PubMed  Google Scholar 

  37. Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57, 201–217 (1987).

    Article  CAS  PubMed  Google Scholar 

  38. Nieder, A. & Miller, E.K. Coding of cognitive magnitude: compressed scaling of numerical information in the primate prefrontal cortex. Neuron 37, 149–157 (2003).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by grants from the Human Frontiers Science Program (C.D.F.), the Howard Hughes Medical Institute (W.T.N.), the US National Institutes of Health (EY 05603, W.T.N.), the Swiss National Science Funds (W.S.) and the Wellcome Trust (W.S.).

Author information

Authors and Affiliations

  1. Department of Neurobiology, Fairchild Building, D209, 299 Campus Drive West, Stanford University, Stanford, 94305-5125, California, USA

    Christopher D Fiorillo & William T Newsome

  2. Department of Physiology, Development and Neuroscience, University of Cambridge, Downing Street, Cambridge, CB2 3DY, UK

    Wolfram Schultz

Authors
  1. Christopher D Fiorillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William T Newsome
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wolfram Schultz
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.D.F. conducted the experiments, analyzed the data and developed the mathematical model of dopamine responses. C.D.F. and W.S. designed the experiments. C.D.F. wrote the manuscript with feedback from W.T.N. and W.S.

Corresponding author

Correspondence to Christopher D Fiorillo.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–4, Supplementary Methods and Supplementary Results (PDF 3905 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fiorillo, C., Newsome, W. & Schultz, W. The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11, 966–973 (2008). https://doi.org/10.1038/nn.2159

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.2159

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing