Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The temporal precision of reward prediction in dopamine neurons

Abstract

Midbrain dopamine neurons are activated when reward is greater than predicted, and this error signal could teach target neurons both the value of reward and when it will occur. We used the dopamine error signal to measure how the expectation of reward was distributed over time. Animals were trained with fixed-duration intervals of 1–16 s between conditioned stimulus onset and reward. In contrast to the weak responses that have been observed after short intervals (1–2 s), activations to reward increased steeply and linearly with the logarithm of the interval. Results with varied stimulus-reward intervals suggest that the neural expectation was substantial after just half an interval had elapsed. Thus, the neural expectation of reward in these experiments was not highly precise and the precision declined sharply with interval duration. The neural precision of expectation appeared to be at least qualitatively similar to the precision of anticipatory licking behavior.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Timing of anticipatory licking behavior (Experiment 1).
Figure 2: Dopamine neurons are sensitive to interval duration (Experiment 1).
Figure 3: Response of dopamine neurons to juice delivered earlier or later than usual (Experiment 2).
Figure 4: Response of dopamine neurons to juice delivered following a stimulus-reward interval that varied across trials (Experiment 3).
Figure 5: Responses of dopamine neurons as a function of a stimulus-reward interval that varies from trial to trial (Experiment 3).
Figure 6: A model of interval timing and the dopamine error signal that could account for the data of Figure 2d.

References

  1. 1

    Buhusi, C.V. & Meck, W.H. What makes us tick? Functional and neural mechanisms of interval timing. Nat. Rev. Neurosci. 6, 755–765 (2005).

    CAS  Article  PubMed  Google Scholar 

  2. 2

    Mauk, M.D. & Buonomano, D.V. The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004).

    CAS  Article  PubMed  Google Scholar 

  3. 3

    Rao, S.M., Mayer, A.R. & Harrington, D.L. The evolution of brain activation during temporal processing. Nat. Neurosci. 4, 317–323 (2001).

    CAS  Article  PubMed  Google Scholar 

  4. 4

    Gibbon, J., Malapani, C., Dale, C.L. & Gallistel, C.R. Toward a neurobiology of temporal cognition: advances and challenges. Curr. Opin. Neurobiol. 7, 170–184 (1997).

    CAS  Article  PubMed  Google Scholar 

  5. 5

    Meck, W.H. Neuropharmacology of timing and time perception. Brain Res. Cogn. Brain Res. 3, 227–242 (1996).

    CAS  Article  PubMed  Google Scholar 

  6. 6

    Matell, M.S. & Meck, W.H. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Brain Res. Cogn. Brain Res. 21, 139–170 (2004).

    Article  PubMed  Google Scholar 

  7. 7

    Houk, J., Adams, J. & Barto, A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia (eds Houk, J., Davis, J. & Beiser, D.) 249–270 (MIT Press, Cambridge, Massachusetts, 1995).

    Google Scholar 

  8. 8

    Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    CAS  Article  PubMed  Google Scholar 

  9. 9

    Schultz, W., Dayan, P. & Montague, R.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    CAS  Article  PubMed  Google Scholar 

  10. 10

    Berns, G.S. & Sejnowski, T.J. A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 10, 108–121 (1998).

    CAS  Article  PubMed  Google Scholar 

  11. 11

    Brown, J., Bullock, D. & Grossberg, S. How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci. 19, 10502–10511 (1999).

    CAS  Article  PubMed  Google Scholar 

  12. 12

    Contreras-Vidal, J.L. & Schultz, W. A predictive reinforcement model of dopamine neurons for learning approach behavior. J. Comput. Neurosci. 6, 191–214 (1999).

    CAS  Article  PubMed  Google Scholar 

  13. 13

    Suri, R.E. & Schultz, W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890 (1999).

    CAS  Article  PubMed  Google Scholar 

  14. 14

    Daw, N.D., Courville, A.C. & Touretzky, D.S. Representation and timing in theories of the dopamine system. Neural Comput. 18, 1637–1677 (2006).

    Article  PubMed  Google Scholar 

  15. 15

    Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).

    CAS  Article  PubMed  Google Scholar 

  16. 16

    Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).

    CAS  Article  PubMed  Google Scholar 

  17. 17

    Schultz, W. The predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).

    CAS  Article  PubMed  Google Scholar 

  18. 18

    Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).

    CAS  Article  PubMed  Google Scholar 

  19. 19

    Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).

    CAS  Article  PubMed  Google Scholar 

  20. 20

    Nakahara, H., Itho, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004).

    CAS  Article  PubMed  Google Scholar 

  21. 21

    Morris, G., Arkadir, D., Nevet, A., Vaadia, E. & Bergman, H. Coincident, but distinct, messages of midbrain dopamine and striatal tonically active neurons. Neuron 43, 133–143 (2004).

    CAS  Article  PubMed  Google Scholar 

  22. 22

    Pan, W.X., Schmidt, R., Wickens, J.R. & Hyland, B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).

    CAS  Article  PubMed  Google Scholar 

  23. 23

    Tobler, P.N., Fiorillo, C.D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).

    CAS  Article  PubMed  Google Scholar 

  24. 24

    Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  25. 25

    Rakitin, B.C. et al. Scalar expectancy theory and peak-interval timing in humans. J. Exp. Psychol. Anim. Behav. Process. 24, 15–33 (1998).

    CAS  Article  PubMed  Google Scholar 

  26. 26

    Kacelnik, A. & Brito e Abreu, F. Risky choice and Weber's Law. J. Theor. Biol. 194, 289–298 (1998).

    CAS  Article  PubMed  Google Scholar 

  27. 27

    Janssen, P. & Shadlen, M.N. A representation of the hazard rate of elapsed time in macaque area LIP. Nat. Neurosci. 8, 234–241 (2005).

    CAS  Article  PubMed  Google Scholar 

  28. 28

    Ghose, G.M. & Maunsell, J.H.R. Attentional modulation in visual cortex depends on task-timing. Nature 419, 616–620 (2002).

    CAS  Article  PubMed  Google Scholar 

  29. 29

    Bateson, M. & Kacelnik, A. Accuracy of memory for amount in the foraging starling, Sturnus vulgaris. Anim. Behav. 50, 431–443 (1995).

    Article  Google Scholar 

  30. 30

    Komura, Y. et al. Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412, 546–549 (2001).

    CAS  Article  PubMed  Google Scholar 

  31. 31

    Brody, C.D., Hernandez, A., Zanos, A. & Romo, R. Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb. Cortex 13, 1196–1207 (2003).

    Article  Google Scholar 

  32. 32

    Leon, M.I. & Shadlen, M.N. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron 38, 317–327 (2003).

    CAS  Article  PubMed  Google Scholar 

  33. 33

    Renoult, L.Roux. S. & Riehle, A. Time is a rubberband: neuronal activity in monkey motor cortex in relation to time estimation. Eur. J. Neurosci. 23, 3098–3108 (2006).

    Article  PubMed  Google Scholar 

  34. 34

    O'Reilly, R.C., Frank, M.J., Hazy, T.E. & Watz, B. PVLV: the primary value and learned value Pavlovian learning algorithm. Behav. Neurosci. 121, 31–49 (2007).

    Article  PubMed  Google Scholar 

  35. 35

    Grossberg, S. & Schmajuk, N.A. Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Netw. 2, 79–102 (1989).

    Article  Google Scholar 

  36. 36

    Medina, J.F., Nores, W.L. & Mauk, M.D. Inhibition of climbing fibers is a signal for the extinction of conditioned eyelid responses. Nature 416, 330–333 (2002).

    CAS  Article  PubMed  Google Scholar 

  37. 37

    Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57, 201–217 (1987).

    CAS  Article  PubMed  Google Scholar 

  38. 38

    Nieder, A. & Miller, E.K. Coding of cognitive magnitude: compressed scaling of numerical information in the primate prefrontal cortex. Neuron 37, 149–157 (2003).

    CAS  Article  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by grants from the Human Frontiers Science Program (C.D.F.), the Howard Hughes Medical Institute (W.T.N.), the US National Institutes of Health (EY 05603, W.T.N.), the Swiss National Science Funds (W.S.) and the Wellcome Trust (W.S.).

Author information

Affiliations

Authors

Contributions

C.D.F. conducted the experiments, analyzed the data and developed the mathematical model of dopamine responses. C.D.F. and W.S. designed the experiments. C.D.F. wrote the manuscript with feedback from W.T.N. and W.S.

Corresponding author

Correspondence to Christopher D Fiorillo.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–4, Supplementary Methods and Supplementary Results (PDF 3905 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fiorillo, C., Newsome, W. & Schultz, W. The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11, 966–973 (2008). https://doi.org/10.1038/nn.2159

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing