Midbrain dopamine neurons are activated when reward is greater than predicted, and this error signal could teach target neurons both the value of reward and when it will occur. We used the dopamine error signal to measure how the expectation of reward was distributed over time. Animals were trained with fixed-duration intervals of 1–16 s between conditioned stimulus onset and reward. In contrast to the weak responses that have been observed after short intervals (1–2 s), activations to reward increased steeply and linearly with the logarithm of the interval. Results with varied stimulus-reward intervals suggest that the neural expectation was substantial after just half an interval had elapsed. Thus, the neural expectation of reward in these experiments was not highly precise and the precision declined sharply with interval duration. The neural precision of expectation appeared to be at least qualitatively similar to the precision of anticipatory licking behavior.
Subscribe to Journal
Get full journal access for 1 year
only $4.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Buhusi, C.V. & Meck, W.H. What makes us tick? Functional and neural mechanisms of interval timing. Nat. Rev. Neurosci. 6, 755–765 (2005).
Mauk, M.D. & Buonomano, D.V. The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004).
Rao, S.M., Mayer, A.R. & Harrington, D.L. The evolution of brain activation during temporal processing. Nat. Neurosci. 4, 317–323 (2001).
Gibbon, J., Malapani, C., Dale, C.L. & Gallistel, C.R. Toward a neurobiology of temporal cognition: advances and challenges. Curr. Opin. Neurobiol. 7, 170–184 (1997).
Meck, W.H. Neuropharmacology of timing and time perception. Brain Res. Cogn. Brain Res. 3, 227–242 (1996).
Matell, M.S. & Meck, W.H. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Brain Res. Cogn. Brain Res. 21, 139–170 (2004).
Houk, J., Adams, J. & Barto, A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia (eds Houk, J., Davis, J. & Beiser, D.) 249–270 (MIT Press, Cambridge, Massachusetts, 1995).
Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
Schultz, W., Dayan, P. & Montague, R.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Berns, G.S. & Sejnowski, T.J. A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 10, 108–121 (1998).
Brown, J., Bullock, D. & Grossberg, S. How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci. 19, 10502–10511 (1999).
Contreras-Vidal, J.L. & Schultz, W. A predictive reinforcement model of dopamine neurons for learning approach behavior. J. Comput. Neurosci. 6, 191–214 (1999).
Suri, R.E. & Schultz, W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890 (1999).
Daw, N.D., Courville, A.C. & Touretzky, D.S. Representation and timing in theories of the dopamine system. Neural Comput. 18, 1637–1677 (2006).
Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).
Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).
Schultz, W. The predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).
Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).
Nakahara, H., Itho, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004).
Morris, G., Arkadir, D., Nevet, A., Vaadia, E. & Bergman, H. Coincident, but distinct, messages of midbrain dopamine and striatal tonically active neurons. Neuron 43, 133–143 (2004).
Pan, W.X., Schmidt, R., Wickens, J.R. & Hyland, B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).
Tobler, P.N., Fiorillo, C.D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).
Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Rakitin, B.C. et al. Scalar expectancy theory and peak-interval timing in humans. J. Exp. Psychol. Anim. Behav. Process. 24, 15–33 (1998).
Kacelnik, A. & Brito e Abreu, F. Risky choice and Weber's Law. J. Theor. Biol. 194, 289–298 (1998).
Janssen, P. & Shadlen, M.N. A representation of the hazard rate of elapsed time in macaque area LIP. Nat. Neurosci. 8, 234–241 (2005).
Ghose, G.M. & Maunsell, J.H.R. Attentional modulation in visual cortex depends on task-timing. Nature 419, 616–620 (2002).
Bateson, M. & Kacelnik, A. Accuracy of memory for amount in the foraging starling, Sturnus vulgaris. Anim. Behav. 50, 431–443 (1995).
Komura, Y. et al. Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412, 546–549 (2001).
Brody, C.D., Hernandez, A., Zanos, A. & Romo, R. Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb. Cortex 13, 1196–1207 (2003).
Leon, M.I. & Shadlen, M.N. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron 38, 317–327 (2003).
Renoult, L.Roux. S. & Riehle, A. Time is a rubberband: neuronal activity in monkey motor cortex in relation to time estimation. Eur. J. Neurosci. 23, 3098–3108 (2006).
O'Reilly, R.C., Frank, M.J., Hazy, T.E. & Watz, B. PVLV: the primary value and learned value Pavlovian learning algorithm. Behav. Neurosci. 121, 31–49 (2007).
Grossberg, S. & Schmajuk, N.A. Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Netw. 2, 79–102 (1989).
Medina, J.F., Nores, W.L. & Mauk, M.D. Inhibition of climbing fibers is a signal for the extinction of conditioned eyelid responses. Nature 416, 330–333 (2002).
Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57, 201–217 (1987).
Nieder, A. & Miller, E.K. Coding of cognitive magnitude: compressed scaling of numerical information in the primate prefrontal cortex. Neuron 37, 149–157 (2003).
This work was supported by grants from the Human Frontiers Science Program (C.D.F.), the Howard Hughes Medical Institute (W.T.N.), the US National Institutes of Health (EY 05603, W.T.N.), the Swiss National Science Funds (W.S.) and the Wellcome Trust (W.S.).
About this article
Cite this article
Fiorillo, C., Newsome, W. & Schultz, W. The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11, 966–973 (2008). https://doi.org/10.1038/nn.2159
Differential reinforcement encoding along the hippocampal long axis helps resolve the explore–exploit dilemma
Nature Communications (2020)
Scientific Reports (2020)
Random access to palatable food stimulates similar addiction-like responses as a fixed schedule, but only a fixed schedule elicits anticipatory activation
Scientific Reports (2019)
Nature Communications (2019)