Abstract
Humans and animals have the ability to predict future events, which they cultivate by continuously searching their environment for sources of predictive information. However, little is known about the neural systems that motivate this behavior. We hypothesized that information-seeking is assigned value by the same circuits that support reward-seeking, such that neural signals encoding reward prediction errors (RPEs) include analogous information prediction errors (IPEs). To test this, we recorded from neurons in the lateral habenula, a nucleus that encodes RPEs, while monkeys chose between cues that provided different chances to view information about upcoming rewards. We found that a subpopulation of lateral habenula neurons transmitted signals resembling IPEs, responding when reward information was unexpectedly cued, delivered or denied. These signals evaluated information sources reliably, even when the monkey's decisions did not. These neurons could provide a common instructive signal for reward-seeking and information-seeking behavior.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Change history
22 September 2011
In the version of this article initially published, the colored circles and horizontal bars above the x axis in the top panels of Figure 8, which indicate the median and central 90% range of each distribution, were misaligned. The error has been corrected in the HTML and PDF versions of the article.
References
Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Rescorla, R.A. & Wagner, A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton Century Crofts, New York, 1972).
Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Seo, H. & Lee, D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 27, 8366–8377 (2007).
Matsumoto, M., Matsumoto, K., Abe, H. & Tanaka, K. Medial prefrontal cell activity signaling prediction errors of action values. Nat. Neurosci. 10, 647–656 (2007).
Kim, H., Sul, J.H., Huh, N., Lee, D. & Jung, M.W. Role of striatum in updating values of chosen actions. J. Neurosci. 29, 14701–14712 (2009).
Oyama, K., Hernadi, I., Iijima, T. & Tsutsui, K. Reward prediction error coding in dorsal striatal neurons. J. Neurosci. 30, 11447–11457 (2010).
Hong, S. & Hikosaka, O. The globus pallidus sends reward-related signals to the lateral habenula. Neuron 60, 720–729 (2008).
Matsumoto, M. & Hikosaka, O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111–1115 (2007).
Matsumoto, M. & Hikosaka, O. Representation of negative motivational value in the primate lateral habenula. Nat. Neurosci. 12, 77–84 (2009).
Holroyd, C.B. & Coles, M.G. The neural basis of human error processing: reinforcement learning, dopamine and the error-related negativity. Psychol. Rev. 109, 679–709 (2002).
O'Doherty, J.P., Dayan, P., Friston, K., Critchley, H. & Dolan, R.J. Temporal difference models and reward-related learning in the human brain. Neuron 38, 329–337 (2003).
McClure, S.M., Berns, G.S. & Montague, P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339–346 (2003).
Salas, R., Baldwin, P., de Biasi, M. & Montague, P.R. BOLD responses to negative reward prediction errors in human habenula. Front. Hum. Neurosci. 4, 36 (2010).
Chew, S.H. & Ho, J.L. Hope: an empirical study of attitude toward the timing of uncertainty resolution. J. Risk Uncertain. 8, 267–288 (1994).
Eliaz, K. & Schotter, A. Experimental testing of intrinsic preferences for noninstrumental information. Am. Econ. Rev. 97, 166–169 (2007).
Luhmann, C.C., Chun, M.M., Yi, D.-J., Lee, D. & Wang, X.-J. Neural dissociation of delay and uncertainty in inter-temporal choice. J. Neurosci. 28, 14459–14466 (2008).
Prokasy, W.F. Jr. The acquisition of observing responses in the absence of differential external reinforcement. J. Comp. Physiol. Psychol. 49, 131–134 (1956).
Fantino, E. Conditioned reinforcement: choice and information. in Handbook of Operant Behavior (eds. Honig, W.K. & Staddon, J.E.R.) (Prentice Hall, Englewood Cliffs, New Jersey, 1977).
Dinsmoor, J.A. Observing and conditioned reinforcement. Behav. Brain Sci. 6, 693–728 (1983).
Daly, H.B. Preference for unpredictability is reversed when unpredictable nonreward is aversive: procedures, data, and theories of appetitive observing response acquisition. in Learning and Memory: The Behavioral and Biological Substrates (eds. Gormezano, I. & Wasserman, E.A.) 81–104 (L.E. Associates, 1992).
Bromberg-Martin, E.S. & Hikosaka, O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63, 119–126 (2009).
Ward, E. Acquisition and extinction of the observing response as a function of stimulus predictive validity. Psychon. Sci. 24, 139–141 (1971).
Hayden, B.Y., Heilbronner, S.R. & Platt, M.L. Ambiguity aversion in rhesus macaques. Front. Neurosci. 4, 166 (2010).
Kreps, D.M. & Porteus, E.L. Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46, 185–200 (1978).
Wyckoff, L.B. Jr. The role of observing responses in discrimination learning. Psychol. Rev. 59, 431–442 (1952).
Hikosaka, O. The habenula: from stress evasion to value-based decision-making. Nat. Rev. Neurosci. 11, 503–513 (2010).
Christoph, G.R., Leonzio, R.J. & Wilcox, K.S. Stimulation of the lateral habenula inhibits dopamine-containing neurons in the substantia nigra and ventral tegmental area of the rat. J. Neurosci. 6, 613–619 (1986).
Ji, H. & Shepard, P.D. Lateral habenula stimulation inhibits rat midbrain dopamine neurons through a GABA(A) receptor–mediated mechanism. J. Neurosci. 27, 6923–6930 (2007).
Shumake, J., Ilango, A., Scheich, H., Wetzel, W. & Ohl, F.W. Differential neuromodulation of acquisition and retrieval of avoidance learning by the lateral habenula and ventral tegmental area. J. Neurosci. 30, 5876–5883 (2010).
Friedman, A. et al. Electrical stimulation of the lateral habenula produces an inhibitory effect on sucrose self-administration. Neuropharmacology 60, 381–387 (2010).
Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).
Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
Sutherland, R.J. & Nakajima, S. Self-stimulation of the habenular complex in the rat. J. Comp. Physiol. Psychol. 95, 781–791 (1981).
Lecourtier, L., Defrancesco, A. & Moghaddam, B. Differential tonic influence of lateral habenula on prefrontal cortex and nucleus accumbens dopamine release. Eur. J. Neurosci. 27, 1755–1762 (2008).
Ikemoto, S. Brain reward circuitry beyond the mesolimbic dopamine system: a neurobiological theory. Neurosci. Biobehav. Rev. 35, 129–150 (2010).
Matsumoto, M. & Hikosaka, O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459, 837–841 (2009).
Joshua, M., Adler, A. & Bergman, H. The dynamics of dopamine in control of motor behavior. Curr. Opin. Neurobiol. 19, 615–620 (2009).
Bromberg-Martin, E.S., Matsumoto, M. & Hikosaka, O. Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815–834 (2010).
Caplin, A. & Leahy, J. Psychological expected utility theory and anticipatory feelings. Q. J. Econ. 116, 55–79 (2001).
Butko, N.J. & Movellan, J.R. Infomax control of eye movements. IEEE Trans. Auto. Ment. Dev. 2, 91–107 (2010).
Wyckoff, L.B. Jr. Toward a quantitative theory of secondary reinforcement. Psychol. Rev. 66, 68–78 (1959).
Beierholm, U.R. & Dayan, P. Pavlovian-instrumental interaction in 'observing behavior'. PLOS Comput. Biol. 6, 9 (2010).
Bromberg-Martin, E.S., Matsumoto, M., Nakahara, H. & Hikosaka, O. Multiple timescales of memory in lateral habenula and dopamine neurons. Neuron 67, 499–510 (2010).
Nakamura, K., Matsumoto, M. & Hikosaka, O. Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus. J. Neurosci. 28, 5331–5343 (2008).
Ranade, S.P. & Mainen, Z.F. Transient firing of dorsal raphe neurons encodes diverse and specific sensory, motor, and reward events. J. Neurophysiol. 102, 3026–3037 (2009).
Redgrave, P. & Gurney, K. The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967–975 (2006).
Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).
Roesch, M.R., Calu, D.J. & Schoenbaum, G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615–1624 (2007).
Houk, J.C., Adams, J.L. & Barto, A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. in Models of Information Processing in the Basal Ganglia (eds. Houk, J.C., Davis, J.L. & Beiser, D.G.) 249–274 (MIT Press, Cambridge, Massachusetts, 1995).
Acknowledgements
We thank M. Matsumoto, S. Hong, I. Monosov, M. Yasuda, S. Yamamoto, Y. Tachibana, H. Kim and D. Lee for valuable discussions. This work was supported by the intramural research program at the National Eye Institute.
Author information
Authors and Affiliations
Contributions
E.S.B.-M. designed and performed the experiments and analyzed the data. O.H. supported these processes. E.S.B.-M. and O.H. discussed the results and wrote the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–11 (PDF 1901 kb)
Rights and permissions
About this article
Cite this article
Bromberg-Martin, E., Hikosaka, O. Lateral habenula neurons signal errors in the prediction of reward information. Nat Neurosci 14, 1209–1216 (2011). https://doi.org/10.1038/nn.2902
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2902