Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reinforcement learning can account for associative and perceptual learning on a visual-decision task

Abstract

We recently showed that improved perceptual performance on a visual motion direction–discrimination task corresponds to changes in how an unmodified sensory representation in the brain is interpreted to form a decision that guides behavior. Here we found that these changes can be accounted for using a reinforcement-learning rule to shape functional connectivity between the sensory and decision neurons. We modeled performance on the basis of the readout of simulated responses of direction-selective sensory neurons in the middle temporal area (MT) of monkey cortex. A reward prediction error guided changes in connections between these sensory neurons and the decision process, first establishing the association between motion direction and response direction, and then gradually improving perceptual sensitivity by selectively strengthening the connections from the most sensitive neurons in the sensory population. The results suggest a common, feedback-driven mechanism for some forms of associative and perceptual learning.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Figure 1: Schematic of the decision-making, reinforcement-learning model.
Figure 2: Discrimination performance of the monkeys and model with training.
Figure 3: Changes in pooling weights with training on a coarse discrimination task.
Figure 4: Changes in choice probability of neurons in the sensory representation with training.
Figure 5: Changes in the pooled responses with training.
Figure 6: Changes in pooling weights with training on a fine discrimination task.
Figure 7: Specificity of learning.

References

  1. Gibson, E.J. Perceptual learning. Annu. Rev. Psychol. 14, 29–56 (1963).

    Article  CAS  PubMed  Google Scholar 

  2. Gilbert, C.D., Sigman, M. & Crist, R.E. The neural basis of perceptual learning. Neuron 31, 681–697 (2001).

    Article  CAS  PubMed  Google Scholar 

  3. Mukai, I. et al. Activations in visual and attention-related areas predict and correlate with the degree of perceptual learning. J. Neurosci. 27, 11401–11411 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Law, C.T. & Gold, J.I. Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat. Neurosci. 11, 505–513 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  6. Schultz, W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115 (2006).

    Article  PubMed  Google Scholar 

  7. Schultz, W. Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288 (2007).

    Article  CAS  PubMed  Google Scholar 

  8. Berridge, K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl.) 191, 391–431 (2007).

    Article  CAS  Google Scholar 

  9. Redgrave, P., Gurney, K. & Reynolds, J. What is reinforced by phasic dopamine signals? Brain Res. Rev. 58, 322–339 (2008).

    Article  CAS  PubMed  Google Scholar 

  10. Bao, S., Chan, V.T. & Merzenich, M.M. Cortical remodeling induced by activity of ventral tegmental dopamine neurons. Nature 412, 79–83 (2001).

    Article  CAS  PubMed  Google Scholar 

  11. Seitz, A. & Watanabe, T. A unified model for perceptual learning. Trends Cogn. Sci. 9, 329–334 (2005).

    Article  PubMed  Google Scholar 

  12. Roitman, J.D. & Shadlen, M.N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22, 9475–9489 (2002).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Hanks, T.D., Ditterich, J. & Shadlen, M.N. Microstimulation of macaque area LIP affects decision-making in a motion discrimination task. Nat. Neurosci. 9, 682–689 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Platt, M.L. & Glimcher, P.W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).

    Article  CAS  PubMed  Google Scholar 

  15. Andersen, R.A. & Buneo, C.A. Intentional maps in posterior parietal cortex. Annu. Rev. Neurosci. 25, 189–220 (2002).

    Article  CAS  PubMed  Google Scholar 

  16. Colby, C.L. & Goldberg, M.E. Space and attention in parietal cortex. Annu. Rev. Neurosci. 22, 319–349 (1999).

    Article  CAS  PubMed  Google Scholar 

  17. Gold, J.I. & Shadlen, M.N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).

    Article  CAS  PubMed  Google Scholar 

  18. Salzman, C.D., Britten, K.H. & Newsome, W.T. Cortical microstimulation influences perceptual judgments of motion direction. Nature 346, 174–177 (1990).

    Article  CAS  PubMed  Google Scholar 

  19. Britten, K.H., Shadlen, M.N., Newsome, W.T. & Movshon, J.A. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745–4765 (1992).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Williams, R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992).

    Google Scholar 

  21. Shadlen, M.N., Britten, K.H., Newsome, W.T. & Movshon, J.A. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J. Neurosci. 16, 1486–1510 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Dayan, P. & Daw, N.D. Decision theory, reinforcement learning and the brain. Cogn. Affect. Behav. Neurosci. 8, 429–453 (2008).

    Article  PubMed  Google Scholar 

  23. Zohary, E., Shadlen, M.N. & Newsome, W.T. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature 370, 140–143 (1994).

    Article  CAS  PubMed  Google Scholar 

  24. Bair, W., Zohary, E. & Newsome, W.T. Correlated firing in macaque visual area MT: time scales and relationship to behavior. J. Neurosci. 21, 1676–1697 (2001).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Barto, A.G. & Anadan, P. Pattern recognizing stochastic learning automata. IEEE Trans. Syst. Man Cybern. 28, 360–374 (1985).

    Article  Google Scholar 

  26. Loewenstein, Y. & Seung, H.S. Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. USA 103, 15224–15229 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Mazurek, M.E., Roitman, J.D., Ditterich, J. & Shadlen, M.N. A role for neural integrators in perceptual decision making. Cereb. Cortex 13, 1257–1269 (2003).

    Article  PubMed  Google Scholar 

  28. Kepecs, A., Uchida, N., Zariwala, H.A. & Mainen, Z.F. Neural correlates, computation and behavioral impact of decision confidence. Nature 455, 227–231 (2008).

    Article  CAS  PubMed  Google Scholar 

  29. Gold, J.I. & Shadlen, M.N. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10–16 (2001).

    Article  PubMed  Google Scholar 

  30. Kiani, R., Hanks, T.D. & Shadlen, M.N. Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment. J. Neurosci. 28, 3017–3029 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Spiegelhalter, D.J. & Lauritzen, S.L. Sequential updating of conditional probabilities on directed graphical structures. Networks 20, 579–605 (1990).

    Article  Google Scholar 

  32. Royer, S. & Pare, D. Conservation of total synaptic weight through balanced synaptic depression and potentiation. Nature 422, 518–522 (2003).

    Article  CAS  PubMed  Google Scholar 

  33. Connolly, P.M., Bennur, S. & Gold, J.I. Correlates of perceptual learning in an oculomotor decision variable. J. Neurosci. 29, 2136–2150 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Purushothaman, G. & Bradley, D.C. Neural population code for fine perceptual decisions in area MT. Nat. Neurosci. 8, 99–106 (2005).

    Article  CAS  PubMed  Google Scholar 

  35. Britten, K.H., Newsome, W.T., Shadlen, M.N., Celebrini, S. & Movshon, J.A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 13, 87–100 (1996).

    Article  CAS  PubMed  Google Scholar 

  36. Sompolinsky, H., Yoon, H., Kang, K. & Shamir, M. Population coding in neuronal systems with correlated noise. Phys. Rev. E 64, 051904 (2001).

    Article  CAS  Google Scholar 

  37. Smith, M.A. & Kohn, A. Spatial and temporal scales of neuronal correlation in primary visual cortex. J. Neurosci. 28, 12591–12603 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Gold, J.I. & Shadlen, M.N. The influence of behavioral context on the representation of a perceptual decision in developing oculomotor commands. J. Neurosci. 23, 632–651 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Jazayeri, M. & Movshon, J.A. A new perceptual illusion reveals mechanisms of sensory decoding. Nature 446, 912–915 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Hol, K. & Treue, S. Different populations of neurons contribute to the detection and discrimination of visual motion. Vision Res. 41, 685–689 (2001).

    Article  CAS  PubMed  Google Scholar 

  41. Ball, K. & Sekuler, R. Direction-specific improvement in motion discrimination. Vision Res. 27, 953–965 (1987).

    Article  CAS  PubMed  Google Scholar 

  42. Fahle, M. Perceptual learning: specificity versus generalization. Curr. Opin. Neurobiol. 15, 154–160 (2005).

    Article  CAS  PubMed  Google Scholar 

  43. Ahissar, M. & Hochstein, S. Task difficulty and the specificity of perceptual learning. Nature 387, 401–406 (1997).

    Article  CAS  PubMed  Google Scholar 

  44. Newsome, W.T. & Pare, E.B. A selective impairment of motion perception following lesions of the middle temporal visual area (MT). J. Neurosci. 8, 2201–2211 (1988).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Celebrini, S. & Newsome, W.T. Neuronal and psychophysical sensitivity to motion signals in extrastriate area MST of the macaque monkey. J. Neurosci. 14, 4109–4124 (1994).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Shamir, M. & Sompolinsky, H. Correlation codes in neuronal networks. in Advances in Neural Information Processing Systems (eds. Dietterich, T.G., Becker, S. & Ghahramani, Z.) (MIT Press, Cambridge, Massachusetts, 2002).

    Google Scholar 

  47. Yoon, H. & Sompolinsky, H. The effect of correlations on the fisher information of population codes. in Advances in Neural Information Processing Systems (eds. Kearns, M.S., Solla, S.A. & Cohn, D.A.) 167–173 (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  48. van Veen, V. & Carter, C.S. Error detection, correction, and prevention in the brain: a brief review of data and theories. Clin. EEG Neurosci. 37, 330–335 (2006).

    Article  CAS  PubMed  Google Scholar 

  49. Dosher, B.A. & Lu, Z.L. Mechanisms of perceptual learning. Vision Res. 39, 3197–3221 (1999).

    Article  CAS  PubMed  Google Scholar 

  50. Britten, K.H., Shadlen, M.N., Newsome, W.T. & Movshon, J.A. Responses of neurons in macaque MT to stochastic motion signals. Vis. Neurosci. 10, 1157–1169 (1993).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We thank L. Ding, M. Nassar, B. Heasley, R. Kalwani, C.-L. Teng, S. Bennur and M. Todd for helpful comments on this manuscript and J. Zweigle for expert technical assistance. This work was supported by the Sloan Foundation, the McKnight Foundation, the Burroughs-Wellcome Fund, and US National Institutes of Health grants R01-EY015260 and T32-EY007035.

Author information

Authors and Affiliations

Authors

Contributions

C.-T.L. and J.I.G. planned the study and wrote the manuscript together. C.-T.L. implemented the model.

Corresponding author

Correspondence to Joshua I Gold.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–9, Supplementary Table 1 and Supplementary Methods (PDF 1122 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Law, CT., Gold, J. Reinforcement learning can account for associative and perceptual learning on a visual-decision task. Nat Neurosci 12, 655–663 (2009). https://doi.org/10.1038/nn.2304

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.2304

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing