Article | Published:

Reinforcement learning can account for associative and perceptual learning on a visual-decision task

Nature Neuroscience volume 12, pages 655663 (2009) | Download Citation

Subjects

Abstract

We recently showed that improved perceptual performance on a visual motion direction–discrimination task corresponds to changes in how an unmodified sensory representation in the brain is interpreted to form a decision that guides behavior. Here we found that these changes can be accounted for using a reinforcement-learning rule to shape functional connectivity between the sensory and decision neurons. We modeled performance on the basis of the readout of simulated responses of direction-selective sensory neurons in the middle temporal area (MT) of monkey cortex. A reward prediction error guided changes in connections between these sensory neurons and the decision process, first establishing the association between motion direction and response direction, and then gradually improving perceptual sensitivity by selectively strengthening the connections from the most sensitive neurons in the sensory population. The results suggest a common, feedback-driven mechanism for some forms of associative and perceptual learning.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Perceptual learning. Annu. Rev. Psychol. 14, 29–56 (1963).

  2. 2.

    , & The neural basis of perceptual learning. Neuron 31, 681–697 (2001).

  3. 3.

    et al. Activations in visual and attention-related areas predict and correlate with the degree of perceptual learning. J. Neurosci. 27, 11401–11411 (2007).

  4. 4.

    & Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat. Neurosci. 11, 505–513 (2008).

  5. 5.

    & Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).

  6. 6.

    Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115 (2006).

  7. 7.

    Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288 (2007).

  8. 8.

    The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl.) 191, 391–431 (2007).

  9. 9.

    , & What is reinforced by phasic dopamine signals? Brain Res. Rev. 58, 322–339 (2008).

  10. 10.

    , & Cortical remodeling induced by activity of ventral tegmental dopamine neurons. Nature 412, 79–83 (2001).

  11. 11.

    & A unified model for perceptual learning. Trends Cogn. Sci. 9, 329–334 (2005).

  12. 12.

    & Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22, 9475–9489 (2002).

  13. 13.

    , & Microstimulation of macaque area LIP affects decision-making in a motion discrimination task. Nat. Neurosci. 9, 682–689 (2006).

  14. 14.

    & Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).

  15. 15.

    & Intentional maps in posterior parietal cortex. Annu. Rev. Neurosci. 25, 189–220 (2002).

  16. 16.

    & Space and attention in parietal cortex. Annu. Rev. Neurosci. 22, 319–349 (1999).

  17. 17.

    & The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).

  18. 18.

    , & Cortical microstimulation influences perceptual judgments of motion direction. Nature 346, 174–177 (1990).

  19. 19.

    , , & The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745–4765 (1992).

  20. 20.

    Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992).

  21. 21.

    , , & A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J. Neurosci. 16, 1486–1510 (1996).

  22. 22.

    & Decision theory, reinforcement learning and the brain. Cogn. Affect. Behav. Neurosci. 8, 429–453 (2008).

  23. 23.

    , & Correlated neuronal discharge rate and its implications for psychophysical performance. Nature 370, 140–143 (1994).

  24. 24.

    , & Correlated firing in macaque visual area MT: time scales and relationship to behavior. J. Neurosci. 21, 1676–1697 (2001).

  25. 25.

    & Pattern recognizing stochastic learning automata. IEEE Trans. Syst. Man Cybern. 28, 360–374 (1985).

  26. 26.

    & Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. USA 103, 15224–15229 (2006).

  27. 27.

    , , & A role for neural integrators in perceptual decision making. Cereb. Cortex 13, 1257–1269 (2003).

  28. 28.

    , , & Neural correlates, computation and behavioral impact of decision confidence. Nature 455, 227–231 (2008).

  29. 29.

    & Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10–16 (2001).

  30. 30.

    , & Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment. J. Neurosci. 28, 3017–3029 (2008).

  31. 31.

    & Sequential updating of conditional probabilities on directed graphical structures. Networks 20, 579–605 (1990).

  32. 32.

    & Conservation of total synaptic weight through balanced synaptic depression and potentiation. Nature 422, 518–522 (2003).

  33. 33.

    , & Correlates of perceptual learning in an oculomotor decision variable. J. Neurosci. 29, 2136–2150 (2009).

  34. 34.

    & Neural population code for fine perceptual decisions in area MT. Nat. Neurosci. 8, 99–106 (2005).

  35. 35.

    , , , & A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 13, 87–100 (1996).

  36. 36.

    , , & Population coding in neuronal systems with correlated noise. Phys. Rev. E 64, 051904 (2001).

  37. 37.

    & Spatial and temporal scales of neuronal correlation in primary visual cortex. J. Neurosci. 28, 12591–12603 (2008).

  38. 38.

    & The influence of behavioral context on the representation of a perceptual decision in developing oculomotor commands. J. Neurosci. 23, 632–651 (2003).

  39. 39.

    & A new perceptual illusion reveals mechanisms of sensory decoding. Nature 446, 912–915 (2007).

  40. 40.

    & Different populations of neurons contribute to the detection and discrimination of visual motion. Vision Res. 41, 685–689 (2001).

  41. 41.

    & Direction-specific improvement in motion discrimination. Vision Res. 27, 953–965 (1987).

  42. 42.

    Perceptual learning: specificity versus generalization. Curr. Opin. Neurobiol. 15, 154–160 (2005).

  43. 43.

    & Task difficulty and the specificity of perceptual learning. Nature 387, 401–406 (1997).

  44. 44.

    & A selective impairment of motion perception following lesions of the middle temporal visual area (MT). J. Neurosci. 8, 2201–2211 (1988).

  45. 45.

    & Neuronal and psychophysical sensitivity to motion signals in extrastriate area MST of the macaque monkey. J. Neurosci. 14, 4109–4124 (1994).

  46. 46.

    & Correlation codes in neuronal networks. in Advances in Neural Information Processing Systems (eds. Dietterich, T.G., Becker, S. & Ghahramani, Z.) (MIT Press, Cambridge, Massachusetts, 2002).

  47. 47.

    & The effect of correlations on the fisher information of population codes. in Advances in Neural Information Processing Systems (eds. Kearns, M.S., Solla, S.A. & Cohn, D.A.) 167–173 (MIT Press, Cambridge, Massachusetts, 1998).

  48. 48.

    & Error detection, correction, and prevention in the brain: a brief review of data and theories. Clin. EEG Neurosci. 37, 330–335 (2006).

  49. 49.

    & Mechanisms of perceptual learning. Vision Res. 39, 3197–3221 (1999).

  50. 50.

    , , & Responses of neurons in macaque MT to stochastic motion signals. Vis. Neurosci. 10, 1157–1169 (1993).

Download references

Acknowledgements

We thank L. Ding, M. Nassar, B. Heasley, R. Kalwani, C.-L. Teng, S. Bennur and M. Todd for helpful comments on this manuscript and J. Zweigle for expert technical assistance. This work was supported by the Sloan Foundation, the McKnight Foundation, the Burroughs-Wellcome Fund, and US National Institutes of Health grants R01-EY015260 and T32-EY007035.

Author information

Affiliations

  1. Department of Neuroscience, University of Pennsylvania, Philadelphia, Pennsylvania, USA.

    • Chi-Tat Law
    •  & Joshua I Gold

Authors

  1. Search for Chi-Tat Law in:

  2. Search for Joshua I Gold in:

Contributions

C.-T.L. and J.I.G. planned the study and wrote the manuscript together. C.-T.L. implemented the model.

Corresponding author

Correspondence to Joshua I Gold.

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–9, Supplementary Table 1 and Supplementary Methods

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn.2304