Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Bioinspired learning

Moving beyond reward prediction errors

Nature Machine Intelligence volume 1, pages 204–205 (2019)Cite this article

Subjects

Classic theories of reinforcement learning and neuromodulation rely on reward prediction errors. A new machine learning technique relies on neuromodulatory signals that are optimized for specific tasks, which may lead to better AI and better explanations of neuroscience data.

One of the greatest success stories in the interaction between artificial intelligence (AI) and neuroscience is reinforcement learning1,2 with a decade-long history of sharing ideas back and forth, of which one of the latest was put forth by Thomas Miconi et al.3 at the 2019 International Conference on Learning Representations. Reinforcement learning was originally inspired by computational models of animal learning that had been developed by neuroscientists4,5. These models were founded on the idea that an agent (that is, an animal) should learn an estimate of the value of different states and actions in the world, based on rewards and punishments, then adapt its behaviour in order to maximize this estimated value5. This was not the perspective taken in classical AI research, but some researchers did adopt it4, and this was key to the development of reinforcement learning2. It has turned out to be an immensely beneficial approach. Indeed, it has helped to produce some of the most noteworthy successes in modern deep learning6,7.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Standard models of dopamine versus backpropamine.

References

  1. Hassabis, D., Kumaran, D., Summerfield, C. & Botvinick, M. Neuron 95, 245–258 (2017).

    Article  Google Scholar 

  2. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 1998).

  3. Miconi, T., Rawal, A., Clune, J. & Stanley, K. O. in Int. Conf. Learning Representations (ICLR, 2019).

  4. Sutton, R. S. & Barto, A. G. Psychol. Rev. 88, 135–170 (1981).

    Article  Google Scholar 

  5. Rescorla, R. A. & Wagner, A. R. in Classical Conditioning II: Current Research and Theory (eds Black, A. H., & Prokasy, W. F.) 64–99 (Appleton-Century-Crofts, 1972).

  6. Silver, D. et al. Nature 529, 484–489 (2016).

    Article  Google Scholar 

  7. Mnih, V. et al. Nature 518, 529–533 (2015).

    Article  Google Scholar 

  8. Iversen, S. D. & Iversen, L. L. Trends Neurosci. 30, 188–193 (2007).

    Article  Google Scholar 

  9. Schultz, W., Dayan, P. & Montague, P. R. Science 275, 1593–1599 (1997).

    Article  Google Scholar 

  10. Brzosko, Z., Schultz, W. & Paulsen, O. eLife 4, e09685 (2015).

    Article  Google Scholar 

  11. Roelfsema, P. R. & Holtmaat, A. Nat. Rev. Neurosci. 19, 166–180 (2018).

    Article  Google Scholar 

  12. Frémaux, N. & Gerstner, W. Front. Neural Circuits 9, 85 (2016).

    Article  Google Scholar 

  13. Sharpe, M. J. et al. Nat. Neurosci. 20, 735–742 (2017).

    Article  Google Scholar 

  14. Takahashi, Y. K. et al. Neuron 95, 1395–1405 (2017).

    Article  Google Scholar 

  15. Coddington, L. T. & Dudman, J. T. Nat. Neurosci. 21, 1563–1573 (2018).

    Article  Google Scholar 

  16. Andrychowicz, M. et al. in 30th Conf. Neural Information Processing Systems 3981–3989 (NIPS, 2016).

Download references

Author information

Authors and Affiliations

  1. Department of Biological Sciences, University of Toronto Scarborough, Toronto, Ontario, Canada

    Blake A. Richards

  2. Department of Cell and Systems Biology, University of Toronto, Toronto, Ontario, Canada

    Blake A. Richards

  3. Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada

    Blake A. Richards

  4. Canadian Institute for Advanced Research, Toronto, Ontario, Canada

    Blake A. Richards

Authors
  1. Blake A. Richards
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Blake A. Richards.

Ethics declarations

Competing interests

The author declares no competing interests.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Richards, B.A. Moving beyond reward prediction errors. Nat Mach Intell 1, 204–205 (2019). https://doi.org/10.1038/s42256-019-0053-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-019-0053-0

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing