Review Article | Published:

Reinforcement learning improves behaviour from evaluative feedback

Nature volume 521, pages 445451 (28 May 2015) | Download Citation

Abstract

Reinforcement learning is a branch of machine learning concerned with using experience gained through interacting with the world and evaluative feedback to improve a system's ability to make behavioural decisions. It has been called the artificial intelligence problem in a microcosm because learning algorithms must act autonomously to perform well and achieve their goals. Partly driven by the increasing availability of rich data, recent years have seen exciting advances in the theory and practice of reinforcement learning, including developments in fundamental technical areas such as generalization, planning, exploration and empirical methodology, leading to increasing applicability to real-life problems.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    & Reinforcement Learning: An Introduction (MIT Press, 1998). This book is the definitive reference on computational reinforcement learning.

  2. 2.

    , & Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996).

  3. 3.

    & Bandit Problems: Sequential Allocation of Experiments (Chapman and Hall, 1985).

  4. 4.

    & Rapid learning for precision oncology. Nature Rev. Clin. Onco. 11, 109–118 (2014).

  5. 5.

    , & Finite-time analysis of the multi-armed bandit problem. Mach. Learn. 47, 235–256 (2002).

  6. 6.

    Learning in Embedded Systems (MIT Press, 1993).

  7. 7.

    , , & A contextual-bandit approach to personalized news article recommendation. In Proc. 19th International World Wide Web Conference 661–670 (2010).

  8. 8.

    On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 285–294 (1933).

  9. 9.

    & Is probability matching smart? Associations between probabilistic choices and cognitive ability. Mem. Cognit. 31, 243–251 (2003).

  10. 10.

    , , & Optimistic Bayesian sampling in contextual-bandit problems. J. Mach. Learn. Res. 13, 2069–2106 (2012).

  11. 11.

    & Prior-free and prior-dependent regret bounds for Thompson sampling. In Proc. Advances in Neural Information Processing Systems 638–646 (2013).

  12. 12.

    & A tutorial on Bayesian nonparametric models. J. Math. Psychol. 56, 1–12 (2012).

  13. 13.

    Learning to predict by the method of temporal differences. Mach. Learn. 3, 9–44 (1988).

  14. 14.

    & Generalization in reinforcement learning: safely approximating the value function. In Proc. Advances in Neural Information Processing Systems 369–376 (1995).

  15. 15.

    Residual algorithms: reinforcement learning with function approximation. In Proc. 12th International Conference on Machine Learning (eds Prieditis, A. & Russell, S.) 30–37 (Morgan Kaufmann, 1995).

  16. 16.

    et al. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proc. 26th Annual International Conference on Machine Learning 993–1000 (2009).

  17. 17.

    , & A convergent O(n) temporal-difference algorithm for off-policy learning with linear function approximation. In Proc. Advances in Neural Information Processing Systems 1609–1616 (2009).

  18. 18.

    et al. Convergent temporal-difference learning with arbitrary smooth function approximation. In Proc. Advances in Neural Information Processing Systems 1204–1212 (2009).

  19. 19.

    , , & Toward off-policy learning control with function approximation. In Proc. 27th International Conference on Machine Learning 719–726 (2010).

  20. 20.

    , & Off-policy TD(λ) with a true online equivalence. In Proc. 30th Conference on Uncertainty in Artificial Intelligence 324 (2014).

  21. 21.

    & Artificial Intelligence: A Modern Approach (Prentice–Hall, 1994).

  22. 22.

    , & Deep blue. Artif. Intell. 134, 57–83 (2002).

  23. 23.

    Some studies in machine learning using the game of checkers. IBM J. Res. Develop. 3, 211–229 (1959).

  24. 24.

    TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6, 215–219 (1994). This article describes the first reinforcement-learning system to solve a truly non-trivial task.

  25. 25.

    , , , & Simulation, learning, and optimization techniques in Watson's game strategies. IBM J. Res. Develop. 56, 1–11 (2012).

  26. 26.

    & Bandit based Monte-Carlo planning. In Proc. 17th European Conference on Machine Learning 282–293 (2006). This article introduces UCT, the decision-making algorithm that revolutionized gameplay in Go.

  27. 27.

    et al. The grand challenge of computer Go: Monte Carlo tree search and extensions. Communications of the ACM 55, 106–113 (2012).

  28. 28.

    , , & Self-optimizing memory controllers: a reinforcement learning approach. In Proc. 35th International Symposium on Computer Architecture 39–50 (2008).

  29. 29.

    , , & Autonomous helicopter flight via reinforcement learning. In Proc. Advances in Neural Information Processing Systems (2003).

  30. 30.

    Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proc. 7th International Conference on Machine Learning 216–224 (Morgan Kaufmann, 1990).

  31. 31.

    & Near-optimal reinforcement learning in polynomial time. Mach. Learn. 49, 209–232 (2002). This article provides the first algorithm and analysis that shows that reinforcement-learning tasks can be solved approximately optimally with a relatively small amount of experience.

  32. 32.

    & R-MAX — a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res. 3, 213–231 (2002).

  33. 33.

    , , & Knows what it knows: a framework for self-aware learning. Mach. Learn. 82, 399–443 (2011).

  34. 34.

    Machine learning as an experimental science. Mach. Learn. 3, 5–8 (1988).

  35. 35.

    , , & The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013).

  36. 36.

    et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). This article describes the application of deep learning in a reinforcement-learning setting to address the challenging task of decision making in an arcade environment.

  37. 37.

    An experimental design for the development of adaptive treatment strategies. Stat. Med. 24, 1455–1481 (2005).

  38. 38.

    , , & Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proc. 4th ACM International Conference on Web Search and Data Mining 297–306 (2011).

  39. 39.

    et al. A novel benchmark methodology and data repository for real-life reinforcement learning. In Proc. Multidisciplinary Symposium on Reinforcement Learning, Poster (2009).

  40. 40.

    , , & Discovering multi-modal characteristics in observational clinical data. In Proc. Machine Learning for Clinical Data Analysis and Healthcare NIPS Workshop (2013).

  41. 41.

    , & Policy invariance under reward transformations: theory and application to reward shaping. In Proc. 16th International Conference on Machine Learning 278–287 (1999).

  42. 42.

    & Teachable robots: understanding human teaching behaviour to build more effective robot learners. Artif. Intell. 172, 716–737 (2008).

  43. 43.

    & Interactively shaping agents via human reinforcement: The TAMER framework. In Proc. 5th International Conference on Knowledge Capture 9–16 (2009).

  44. 44.

    et al. A strategy-aware technique for learning behaviors from discrete human feedback. In Proc. 28th Association for the Advancement of Artificial Intelligence Conference (2014).

  45. 45.

    & Algorithms for inverse reinforcement learning. In Proc. International Conference on Machine Learning 663–670 (2000).

  46. 46.

    , , & Apprenticeship learning about multiple intentions. In Proc. International Conference on Machine Learning 897–904 (2011).

  47. 47.

    , , & Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Trans. Auto. Mental Dev. 2, 70–82 (2010).

  48. 48.

    The chess machine: an example of dealing with a complex task by adaptation. In Proc. Western Joint Computer Conference 101–108 (1955).

  49. 49.

    Some methods of artificial intelligence and heuristic programming. In Proc. Symposium on the Mechanization of Thought Processes 24–27 (1958).

  50. 50.

    & Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170 (1981).

  51. 51.

    , & A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

  52. 52.

    & Reinforcement learning and the brain: the good, the bad and the ugly. Curr. Opin. Neurobiol. 18, 185–196 (2008).

  53. 53.

    Neuroscience: dopamine ramps up. Nature 500, 533–535 (2013).

  54. 54.

    Action, outcome, and value a dual-system framework for morality. Pers. Soc. Psychol. Rev. 17, 273–292 (2013).

  55. 55.

    Stochastic games. Proc. Natl Acad. Sci. USA 39, 1095–1100 (1953).

  56. 56.

    Dynamic Programming (Princeton Univ. Press, 1957).

  57. 57.

    , & Reinforcement learning in robotics: a survey. Int. J. Rob. Res. 32, 1238–1274 (2013).

  58. 58.

    & Q-learning. Mach. Learn. 8, 279–292 (1992). This article introduces the first provably correct approach to reinforcement learning for both prediction and decision making.

  59. 59.

    , & Convergence of stochastic iterative dynamic programming algorithms. In Advances in Neural Information Processing Systems 6, 703–710 (Morgan Kaufmann, 1994).

  60. 60.

    , & The adaptive k-meteorologists problem and its application to structure learning and feature selection in reinforcement learning. In Proc. 26th International Conference on Machine Learning 32–40 (2009).

Download references

Acknowledgements

The author appreciates his discussions with his colleagues that led to this synthesis of current work.

Author information

Affiliations

  1. Department of Computer Science, Brown University, Providence, Rhode Island 02912, USA.

    • Michael L. Littman

Authors

  1. Search for Michael L. Littman in:

Competing interests

The author declares no competing financial interests.

Corresponding author

Correspondence to Michael L. Littman.

Reprints and permissions information is available at www.nature.com/reprints.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nature14540

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.