Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Deep reinforcement learning to study combinatorial expansion of a behavior repertoire


Despite rich behavioral evidence, it is unclear how the brain expands its behavior repertoire. By building theoretical models with a deep reinforcement learning algorithm, I show that the brain composes a behavior to solve a novel task by combining previously acquired skills and augmenting their variability.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Get just this article for as long as you need it


Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Behavior composition via transfer of combinatorial neural representations of action values.


  1. Kohler, W. The Mentality of Apes (trans Winter, E.) (Liveright, 1976). This book presents the intelligent behavior repertoires of chimpanzees.

  2. Epstein, R., Kirshnit, C. E., Lanza, R. P. & Rubin, L. C. ‘Insight’ in the pigeon: antecedents and determinants of an intelligent performance. Nature 308, 61–62 (1984). This paper reports the sudden acquisition of a problem solution in pigeons by combining pre-learned motor skills.

    Article  CAS  Google Scholar 

  3. Arthur, W. B. The Nature of Technology: What It is and How It Evolves (Free Press, 2009). This book presents the concept of combinatorial evolution of technology.

  4. Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J. & Kurth-Nelson, Z. Deep reinforcement learning and its neuroscientific implications. Neuron 107, 603–616 (2020). This paper emphasizes the benefits of collaborative efforts between deep reinforcement learning and neuroscience.

    Article  CAS  Google Scholar 

  5. Haarnoja, T. et al. Composable deep reinforcement learning for robotic manipulation. 2018 IEEE International Conf. on Robotics and Automation (ICRA) 6244–6251 (2018). This paper reports a deep reinforcement learning-based theoretical framework for combinatorial policy composition in machines.

Download references

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This is a summary of: Makino, H. Arithmetic value representation for hierarchical behavior composition. Nat. Neurosci. (2022).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Deep reinforcement learning to study combinatorial expansion of a behavior repertoire. Nat Neurosci 26, 2–3 (2023).

Download citation

  • Published:

  • Issue Date:

  • DOI:


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing