Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Empowerment contributes to exploration behaviour in a creative video game

Nature Human Behaviour volume 7, pages 1481–1489 (2023)Cite this article

Subjects

Abstract

Studies of human exploration frequently cast people as serendipitously stumbling upon good options. Yet these studies may not capture the richness of exploration strategies that people exhibit in more complex environments. Here we study behaviour in a large dataset of 29,493 players of the richly structured online game ‘Little Alchemy 2’. In this game, players start with four elements, which they can combine to create up to 720 complex objects. We find that players are driven not only by external reward signals, such as an attempt to produce successful outcomes, but also by an intrinsic motivation to create objects that empower them to create even more objects. We find that this drive for empowerment is eliminated when playing a game variant that lacks recognizable semantics, indicating that people use their knowledge about the world and its possibilities to guide their exploration. Our results suggest that the drive for empowerment may be a potent source of intrinsic motivation in richly structured domains, particularly those that lack explicit reward signals.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: ‘Little Alchemy 2’.
Fig. 2: Empowerment results.
Fig. 3: Experiment setup of ‘Tiny Alchemy’ and ‘Tiny Pixels’.
Fig. 4: Importance of richly structured game semantics.

Data availability

Anonymized participant data of the experiments and model simulation data are available at https://github.com/franziskabraendle/alchemy_empowerment (ref. 32). Third party data of participants playing the original game may be shared upon reasonable request (franziska.braendle@tuebingen.mpg.de).

Code availability

The code used for all experiments, models and analyses is available at https://github.com/franziskabraendle/alchemy_empowerment (ref. 32).

References

  1. Schulz, E. & Gershman, S. J. The algorithmic architecture of exploration in the human brain. Curr. Opin. Neurobiol. 55, 7–14 (2019).

    Article  CAS  PubMed  Google Scholar 

  2. Wilson, R. C., Bonawitz, E., Costa, V. D. & Ebitz, R. B. Balancing exploration and exploitation with information and randomization. Curr. Opin. Behav. Sci. 38, 49–56 (2021).

    Article  PubMed  Google Scholar 

  3. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

  4. Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A. & Cohen, J. D. Humans use directed and random exploration to solve the explore–exploit dilemma. J. Exp. Psychol. Gen. 143, 155–164 (2014).

    Article  Google Scholar 

  5. Speekenbrink, M. & Konstantinidis, E. Uncertainty and exploration in a restless bandit problem. Top. Cogn. Sci. 7, 351–367 (2015).

    Article  PubMed  Google Scholar 

  6. Gershman, S. J. Deconstructing the human algorithms for exploration. Cognition 173, 34–42 (2018).

    Article  PubMed  Google Scholar 

  7. Gershman, S. Uncertainty and exploration. Decision 6, 277–286 (2019).

    Article  PubMed  Google Scholar 

  8. Brändle, F., Binz, M. & Schulz, E. in The Drive for Knowledge (eds Cogliati Dezza, I. et al) Ch. 7 (Cambridge Univ. Press, 2022).

  9. Chu, J. & Schulz, L. Not playing by the rules: exploratory play, rational action, and efficient search. Open Mind 7, 294–317 (2023).

  10. Gottlieb, J., Oudeyer, P.-Y., Lopes, M. & Baranes, A. Information-seeking, curiosity, and attention: computational and neural mechanisms. Trends Cogn. Sci. 17, 585–593 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Payzan-LeNestour, E. & Bossaerts, P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 7, e1001048 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Knox, W. B., Otto, A. R., Stone, P. & Love, B. C. The nature of belief-directed exploratory choice in human decision-making. Front. Psychol. https://doi.org/10.3389/fpsyg.2011.00398 (2012).

  13. Schulz, E., Wu, C. M., Ruggeri, A. & Meder, B. Searching for rewards like a child means less generalization and more directed exploration. Psychol. Sci. 30, 1561–1572 (2019).

    Article  PubMed  Google Scholar 

  14. Little Alchemy 2. Google Play https://play.google.com/store/apps/details?id=com.recloak.littlealchemy2 (2021).

  15. Jiang, M. et al. Wordcraft: an environment for benchmarking commonsense agents. Preprint at arXiv https://doi.org/10.48550/arXiv.2007.09185 (2020).

  16. Schulz, E., Franklin, N. T. & Gershman, S. J. Finding structure in multi-armed bandits. Cogn. Psychol. 119, 101261 (2020).

    Article  PubMed  Google Scholar 

  17. Schulz, E. et al. Structured, uncertainty-driven exploration in real-world consumer choice. Proc. Natl Acad. Sci. USA 116, 13903–13908 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Klyubin, A. S., Polani, D. & Nehaniv, C. L. All else being equal be empowered. In Proc. 8th European Conference on Advances in Artificial Life (eds Capcarrère, M.S. et al) 744–753 (Springer-Verlag, Berlin, 2005).

  19. Colantonio, J. & Bonawitz, E. Awesome play: awe increases preschooler’s exploration and discovery. In Proc. 40th Annual Conference of the Cognitive Science Society (eds Kalish, C. et al.) 1536–1541 (Cognitive Science Society, Seattle, 2018).

  20. Salge, C., Glackin, C. & Polani, D. in Guided Self-Organization: Inception (ed. Prokopenko, M.) 67–114 (Springer, 2014).

  21. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Joulin, A. et al. Fasttext. zip: compressing text classification models. Preprint at arXiv https://doi.org/10.48550/arXiv.1612.03651 (2016).

  23. Bhatia, S. Associative judgment and vector space semantics. Psychol. Rev. 124, 1–20 (2017).

  24. Fründ, I., Wichmann, F. A. & Macke, J. H. Quantifying the effect of intertrial dependence on perceptual decisions. J. Vis. https://doi.org/10.1167/14.7.9 (2014).

  25. Schmidhuber, J. Powerplay: training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Front. Psychol. 4, 313 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Nasiriany, S., Pong, V. H., Lin, S. & Levine, S. Planning with goal-conditioned policies. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (eds Wallach, H. et al.) 14843–14854 (Neural Information Processing Systems Foundation, San Diego, 2019).

  27. Campero, A. et al. Learning with AMIGo: adversarially motivated intrinsic goals. Preprint at arXiv https://doi.org/10.48550/arXiv.2006.12122 (2020).

  28. Chitnis, R., Silver, T., Tenenbaum, J., Kaelbling, L. P. & Lozano-Perez, T. GLIB: efficient exploration for relational model-based reinforcement learning via goal-literal babbling. Preprint at arXiv https://doi.org/10.48550/arXiv.2001.08299 (2020).

  29. Pathak, D., Gandhi, D. & Gupta, A. Self-supervised exploration via disagreement. In Proc. 36th International Conference on Machine Learning (eds Chaudhuri, K. & Salakhutdinov, R.) 5062–5071 (PMLR, Cambridge, MA, 2019).

  30. Gottlieb, J. & Oudeyer, P.-Y. Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19, 758–770 (2018).

    Article  CAS  PubMed  Google Scholar 

  31. Chu, J. & Schulz, L. E. Play, curiosity, and cognition. Annu. Rev. Dev. Psychol. 2, 317–343 (2020).

    Article  Google Scholar 

  32. Brändle, F., Stocks, L. J. & Schulz, E. franziskabraendle/alchemy_empowerment. Zenodo https://doi.org/10.5281/zenodo.8010316 (2023).

Download references

Acknowledgements

We thank J. Koziol for the ‘Little Alchemy 2’ dataset. This work was supported by the Max Planck Society, the Volkswagen Foundation (VW98569, E.S.) and a Jacobs Research Fellowship (E.S.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the paper.

Author information

Authors and Affiliations

  1. Max Planck Institute for Biological Cybernetics, Tübingen, Germany

    Franziska Brändle, Lena J. Stocks & Eric Schulz

  2. Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA

    Joshua B. Tenenbaum

  3. Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, USA

    Joshua B. Tenenbaum & Samuel J. Gershman

  4. Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, USA

    Samuel J. Gershman

Authors
  1. Franziska Brändle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lena J. Stocks
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joshua B. Tenenbaum
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Samuel J. Gershman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eric Schulz
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

F.B., L.J.S., S.J.G. and E.S. conceived the study. F.B. and E.S. collected the data. F.B., L.J.S. and E.S. performed the analyses. F.B., L.J.S., J.B.T., S.J.G. and E.S. wrote the paper.

Corresponding author

Correspondence to Franziska Brändle.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Kou Murayama and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary information and Supplementary Figs 1–6 and Tables 1–10.

Reporting Summary

Peer Review File

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Brändle, F., Stocks, L.J., Tenenbaum, J.B. et al. Empowerment contributes to exploration behaviour in a creative video game. Nat Hum Behav 7, 1481–1489 (2023). https://doi.org/10.1038/s41562-023-01661-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-023-01661-2

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing