Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Language and culture internalization for human-like autotelic AI

Nature Machine Intelligence volume 4, pages 1068–1076 (2022)Cite this article

Subjects

Abstract

Building autonomous agents able to grow open-ended repertoires of skills across their lives is a fundamental goal of artificial intelligence (AI). A promising developmental approach recommends the design of intrinsically motivated agents that learn new skills by generating and pursuing their own goals—autotelic agents. But despite recent progress, existing algorithms still show serious limitations in terms of goal diversity, exploration, generalization or skill composition. This Perspective calls for the immersion of autotelic agents into rich socio-cultural worlds, an immensely important attribute of our environment that shapes human cognition but is mostly omitted in modern AI. Inspired by the seminal work of Vygotsky, we propose Vygotskian autotelic agents—agents able to internalize their interactions with others and turn them into cognitive tools. We focus on language and show how its structure and informational content may support the development of new cognitive functions in artificial agents as it does in humans. We justify the approach by uncovering several examples of new artificial cognitive functions emerging from interactions between language and embodiment in recent works at the intersection of deep reinforcement learning and natural language processing. Looking forward, we highlight future opportunities and challenges for Vygotskian autotelic AI research, including the use of language models as cultural models supporting artificial cognitive development.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: From multi-goal RL to autotelic RL and Vygotskian autotelic RL.
Fig. 2: The three components of Vygotskian autotelic agents, that is, socio-cultural interactions, linguistic extraction and internalized linguistic production.

Similar content being viewed by others

References

  1. Sutton, R. S. & Barto, A. G. Introduction to Reinforcement Learning (MIT Press, 1998).

  2. Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016).

    Article  Google Scholar 

  3. Bellemare, M. G. et al. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588, 77–82 (2020).

    Article  Google Scholar 

  4. Degrave, J. et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602, 414–419 (2022).

    Article  Google Scholar 

  5. Piaget, J. The Origins of Intelligence in Children (transl. Cook, M.) (WW Norton, 1952).

  6. Dautenhahn, K. & Billard, A. Studying robot social cognition within a developmental psychology framework. Proc. Eurobot 187–194 (IEEE, 1999).

  7. Colas, C., Karch, T., Sigaud, O. & Oudeyer, P.-Y. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. J. Artif. Intell. Res. 74, 1159–1199 (2022).

    Article  MathSciNet  MATH  Google Scholar 

  8. Schaul, T., Horgan, D., Gregor, K. & Silver, D. Universal value function approximators. Proc. ICML 37, 1312–1320 (PMLR, 2015).

  9. Berlyne, D. E. Curiosity and exploration. Science 153, 25–33 (1966).

    Article  Google Scholar 

  10. Kidd, C. & Hayden, B. Y. The psychology and neuroscience of curiosity. Neuron 88, 449–460 (2015).

    Article  Google Scholar 

  11. Gottlieb, J. & Oudeyer, P.-Y. Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19, 758–770 (2018).

    Article  Google Scholar 

  12. Schmidhuber, J. Curious model-building control systems. IEEE International Joint Conference on Neural Networks 2, 1458–1463 (IEEE, 1991).

  13. Barto, A. G. & Simsek, O. Intrinsic motivation for reinforcement learning systems. Proc. Thirteenth Yale Workshop on Adaptive and Learning Systems 113–118 (Yale Univ. Press, 2005).

  14. Oudeyer, P.-Y., Kaplan, F. & Hafner, V. V. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11, 265–286 (IEEE, 2007).

  15. Oudeyer, P.-Y. & Kaplan, F. What is intrinsic motivation? A typology of computational approaches. Front. Neurorobot. 1, 1–6 (2009).

    Google Scholar 

  16. Colas, C. et al. Language as a cognitive tool to imagine goals in curiosity driven exploration. Proc. NeurIPS 33, 3761–3774 (Curran Associates, 2020).

  17. Witty, S. et al. Measuring and characterizing generalization in deep reinforcement learning. Appl. AI Lett. https://doi.org/10.1002/ail2.45 (2021).

    Article  Google Scholar 

  18. Shanahan, M. & Mitchell, M. Abstraction for deep reinforcement learning. Proc. IJCAI 31, 5588–5596 (2022).

    Google Scholar 

  19. Vygotsky, L. S. Thought and Language (MIT Press, 1934).

  20. Berk, L. E. Why children talk to themselves. Sci. Am. 271, 78–83 (1994).

    Article  Google Scholar 

  21. Lupyan, G. in Psychology of Learning and Motivation 57, 255–297 (ed. Ross, B. H.) (Elsevier, 2012).

  22. Gentner, D. & Hoyos, C. Analogy and abstraction. Top. Cogn. Sci. 9, 672–693 (2017).

    Article  Google Scholar 

  23. Whorf, B. L. Language, Thought, and Reality: Selected Writings of Benjamin Lee Whorf (edited by Carroll J. B.) (MIT Press, 1956).

  24. Rumelhart, D. E., Smolensky, P., McClelland, J. L. & Hinton, G. in Parallel Distributed Processing: Explorations in the Microstructures of Cognition (1986).

  25. Lakoff, G. & Johnson, M. Metaphors We Live By (Univ. Chicago Press, 2008).

  26. Hesse, M. The cognitive claims of metaphor. J. Speculative Phil. 2, 1–16 (1988).

    Google Scholar 

  27. Dennett, D. C. Consciousness Explained (Penguin, 1993).

  28. Clark, A. in Language and Thought 162–183 (eds Carruthers, P. & Boucher, J.) (Cambridge Univ. Press, 1998).

  29. Carruthers, P. Modularity, language, and the flexibility of thought. Behav. Brain Sci. 25, 705–719 (2002).

    Article  Google Scholar 

  30. Tomasello, M. The Cultural Origins of Human Cognition (Harvard Univ. Press, 1999).

  31. Tomasello, M., Carpenter, M., Call, J., Behne, T. & Moll, H. Understanding and sharing intentions: the origins of cultural cognition. Behav. Brain Sci. 28, 675–691 (2005).

    Article  Google Scholar 

  32. Brewer, K., Pollock, N. & Wright, F. V. Addressing the challenges of collaborative goal setting with children and their families. Phys. Occup. Ther. Pediatr. 34, 138–152 (2014).

    Article  Google Scholar 

  33. Carruthers, P. & Boucher, J. Language and Thought (Cambridge Univ. Press, 1998).

  34. Dove, G. Language as a disruptive technology: abstract concepts, embodiment and the flexible mind. Phil. Trans. R. Soc. B 373, 705–719 (2018).

    Article  Google Scholar 

  35. Brown, T. B. et al. Language models are few-shot learners. Proc. NeurIPS 33, 1877–1901 (2020).

    Google Scholar 

  36. Chowdhery, A. et al. PaLM: scaling language modeling with pathways. Preprint at https://arxiv.org/abs/2204.02311 (2022).

  37. Radford, A. et al. Learning transferable visual models from natural language supervision. Proc. ICML 38, 8748–8763 (2021). PMLR.

    Google Scholar 

  38. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C. & Chen, M. Hierarchical text-conditional image generation with CLIP latents. Preprint at https://arxiv.org/abs/2204.06125 (2022).

  39. Alayrac, J.-B. et al. Flamingo: a visual language model for few-shot learning. Proc. NeurIPS 36 (2022)

  40. Creswell, A., Shanahan, M. & Higgins, I. Selection-inference: exploiting large language models for interpretable logical reasoning. Preprint at https://arxiv.org/abs/2205.09712 (2022).

  41. West, P. et al. Symbolic knowledge distillation: from general language models to commonsense models. Proc. NAACL 4602–4625 (ACL, 2022).

  42. Hershcovich, D. et al. Challenges and strategies in cross-cultural NLP. Proc. ACL 60, 6997–7013 (2022). Association for Computer Linguistics.

    Google Scholar 

  43. Arora, A., Kaffee, L.-A. & Augenstein, I. Probing pre-trained language models for cross-cultural differences in values. Proc. ACL (2022).

  44. Zlatev, J. The epigenesis of meaning in human beings, and possibly in robots. Minds Mach 11, 155–195 (2001).

    Article  MATH  Google Scholar 

  45. Lindblom, J. & Ziemke, T. Social situatedness of natural and artificial intelligence: Vygotsky and beyond. Adapt. Behav. 11, 79–96 (2003).

    Article  Google Scholar 

  46. Mirolli, M. & Parisi, D. Towards a Vygotskyan cognitive robotics: the role of language as a cognitive tool. New Ideas Psychol. 29, 298–311 (2011).

    Article  Google Scholar 

  47. Cangelosi, A. et al. Integration of action and language knowledge: a roadmap for developmental robotics. IEEE Transactions on Autonomous Mental Development 2, 167–195 (2010). IEEE.

    Article  Google Scholar 

  48. Jaques, N. et al. Social Influence as intrinsic motivation for multi-agent deep reinforcement learning. Proc. ICML 36, 3040–3049 (2019). PMLR.

    Google Scholar 

  49. Sigaud, O., Colas, C., Akakzia, A., Chetouani, M. & Oudeyer, P.-Y. Towards teachable autonomous agents. Preprint at https://arxiv.org/abs/2105.11977 (2021).

  50. Harari, Y. N. Sapiens: A Brief History of Humankind (Random House, 2014).

  51. Vyshedskiy, A. Language evolution to revolution: the leap from rich-vocabulary non-recursive communication system to recursive language 70,000 years ago was associated with acquisition of a novel component of imagination, called prefrontal synthesis, enabled by a mutation that slowed down the prefrontal cortex maturation simultaneously in two or more children—the Romulus and Remus hypothesis. Res. Ideas Outcomes no. 7528 (2019).

  52. Goldberg, A. E. in The Emergence of Language 197–212 (ed. MacWhinney, B.) (Psychology Press, 1999).

  53. Hoffmann, T. Construction grammar and creativity: evolution, psychology, and cognitive science. Cogn. Semiot. 13 (2020).

  54. Chomsky, N. Syntactic Structures (Mouton, 1957).

  55. Fodor, J. A. The Language of Thought (Harvard Univ. Press, 1975).

  56. Wittgenstein, L. Philosophical Investigations (Wiley, 1953).

  57. McDowell, J. Mind and World (Harvard Univ Press, 1996).

  58. Sperber, D., Premack, D. & Premack, A. J. Causal Cognition: A Multidisciplinary Debate (Clarendon Press, 1995).

  59. Allen, C. & Bekoff, M. Species of Mind: The Philosophy and Biology of Cognitive Ethology (MIT Press, 1999).

  60. Waxman, S. R. & Markow, D. B. Words as invitations to form categories: evidence from 12-to 13-month-old infants. Cogn. Psychol. 29, 257–302 (1995).

    Article  Google Scholar 

  61. Yoshida, H. & Smith, L. B. Sound symbolism and early word learning in two languages. In Proc. Annual Meeting of the Cognitive Science Society 25, 1287–1292 (2003).

  62. Gentner, D. & Loewenstein, J. Relational Language and Relational Thought (Erlbaum, 2002).

  63. Henrich, J. & McElreath, R. The evolution of cultural evolution. Evol. Anthropol. 12, 123–135 (2003).

    Article  Google Scholar 

  64. Morgan, T. J. et al. Experimental evidence for the co-evolution of hominin tool-making teaching and language. Nat. Commun. 6, 1–8 (2015).

    Article  Google Scholar 

  65. Chopra, S., Tessler, M. H. & Goodman, N. D. The first crank of the cultural ratchet: learning and transmitting concepts through language. Cogn. Sci. 41, 226–232 (2019).

    Google Scholar 

  66. Bruner, J. Acts of Meaning (Harvard Univ. Press, 1990).

  67. Carruthers, P. in Language and Thought 94–120 (eds Carruthers, P. and Boucher, J.) (Cambridge Univ. Press, 1998).

  68. Lupyan, G. in Modeling Language, Cognition and Action (eds Cangelosi, A. et al.) 87–96 (World Scientific, 2005).

  69. Hill, F. et al. Emergent systematic generalization in a situated agent. Proc. ICLR 8 (2020).

  70. Jiang, Y., Gu, S., Murphy, K. & Finn, C. Language as an abstraction for hierarchical deep reinforcement learning. Proc. NeurIPS 32 (Curran Associates, 2019).

  71. Lynch, C. & Sermanet, P. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems XVII (2021).

  72. Chevalier-Boisvert, M. et al. Baby-AI: first steps towards grounded language learning with a human in the loop. Proc. ICLR 7 (2019).

  73. Yuan, X. et al. Interactive language learning by question answering. In Proc. EMNLP 9, 2796–2813 (Association for Computational Linguistics, 2019).

  74. Colas, C., Oudeyer, P., Sigaud, O., Fournier, P. & Chetouani, M. CURIOUS: intrinsically motivated modular multi-goal reinforcement learning. Proc. ICML 36, 1331–1340 (2019). PMLR.

    Google Scholar 

  75. Stooke, A. et al. Open-ended learning leads to generally capable agents. Preprint at https://arxiv.org/abs/2107.12808 (2021).

  76. Bahdanau, D. et al. Learning to understand goal specifications by modelling reward. Proc. ICLR 7 (2019).

  77. Tam, A. C. et al. Semantic exploration from language abstractions and pretrained representations. Proc. NeurIPS, 36 (2022).

  78. Mu, J. et al. Improving intrinsic exploration with language abstractions. Proc. NeurIPS, 36 (2022).

  79. Hermann, K. M. et al. Grounded language learning in a simulated 3D world. Preprint at https://arxiv.org/abs/1706.06551 (2017).

  80. Hill, F., Mokra, S., Wong, N. & Harley, T. Human instruction-following with deep reinforcement learning via transfer-learning from text. Preprint at https://arxiv.org/abs/2005.09382 (2020).

  81. Sharma, P., Torralba, A. & Andreas, J. Skill induction and planning with latent language. Proc. ACL 60, 1713–1736 (2021). Association for Computational Linguistics.

    Google Scholar 

  82. Karch, T., Teodorescu, L., Hofmann, K., Moulin-Frier, C. & Oudeyer, P.-Y. Grounding spatio-temporal language with transformers. Proc. NeurIPS 24, 5236–5249 (2021). Curran Associates, Inc.

    Google Scholar 

  83. Akakzia, A., Colas, C., Oudeyer, P.-Y., Chetouani, M. & Sigaud, O. Grounding language to autonomously-acquired skills via goal generation. Proc. ICLR 9 (2021).

  84. Ramesh, A. et al. Zero-shot text-to-image generation. Preprint at https://arxiv.org/abs/2102.12092 (2021).

  85. Hermer-Vazquez, L. Language, space, and the development of cognitive flexibility in humans: the case of two spatial memory tasks. Cognition 79, 263–299 (2001).

    Article  Google Scholar 

  86. Wong, C., Ellis, K., Tenenbaum, J. B. & Andreas, J. Leveraging language to learn program abstractions and search heuristics. Proc. ICML 38, 11193–11204 (2021). PMLR.

    Google Scholar 

  87. Chen, V., Gupta, A. & Marino, K. Ask your humans: using human instructions to improve generalization in reinforcement learning. Proc. ICLR 9 (2021).

  88. Mirchandani, S., Karamcheti, S. & Sadigh, D. ELLA: exploration through learned language abstraction. Adva. Neural Inf. Process. Syst. 34, 29529–29540 (2021).

    Google Scholar 

  89. Shridhar, M. et al. ALFWorld: aligning text and embodied environments for interactive learning. Proc. ICLR 9 (2021).

  90. Côté, M.-A. et al. TextWorld: a learning environment for text-based games. Computer Games—7th Workshop at IJCAI (2018).

  91. Das, A. et al. Embodied question answering. Proc. CVPR, 1-10 (IEEE, 2018).

  92. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. Proc. NAACL 4171–4186 (Association for Computational Linguistics, 2019).

  93. Huang, W., Abbeel, P., Pathak, D. & Mordatch, I. Language models as zero-shot planners: extracting actionable knowledge for embodied agents. Proc. ICML 39, 9118–9147 (2022). PMLR.

    Google Scholar 

  94. Ahn, M. et al. Do as i can, not as i say: grounding language in robotic affordances. Preprint at https://arxiv.org/abs/2204.01691 (2022).

  95. Fan, L. et al. Minedojo: building open-ended embodied agents with internet-scale knowledge. Proc. AAAI (2022).

  96. Zeng, A. et al. Socratic models: composing zero-shot multimodal reasoning with language. Preprint at https://arxiv.org/abs/2204.00598 (2022).

  97. McClelland, J. L. et al. Letting structure emerge: connectionist and dynamical systems approaches to cognition. Trends Cogn. Sci. 14, 348–356 (2010).

    Article  Google Scholar 

  98. Santoro, A., Lampinen, A., Mathewson, K., Lillicrap, T. & Raposo, D. Symbolic behaviour in artificial intelligence. Preprint at https://arxiv.org/abs/2102.03406 (2021).

  99. Cideron, G., Seurin, M., Strub, F. & Pietquin, O. HIGhER: improving instruction following with hindsight generation for experience replay. In IEEE Symposium Series on Computational Intelligence 225–232 (IEEE, 2020).

  100. Zhou, L. & Small, K. Inverse reinforcement learning with natural language goals. Proc. AAAI 35, 11116–11124 (2021).

    Article  Google Scholar 

  101. Nguyen, K., Misra, D., Schapire, R., Dudík, M. & Shafto, P. Interactive learning from activity description. Proc. ICML 38, 8096–8108 (2021). PMLR.

    Google Scholar 

  102. Yan, C. et al. Intra-agent speech permits zero-shot task acquisition. Proc. NeurIPS 36 (2022).

  103. Andrychowicz, M. et al. Hindsight experience replay. Proc. NeurIPS (2017).

  104. Dayan, P. & Hinton, G. E. Feudal Reinforcement Learning. Advances Neural Inf. Process. Syst. 5, 271–278 (1993).

    MATH  Google Scholar 

  105. Sutton, R. S., Precup, D. & Singh, S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112, 181–211 (1999).

    Article  MathSciNet  MATH  Google Scholar 

  106. Lampinen, A. K. et al. Tell ME Why!—Explanations support learning of relational and causal structure. Proc. ICML 39, 11868–11890 (2022). PMLR.

    Google Scholar 

  107. Nair, A. et al. Visual reinforcement learning with imagined goals. Proc. NeurIPS 31 (Curran Associates, Inc. 2018).

  108. Runco, M. A. & Jaeger, G. J. The standard definition of creativity. Creat. Res. J. 24, 92–96 (2012).

    Article  Google Scholar 

  109. Shah, D. S., Schwartz, H. A. & Hovy, D. Predictive biases in natural language processing models: a conceptual framework and overview. In Proc. 58th Annual Meeting of the Association for Computational Linguistics 5248–5264 (Association for Computational Linguistics, 2020).

  110. Liang, P. P., Wu, C., Morency, L.-P. & Salakhutdinov, R. Towards understanding and mitigating social biases in language models. Proc 38th International Conference on Machine Learning, Proc. Machine Learning Research Vol. 139 (eds Meila, M. & Zhang, T.) 6565–6576 (PMLR, 2021).

  111. Weidinger, L. et al. Ethical and social risks of harm from language models. Preprint at https://arxiv.org/abs/2112.04359 (2021).

  112. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proc. 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623 (Association for Computing Machinery, 2021).

  113. Bonawitz, E. et al. The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery. Cognition 120, 322–330 (2011).

    Article  Google Scholar 

  114. Bisk, Y. et al. Experience grounds language. In Proc. EMNLP, 8718–8735 (Association for Computational Linguistics, 2020).

  115. Tomasello, M. Becoming Human—A Theory of Ontogeny (Harvard Univ. Press, 2019).

  116. Vélez, N. & Gweon, H. Learning from other minds: an optimistic critique of reinforcement learning models of social learning. Curr. Opin. Behav. Sci. 38, 110–115 (2021).

    Article  Google Scholar 

  117. Elliott, E. M. et al. Multilab direct replication of Flavell, Beach, and Chinsky (1966): spontaneous verbal rehearsal in a memory task as a function of age. Adv. Methods Pract. Psychol. Sci. 4, 1–20 (2021).

    Google Scholar 

  118. Atance, C. M. Future thinking in young children. Curr. Dir. Psychol. Sci. 17, 295–298 (2008).

    Article  Google Scholar 

Download references

Acknowledgements

We thank O. Sigaud for his helpful feedback. C.C. and T.K. are partially funded by the French Ministère des Armées - Direction Générale de l’Armement. C.M.-F. is partially funded by the Inria Exploratory action ORIGINS (https://www.inria.fr/en/origins) as well as the French National Research Agency (https://anr.fr/, project ECOCURL, grant ANR-20-CE23-0006).

Author information

Author notes

  1. These authors contributed equally: Cédric Colas, Tristan Karch.

Authors and Affiliations

  1. INRIA Bordeaux – Sud-Ouest, Talence, France

    Cédric Colas, Tristan Karch, Clément Moulin-Frier & Pierre-Yves Oudeyer

  2. Université de Bordeaux, Talence, France

    Tristan Karch

  3. ENSTA Paris, Palaiseau, France

    Clément Moulin-Frier & Pierre-Yves Oudeyer

  4. Microsoft Research, Montreal, Canada

    Pierre-Yves Oudeyer

Authors
  1. Cédric Colas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tristan Karch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Clément Moulin-Frier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pierre-Yves Oudeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cédric Colas.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Andrew Lampinen, Tom Ziemke and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Colas, C., Karch, T., Moulin-Frier, C. et al. Language and culture internalization for human-like autotelic AI. Nat Mach Intell 4, 1068–1076 (2022). https://doi.org/10.1038/s42256-022-00591-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00591-4

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics