A new competition presents AI agents with cognition challenges to test their animal intelligence.
The past decade has seen great progress in artificial intelligence (AI). Machines can now categorize and generate images, make complex physical and social inferences, and reach, or exceed, human performance in many games. However, the long-term goal of recreating human-like general intelligence remains out of reach and some argue that a radical change in approach is needed1.
Animal-level intelligence provides a natural stepping stone on the path towards human-level AI. Tests in animal cognition research normally involve presenting an animal with a problem or environment that it would not naturally encounter and seeing if it can ‘work out’ how to obtain a reward — typically food. These tasks are carefully designed to probe for a particular cognitive capacity, such that successful performance on the task provides evidence that the animal has the capacity in question. Researchers have used this approach to test for capacities such as episodic memory, planning, spatial reasoning and social cognition in animals as varied as dogs, goats, chimpanzees and spiders.
In contrast to most animals, modern AI systems cannot just be placed in new environments and be expected to perform intelligently. Consider AlphaZero, a general algorithm that can be trained to better-than-human levels at a wide range of perfect information games. Without extensive retraining, however, it cannot adapt to never-seen-before games on the fly, and different games may require different input spaces, fundamentally preventing transfer between them. While AlphaZero is an impressive feat of AI, this case illustrates the large difference between current generalization capabilities of state-of-the-art AI systems and animals.
In animal cognition research, a wide range of species with different types of embodiment and (biological) actuators have been tested using a variety of experimental paradigms. These paradigms typically abstract away from interspecies differences by focusing on intelligent behaviour mediated by the shared sensory modality of vision. At the same time, we have seen rapid progress in the ability to train AI systems through visual inputs alone2. Thus, it is an ideal time for making direct comparisons between animals and AI. This is the aim of the Animal-AI Olympics, a new AI competition that translates vision-based animal cognition tasks into a testbed for cognitive AI. To keep the comparison to the animal case as close as possible, the participants (like the animals) will not know the exact tasks in advance. Participants will instead have to submit an agent that they believe will display robust food retrieval behaviour in tasks unknown to the developer.
We will be releasing a ‘playground’, a simple simulation environment for intelligent agents based on the Unity platform3. This environment has basic physics rules and a set of objects such as food, walls, negative-reward zones, pushable blocks and more. The playground can be configured by the participants and they can spawn any combination of objects in preset or random positions (pictured). It will be important for the participants to design good environments for their agents to learn in. Configuration files for the playground can also be exchanged between participants should they wish to collaborate. The competition tasks will include ten cognitive categories each with ten subtasks. This gives us one hundred distinct tasks, each of which will be run multiple times with minor variations for testing purposes. The categories will range from basic food retrieval — where only food is in the environment — to tasks that require capacities such as object permanence, object manipulation and an understanding of the basic physics of the environment to solve.
We expect this to be a hard challenge for modern AI systems, and want to give publicity to interesting approaches that make even small advancements in this area. We also hope that this will be a good testbed for approaches that use continual, transfer or one-shot learning as well as non-goal-directed learning methods, such as curiosity, intrinsic motivation and intuitive physics modelling4,5. Being able to solve all the tasks in a category would demonstrate real cognitive capacities comparable to those found in animals.
We released the playground at the end of April so that there is time for community feedback to be incorporated before the full release of the competition at the end of June. The competition itself will run from June to November, with participants able to submit to a live leader board throughout. The results will be announced at the 2019 Conference on Neural Information Processing Systems in December.
We hope this competition sparks further research in cognitive AI and that it becomes a useful ongoing testbed. We expect it to help pinpoint the current challenges and limitations of AI for large-scale real-world application involving interaction with unknown environments. We have made great progress on the hard problems; it is now time to tackle the easy ones.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. Behav. Brain Sci. 40, e253 (2017).
Mnih, V. et al. Nature 518, 529–533 (2015).
Juliani, A. et al. Preprint at https://arxiv.org/abs/1809.02627 (2018).
Haber, N., Mrowca, D., Fei-Fei, L. & Yamins, D. L. Preprint at https://arxiv.org/abs/1802.07461 (2018).
Ullman, T. D., Spelke, E., Battaglia, P. & Tenenbaum, J. B. Trends Cogn. Sci. 21, 649–665 (2017).
This work was supported by the Leverhulme Trust and Templeton World Charity Foundation.
The authors declare no competing interests.
About this article
Cite this article
Crosby, M., Beyret, B. & Halina, M. The Animal-AI Olympics. Nat Mach Intell 1, 257 (2019). https://doi.org/10.1038/s42256-019-0050-3
Mind & Language (2021)
Performance Research (2020)
Proceedings of the National Academy of Sciences (2020)
Nature Neuroscience (2020)
Nature Machine Intelligence (2020)