Social cognition depends on our capacity for ‘mentalizing’, or explaining an agent’s behaviour in terms of their mental states. The development and neural substrates of mentalizing are well-studied, but its computational basis is only beginning to be probed. Here we present a model of core mentalizing computations: inferring jointly an actor’s beliefs, desires and percepts from how they move in the local spatial environment. Our Bayesian theory of mind (BToM) model is based on probabilistically inverting artificial-intelligence approaches to rational planning and state estimation, which extend classical expected-utility agent models to sequential actions in complex, partially observable domains. The model accurately captures the quantitative mental-state judgements of human participants in two experiments, each varying multiple stimulus dimensions across a large number of stimuli. Comparative model fits with both simpler ‘lesioned’ BToM models and a family of simpler non-mentalistic motion features reveal the value contributed by each component of our model.
This is a preview of subscription content
Subscribe to Nature+
Get immediate online access to the entire Nature family of 50+ journals
Subscribe to Journal
Get full journal access for 1 year
only $9.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Knill, D. & Richards, W. Perception as Bayesian Inference (Cambridge Univ. Press, 1996).
Marr, D . Vision (Freeman, 1982).
Weiss, Y., Simoncelli, E. P. & Adelson, E. H . Motion illusions as optimal percepts. Nat. Neurosci. 5, 598–604 (2002).
Carey, S . The Origin of Concepts (Oxford Univ. Press, 2009).
Csibra, G., Biró, S., Koós, O. & Gergely, G . One-year-old infants use teleological representations of actions productively. Cogn. Sci. 27, 111–133 (2003).
Hamlin, J. K., Wynn, K. & Bloom, P. Social evaluation by preverbal infants. Nature 450, 557–560 (2007).
Leslie, A. M., Friedman, O. & German, T. P . Core mechanisms in ‘theory of mind’. Trends Cogn. Sci. 8, 528–533 (2005).
Onishi, K. H. & Baillargeon, R. Do 15-month-old infants understand false beliefs? Science 308, 255–258 (2005).
Woodward, A. L. Infants selectively encode the goal object of an actor’s reach. Cognition 69, 1–34 (1998).
Wellman, H. M. Making Minds: How Theory of Mind Develops (Oxford Univ. Press, 2014).
Wimmer, H. & Perner, J . Beliefs about beliefs: representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition 13, 103–128 (1983).
Battaglia, P. W., Hamrick, J. B. & Tenenbaum, J. B . Simulation as an engine of physical scene understanding. Proc. Natl Acad. Sci. USA 110, 18327–18332 (2013).
Körding, K. P. & Wolpert, D. M. Bayesian integration in sensorimotor learning. Nature 427, 244–247 (2004).
Baker, C. L., Saxe, R. & Tenenbaum, J. B. Action understanding as inverse planning. Cognition 113, 329–349 (2009).
Jern, A. & Kemp, C. A decision network account of reasoning about other people’s choices. Cognition 142, 12–38 (2015).
Jara-Ettinger, J., Gweon, H., Tenenbaum, J. B. & Schulz, L. E. Children’s understanding of the costs and rewards underlying rational action. Cognition 140, 14–23 (2015).
Lucas, C. G. et al. The child as econometrician: a rational model of preference understanding in children. PLoS ONE 9, e92160 (2014).
E. Oztop, D. Wolpert & M. Kawato . Mental state inference using visual control parameters. Cogn. Brain Res. 22, 129–151 (2005).
Pantelis, P. C. et al. Inferring the intentional states of autonomous virtual agents. Cognition 130, 360–379 (2014).
Blythe, P. W., Todd, P. M. & Miller, G. F. in Simple Heuristics that Make Us Smart (eds Gigerenzer G., Todd, P. M. & the ABC Research Group) 257–286 (Oxford Univ. Press, 1999).
Zacks, J. M. Using movement and intentions to understand simple events. Cogn. Sci. 28, 979–1008 (2004).
Goodman, N. D., Baker, C. L. & Tenenbaum, J. B . Cause and intent: social reasoning in causal learning. In Proc. 31st Annu. Conf. Cognitive Science Society (eds Taatgen, N. & van Rijn, H.) 2759–2764 (Cognitive Science Society, 2009).
Shafto, P., Goodman, N. D. & Frank, M. C . Learning from others: the consequences of psychological reasoning for human learning. Persp. Psychol. Sci. 7, 341–351 (2012).
Rafferty A. N., LaMar M. M. & Griffiths, T. L. Inferring learners’ knowledge from their actions. Cogn. Sci. 39, 584–618 (2015).
Hawthorne-Madell, D. & Goodman, N. D. So good it has to be true: wishful thinking in theory of mind. In Proc. 37th Annu. Conf. Cognitive Science Society (eds Noelle, D. C. et al.) 884–889 (Cognitive Science Society, 2015).
Butterfield, J., Jenkins, O. C., Sobel, D. M. & Schwertfeger, J. Modeling aspects of theory of mind with Markov random fields. Int. J. Soc. Robot. 1, 41–51 (2009).
Shafto, P., Eaves, B., Navarro, D. J. & Perfors, A . Epistemic trust: modeling children’s reasoning about others’ knowledge and intent. Dev. Sci. 15, 436–447 (2012).
Gergely, G. & Csibra, G . Teleological reasoning in infancy: the naïve theory of rational action. Trends Cogn. Sci. 7, 287–292 (2003).
Perner, J . Understanding the Representational Mind (MIT Press, 1991).
Kaelbling, L. P., Littman, M. L. & Cassandra, A. R . Planning and acting in partially observable stochastic domains. Artificial Intelligence 101, 99–134 (1998).
von Neumann, J. & Morgenstern, O . Theory of Games and Economic Behavior (Princeton Univ. Press, 1953).
Gmytrasiewicz, P. J. & Doshi, P. A framework for sequential planning in multi-agent settings. J. Artif. Intell. Res. 24, 49–79 (2005).
Davis, L. S. & Benedikt, M. L . Computational models of space: isovists and isovist fields. Comput. Graph. Image Process. 11, 49–72 (1979).
Morariu, V. I., Prasad, V. S. N. & Davis L. S . Human activity understanding using visibility context. In IEEE/RSJ IROS Workshop: From Sensors to Human Spatial Concepts (FS2HSC) (2007).
Tremoulet, P. D. & Feldman, J . The influence of spatial context and the role of intentionality in the interpretation of animacy from motion. Percept. Psychophys. 29, 943–951 (2006).
Perner, J. & Ruffman, T. Infants’ insight into the mind: how deep? Science 308, 214–216 (2005).
Csibra, G. & Volein, Á . Infants can infer the presence of hidden objects from referential gaze information. Br. J. Dev. Psychol. 26, 1–11 (2008).
Moll, H. & Tomasello, M . 12- and 18-month-old infants follow gaze to spaces behind barriers. Dev. Sci. 7, F1–F9 (2004).
Hsu, D., Lee, W. S. & Rong, N . What makes some POMDP problems easy to approximate? In Advances in Neural Information Processing Systems (NIPS 2007) Vol. 20 (eds Platt, J. C. ) (NIPS Foundation, 2007).
Silver, D. & Veness, J . Monte-Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems (NIPS 2010) Vol. 23 (eds Lafferty, J. D. et al.) (NIPS Foundation, 2010).
Somani, A., Ye, N., Hsu, D. & Lee, W. S. Despot: online POMDP planning with regularization. In Advances in Neural Information Processing Systems (NIPS 2013) Vol. 26 (eds Burges, C. J. C. et al.) (NIPS Foundation, 2013).
Lovejoy, W. S . Computationally feasible bounds for partially observed Markov decision processes. Oper. Res. 39, 162–175 (1991).
Kurniawati, H., Hsu, D. & Lee, W. S. SARSOP: efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Proc. Robotics: Science and Systems Vol. 4 (eds Brock, O., Trinkle, J. & Ramos, F.) 65–72 (MIT, 2009).
Kulkarni, T. D., Kohli, P., Tenenbaum, J. B. & Mansinghka, V . Picture: an imperative probabilistic programming language for scene perception. In IEEE Conf. Computer Vision and Pattern Recognition 4390–4399 (Computer Vision Foundation, 2015).
de Villiers, J. G. & de Villiers, P. A. in Language Acquisition (ed. Foster-Cohen, S. ) Ch. 7, 169–195 (Palgrave Macmillan, 2009).
Baker, C. L., Goodman, N. D. & Tenenbaum, J. B . Theory-based social goal inference. In Proc. 30th Annu. Conf. Cognitive Science Society 1447–1455 (Cognitive Science Society, 2008).
Hamlin, J. K., Ullman, T. D., Tenenbaum, J. B., Goodman, N. D. & Baker, C. L. The mentalistic basis of core social cognition: experiments in preverbal infants and a computational model. Dev. Sci. 16, 209–226 (2013).
Littman, M. L . Markov games as a framework for multi-agent reinforcement learning. In Proc. 11th Int. Conf. Machine Learning Vol. 9166 (ed. Perner, P.) 157–163 (Springer, 1994).
Yoshida, W., Dolan, R. J. & Friston, K. J. Game theory of mind. PLoS Comput. Biol. 4, 1–14 (2008).
Doshi, P., Qu, X., Goodie, A. & Young, D. Modeling recursive reasoning by humans using empirically informed interactive POMDPs. In Proc. 9th Int. Conf. Autonomous Agents and Multiagent Systems (AAMAS) (International Foundation for Autonomous Agents and Multiagent Systems, 2010).
Stuhlmüller, A. & Goodman, N. D . Reasoning about reasoning by nested conditioning: modeling theory of mind with probabilistic programs. J. Cogn. Syst. Res. 28, 80–99 (2013).
Spelke, E. S. & Kinzler, K. D. Core knowledge. Dev. Sci. 10, 89–96 (2007).
Kahneman, D . Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011).
Ong, S. C. W., Png, S. W., Hsu, D. & Lee, W. S. POMDPs for robotic tasks with mixed observability. In Robotics: Science and Systems Vol. 5 (MIT Press, 2009).
Cohen, P. R. Empirical Methods in Artificial Intelligence (MIT Press, 1995).
This work was supported by the Center for Brains, Minds & Machines (CBMM), under NSF STC award CCF-1231216; by NSF grant IIS-1227495 and by DARPA grant IIS-1227504. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing interests.
About this article
Cite this article
Baker, C., Jara-Ettinger, J., Saxe, R. et al. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nat Hum Behav 1, 0064 (2017). https://doi.org/10.1038/s41562-017-0064
Nature Reviews Psychology (2022)
Nature Human Behaviour (2022)
Nature Reviews Psychology (2022)
Scientific Reports (2022)
Cognitive Computation (2022)