Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Artificial intelligence and illusions of understanding in scientific research

Nature volume 627, pages 49–58 (2024)Cite this article

Subjects

Abstract

Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists’ visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community’s ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Illusions of understanding in AI-driven scientific research.

Similar content being viewed by others

References

  1. Crabtree, G. Self-driving laboratories coming of age. Joule 4, 2538–2541 (2020).

    Article  CAS  Google Scholar 

  2. Wang, H. et al. Scientific discovery in the age of artificial intelligence. Nature 620, 47–60 (2023). This review explores how AI can be incorporated across the research pipeline, drawing from a wide range of scientific disciplines.

    Article  CAS  PubMed  ADS  Google Scholar 

  3. Dillion, D., Tandon, N., Gu, Y. & Gray, K. Can AI language models replace human participants? Trends Cogn. Sci. 27, 597–600 (2023).

    Article  PubMed  Google Scholar 

  4. Grossmann, I. et al. AI and the transformation of social science research. Science 380, 1108–1109 (2023). This forward-looking article proposes a variety of ways to incorporate generative AI into social-sciences research.

    Article  CAS  PubMed  ADS  Google Scholar 

  5. Gil, Y. Will AI write scientific papers in the future? AI Mag. 42, 3–15 (2022).

    Google Scholar 

  6. Kitano, H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Syst. Biol. Appl. 7, 29 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Benjamin, R. Race After Technology: Abolitionist Tools for the New Jim Code (Oxford Univ. Press, 2020). This book examines how social norms about race become embedded in technologies, even those that are focused on providing good societal outcomes.

  8. Broussard, M. More Than a Glitch: Confronting Race, Gender, and Ability Bias in Tech (MIT Press, 2023).

  9. Noble, S. U. Algorithms of Oppression: How Search Engines Reinforce Racism (New York Univ. Press, 2018).

  10. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: can language models be too big? in Proc. 2021 ACM Conference on Fairness, Accountability, and Transparency 610–623 (Association for Computing Machinery, 2021). One of the first comprehensive critiques of large language models, this article draws attention to a host of issues that ought to be considered before taking up such tools.

  11. Crawford, K. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence (Yale Univ. Press, 2021).

  12. Johnson, D. G. & Verdicchio, M. Reframing AI discourse. Minds Mach. 27, 575–590 (2017).

    Article  Google Scholar 

  13. Atanasoski, N. & Vora, K. Surrogate Humanity: Race, Robots, and the Politics of Technological Futures (Duke Univ. Press, 2019).

  14. Mitchell, M. & Krakauer, D. C. The debate over understanding in AI’s large language models. Proc. Natl Acad. Sci. USA 120, e2215907120 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kidd, C. & Birhane, A. How AI can distort human beliefs. Science 380, 1222–1223 (2023).

    Article  CAS  PubMed  ADS  Google Scholar 

  16. Birhane, A., Kasirzadeh, A., Leslie, D. & Wachter, S. Science in the age of large language models. Nat. Rev. Phys. 5, 277–280 (2023).

    Article  Google Scholar 

  17. Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 100804 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  18. Hullman, J., Kapoor, S., Nanayakkara, P., Gelman, A. & Narayanan, A. The worst of both worlds: a comparative analysis of errors in learning from data in psychology and machine learning. In Proc. 2022 AAAI/ACM Conference on AI, Ethics, and Society (eds Conitzer, V. et al.) 335–348 (Association for Computing Machinery, 2022).

  19. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019). This paper articulates the problems with attempting to explain AI systems that lack interpretability, and advocates for building interpretable models instead.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Crockett, M. J., Bai, X., Kapoor, S., Messeri, L. & Narayanan, A. The limitations of machine learning models for predicting scientific replicability. Proc. Natl Acad. Sci. USA 120, e2307596120 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Lazar, S. & Nelson, A. AI safety on whose terms? Science 381, 138 (2023).

    Article  PubMed  ADS  Google Scholar 

  22. Collingridge, D. The Social Control of Technology (St Martin’s Press, 1980).

  23. Wagner, G., Lukyanenko, R. & Paré, G. Artificial intelligence and the conduct of literature reviews. J. Inf. Technol. 37, 209–226 (2022).

    Article  Google Scholar 

  24. Hutson, M. Artificial-intelligence tools aim to tame the coronavirus literature. Nature https://doi.org/10.1038/d41586-020-01733-7 (2020).

    Article  PubMed  Google Scholar 

  25. Haas, Q. et al. Utilizing artificial intelligence to manage COVID-19 scientific evidence torrent with Risklick AI: a critical tool for pharmacology and therapy development. Pharmacology 106, 244–253 (2021).

    Article  CAS  PubMed  Google Scholar 

  26. Müller, H., Pachnanda, S., Pahl, F. & Rosenqvist, C. The application of artificial intelligence on different types of literature reviews – a comparative study. In 2022 International Conference on Applied Artificial Intelligence (ICAPAI) https://doi.org/10.1109/ICAPAI55158.2022.9801564 (Institute of Electrical and Electronics Engineers, 2022).

  27. van Dinter, R., Tekinerdogan, B. & Catal, C. Automation of systematic literature reviews: a systematic literature review. Inf. Softw. Technol. 136, 106589 (2021).

    Article  Google Scholar 

  28. Aydın, Ö. & Karaarslan, E. OpenAI ChatGPT generated literature review: digital twin in healthcare. In Emerging Computer Technologies 2 (ed. Aydın, Ö.) 22–31 (İzmir Akademi Dernegi, 2022).

  29. AlQuraishi, M. AlphaFold at CASP13. Bioinformatics 35, 4862–4865 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  31. Lee, J. S., Kim, J. & Kim, P. M. Score-based generative modeling for de novo protein design. Nat. Computat. Sci. 3, 382–392 (2023).

    Article  CAS  Google Scholar 

  32. Gómez-Bombarelli, R. et al. Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach. Nat. Mater. 15, 1120–1127 (2016).

    Article  PubMed  ADS  Google Scholar 

  33. Krenn, M. et al. On scientific understanding with artificial intelligence. Nat. Rev. Phys. 4, 761–769 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Extance, A. How AI technology can tame the scientific literature. Nature 561, 273–274 (2018).

    Article  CAS  PubMed  ADS  Google Scholar 

  35. Hastings, J. AI for Scientific Discovery (CRC Press, 2023). This book reviews current and future incorporation of AI into the scientific research pipeline.

  36. Ahmed, A. et al. The future of academic publishing. Nat. Hum. Behav. 7, 1021–1026 (2023).

    Article  PubMed  Google Scholar 

  37. Gray, K., Yam, K. C., Zhen’An, A. E., Wilbanks, D. & Waytz, A. The psychology of robots and artificial intelligence. In The Handbook of Social Psychology (eds Gilbert, D. et al.) (in the press).

  38. Argyle, L. P. et al. Out of one, many: using language models to simulate human samples. Polit. Anal. 31, 337–351 (2023).

    Article  Google Scholar 

  39. Aher, G., Arriaga, R. I. & Kalai, A. T. Using large language models to simulate multiple humans and replicate human subject studies. In Proc. 40th International Conference on Machine Learning (eds Krause, A. et al.) 337–371 (JMLR.org, 2023).

  40. Binz, M. & Schulz, E. Using cognitive psychology to understand GPT-3. Proc. Natl Acad. Sci. USA 120, e2218523120 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Ornstein, J. T., Blasingame, E. N. & Truscott, J. S. How to train your stochastic parrot: large language models for political texts. Github, https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf (2023).

  42. He, S. et al. Learning to predict the cosmological structure formation. Proc. Natl Acad. Sci. USA 116, 13825–13832 (2019).

    Article  MathSciNet  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  43. Mahmood, F. et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. IEEE Trans. Med. Imaging 39, 3257–3267 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  44. Teixeira, B. et al. Generating synthetic X-ray images of a person from the surface geometry. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 9059–9067 (Institute of Electrical and Electronics Engineers, 2018).

  45. Marouf, M. et al. Realistic in silico generation and augmentation of single-cell RNA-seq data using generative adversarial networks. Nat. Commun. 11, 166 (2020).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  46. Watts, D. J. A twenty-first century science. Nature 445, 489 (2007).

    Article  CAS  PubMed  ADS  Google Scholar 

  47. boyd, d. & Crawford, K. Critical questions for big data. Inf. Commun. Soc. 15, 662–679 (2012). This article assesses the ethical and epistemic implications of scientific and societal moves towards big data and provides a parallel case study for thinking about the risks of artificial intelligence.

    Article  Google Scholar 

  48. Jolly, E. & Chang, L. J. The Flatland fallacy: moving beyond low–dimensional thinking. Top. Cogn. Sci. 11, 433–454 (2019).

    Article  PubMed  Google Scholar 

  49. Yarkoni, T. & Westfall, J. Choosing prediction over explanation in psychology: lessons from machine learning. Perspect. Psychol. Sci. 12, 1100–1122 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Radivojac, P. et al. A large-scale evaluation of computational protein function prediction. Nat. Methods 10, 221–227 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Bileschi, M. L. et al. Using deep learning to annotate the protein universe. Nat. Biotechnol. 40, 932–937 (2022).

    Article  CAS  PubMed  Google Scholar 

  52. Barkas, N. et al. Joint analysis of heterogeneous single-cell RNA-seq dataset collections. Nat. Methods 16, 695–698 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Demszky, D. et al. Using large language models in psychology. Nat. Rev. Psychol. 2, 688–701 (2023).

    Article  Google Scholar 

  54. Karjus, A. Machine-assisted mixed methods: augmenting humanities and social sciences with artificial intelligence. Preprint at https://arxiv.org/abs/2309.14379 (2023).

  55. Davies, A. et al. Advancing mathematics by guiding human intuition with AI. Nature 600, 70–74 (2021).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  56. Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D. & Griffiths, T. L. Using large-scale experiments and machine learning to discover theories of human decision-making. Science 372, 1209–1214 (2021).

    Article  CAS  PubMed  ADS  Google Scholar 

  57. Ilyas, A. et al. Adversarial examples are not bugs, they are features. Preprint at https://doi.org/10.48550/arXiv.1905.02175 (2019)

  58. Semel, B. M. Listening like a computer: attentional tensions and mechanized care in psychiatric digital phenotyping. Sci. Technol. Hum. Values 47, 266–290 (2022).

    Article  Google Scholar 

  59. Gil, Y. Thoughtful artificial intelligence: forging a new partnership for data science and scientific discovery. Data Sci. 1, 119–129 (2017).

    Article  Google Scholar 

  60. Checco, A., Bracciale, L., Loreti, P., Pinfield, S. & Bianchi, G. AI-assisted peer review. Humanit. Soc. Sci. Commun. 8, 25 (2021).

    Article  Google Scholar 

  61. Thelwall, M. Can the quality of published academic journal articles be assessed with machine learning? Quant. Sci. Stud. 3, 208–226 (2022).

    Article  Google Scholar 

  62. Dhar, P. Peer review of scholarly research gets an AI boost. IEEE Spectrum spectrum.ieee.org/peer-review-of-scholarly-research-gets-an-ai-boost (2020).

  63. Heaven, D. AI peer reviewers unleashed to ease publishing grind. Nature 563, 609–610 (2018).

    Article  CAS  PubMed  ADS  Google Scholar 

  64. Conroy, G. How ChatGPT and other AI tools could disrupt scientific publishing. Nature 622, 234–236 (2023).

    Article  CAS  PubMed  ADS  Google Scholar 

  65. Nosek, B. A. et al. Replicability, robustness, and reproducibility in psychological science. Annu. Rev. Psychol. 73, 719–748 (2022).

    Article  PubMed  Google Scholar 

  66. Altmejd, A. et al. Predicting the replicability of social science lab experiments. PLoS ONE 14, e0225826 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. Yang, Y., Youyou, W. & Uzzi, B. Estimating the deep replicability of scientific findings using human and artificial intelligence. Proc. Natl Acad. Sci. USA 117, 10762–10768 (2020).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  68. Youyou, W., Yang, Y. & Uzzi, B. A discipline-wide investigation of the replicability of psychology papers over the past two decades. Proc. Natl Acad. Sci. USA 120, e2208863120 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  69. Rabb, N., Fernbach, P. M. & Sloman, S. A. Individual representation in a community of knowledge. Trends Cogn. Sci. 23, 891–902 (2019). This comprehensive review paper documents the empirical evidence for distributed cognition in communities of knowledge and the resultant vulnerabilities to illusions of understanding.

    Article  PubMed  Google Scholar 

  70. Rozenblit, L. & Keil, F. The misunderstood limits of folk science: an illusion of explanatory depth. Cogn. Sci. 26, 521–562 (2002). This paper provided an empirical demonstration of the illusion of explanatory depth, and inspired a programme of research in cognitive science on communities of knowledge.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Hutchins, E. Cognition in the Wild (MIT Press, 1995).

  72. Lave, J. & Wenger, E. Situated Learning: Legitimate Peripheral Participation (Cambridge Univ. Press, 1991).

  73. Kitcher, P. The division of cognitive labor. J. Philos. 87, 5–22 (1990).

    Article  Google Scholar 

  74. Hardwig, J. Epistemic dependence. J. Philos. 82, 335–349 (1985).

    Article  Google Scholar 

  75. Keil, F. in Oxford Studies In Epistemology (eds Gendler, T. S. & Hawthorne, J.) 143–166 (Oxford Academic, 2005).

  76. Weisberg, M. & Muldoon, R. Epistemic landscapes and the division of cognitive labor. Philos. Sci. 76, 225–252 (2009).

    Article  Google Scholar 

  77. Sloman, S. A. & Rabb, N. Your understanding is my understanding: evidence for a community of knowledge. Psychol. Sci. 27, 1451–1460 (2016).

    Article  PubMed  Google Scholar 

  78. Wilson, R. A. & Keil, F. The shadows and shallows of explanation. Minds Mach. 8, 137–159 (1998).

    Article  Google Scholar 

  79. Keil, F. C., Stein, C., Webb, L., Billings, V. D. & Rozenblit, L. Discerning the division of cognitive labor: an emerging understanding of how knowledge is clustered in other minds. Cogn. Sci. 32, 259–300 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  80. Sperber, D. et al. Epistemic vigilance. Mind Lang. 25, 359–393 (2010).

    Article  Google Scholar 

  81. Wilkenfeld, D. A., Plunkett, D. & Lombrozo, T. Depth and deference: when and why we attribute understanding. Philos. Stud. 173, 373–393 (2016).

    Article  Google Scholar 

  82. Sparrow, B., Liu, J. & Wegner, D. M. Google effects on memory: cognitive consequences of having information at our fingertips. Science 333, 776–778 (2011).

    Article  CAS  PubMed  ADS  Google Scholar 

  83. Fisher, M., Goddu, M. K. & Keil, F. C. Searching for explanations: how the internet inflates estimates of internal knowledge. J. Exp. Psychol. Gen. 144, 674–687 (2015).

    Article  PubMed  Google Scholar 

  84. De Freitas, J., Agarwal, S., Schmitt, B. & Haslam, N. Psychological factors underlying attitudes toward AI tools. Nat. Hum. Behav. 7, 1845–1854 (2023).

    Article  PubMed  Google Scholar 

  85. Castelo, N., Bos, M. W. & Lehmann, D. R. Task-dependent algorithm aversion. J. Mark. Res. 56, 809–825 (2019).

    Article  Google Scholar 

  86. Cadario, R., Longoni, C. & Morewedge, C. K. Understanding, explaining, and utilizing medical artificial intelligence. Nat. Hum. Behav. 5, 1636–1642 (2021).

    Article  PubMed  Google Scholar 

  87. Oktar, K. & Lombrozo, T. Deciding to be authentic: intuition is favored over deliberation when authenticity matters. Cognition 223, 105021 (2022).

    Article  PubMed  Google Scholar 

  88. Bigman, Y. E., Yam, K. C., Marciano, D., Reynolds, S. J. & Gray, K. Threat of racial and economic inequality increases preference for algorithm decision-making. Comput. Hum. Behav. 122, 106859 (2021).

    Article  Google Scholar 

  89. Claudy, M. C., Aquino, K. & Graso, M. Artificial intelligence can’t be charmed: the effects of impartiality on laypeople’s algorithmic preferences. Front. Psychol. 13, 898027 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  90. Snyder, C., Keppler, S. & Leider, S. Algorithm reliance under pressure: the effect of customer load on service workers. Preprint at SSRN https://doi.org/10.2139/ssrn.4066823 (2022).

  91. Bogert, E., Schecter, A. & Watson, R. T. Humans rely more on algorithms than social influence as a task becomes more difficult. Sci Rep. 11, 8028 (2021).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  92. Raviv, A., Bar‐Tal, D., Raviv, A. & Abin, R. Measuring epistemic authority: studies of politicians and professors. Eur. J. Personal. 7, 119–138 (1993).

    Article  Google Scholar 

  93. Cummings, L. The “trust” heuristic: arguments from authority in public health. Health Commun. 29, 1043–1056 (2014).

    Article  PubMed  ADS  Google Scholar 

  94. Lee, M. K. Understanding perception of algorithmic decisions: fairness, trust, and emotion in response to algorithmic management. Big Data Soc. 5, https://doi.org/10.1177/2053951718756684 (2018).

  95. Kissinger, H. A., Schmidt, E. & Huttenlocher, D. The Age of A.I. And Our Human Future (Little, Brown, 2021).

  96. Lombrozo, T. Explanatory preferences shape learning and inference. Trends Cogn. Sci. 20, 748–759 (2016). This paper provides an overview of philosophical theories of explanatory virtues and reviews empirical evidence on the sorts of explanations people find satisfying.

    Article  PubMed  Google Scholar 

  97. Vrantsidis, T. H. & Lombrozo, T. Simplicity as a cue to probability: multiple roles for simplicity in evaluating explanations. Cogn. Sci. 46, e13169 (2022).

    Article  PubMed  Google Scholar 

  98. Johnson, S. G. B., Johnston, A. M., Toig, A. E. & Keil, F. C. Explanatory scope informs causal strength inferences. In Proc. 36th Annual Meeting of the Cognitive Science Society 2453–2458 (Cognitive Science Society, 2014).

  99. Khemlani, S. S., Sussman, A. B. & Oppenheimer, D. M. Harry Potter and the sorcerer’s scope: latent scope biases in explanatory reasoning. Mem. Cognit. 39, 527–535 (2011).

    Article  PubMed  Google Scholar 

  100. Liquin, E. G. & Lombrozo, T. Motivated to learn: an account of explanatory satisfaction. Cogn. Psychol. 132, 101453 (2022).

    Article  PubMed  Google Scholar 

  101. Hopkins, E. J., Weisberg, D. S. & Taylor, J. C. V. The seductive allure is a reductive allure: people prefer scientific explanations that contain logically irrelevant reductive information. Cognition 155, 67–76 (2016).

    Article  PubMed  Google Scholar 

  102. Weisberg, D. S., Hopkins, E. J. & Taylor, J. C. V. People’s explanatory preferences for scientific phenomena. Cogn. Res. Princ. Implic. 3, 44 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  103. Jerez-Fernandez, A., Angulo, A. N. & Oppenheimer, D. M. Show me the numbers: precision as a cue to others’ confidence. Psychol. Sci. 25, 633–635 (2014).

    Article  PubMed  Google Scholar 

  104. Kim, J., Giroux, M. & Lee, J. C. When do you trust AI? The effect of number presentation detail on consumer trust and acceptance of AI recommendations. Psychol. Mark. 38, 1140–1155 (2021).

    Article  Google Scholar 

  105. Nguyen, C. T. The seductions of clarity. R. Inst. Philos. Suppl. 89, 227–255 (2021). This article describes how reductive and quantitative explanations can generate a sense of understanding that is not necessarily correlated with actual understanding.

    Article  Google Scholar 

  106. Fisher, M., Smiley, A. H. & Grillo, T. L. H. Information without knowledge: the effects of internet search on learning. Memory 30, 375–387 (2022).

    Article  Google Scholar 

  107. Eliseev, E. D. & Marsh, E. J. Understanding why searching the internet inflates confidence in explanatory ability. Appl. Cogn. Psychol. 37, 711–720 (2023).

    Article  Google Scholar 

  108. Fisher, M. & Oppenheimer, D. M. Who knows what? Knowledge misattribution in the division of cognitive labor. J. Exp. Psychol. Appl. 27, 292–306 (2021).

    Article  PubMed  Google Scholar 

  109. Chromik, M., Eiband, M., Buchner, F., Krüger, A. & Butz, A. I think I get your point, AI! The illusion of explanatory depth in explainable AI. In 26th International Conference on Intelligent User Interfaces (eds Hammond, T. et al.) 307–317 (Association for Computing Machinery, 2021).

  110. Strevens, M. No understanding without explanation. Stud. Hist. Philos. Sci. A 44, 510–515 (2013).

    Article  Google Scholar 

  111. Ylikoski, P. in Scientific Understanding: Philosophical Perspectives (eds De Regt, H. et al.) 100–119 (Univ. Pittsburgh Press, 2009).

  112. Giudice, M. D. The prediction–explanation fallacy: a pervasive problem in scientific applications of machine learning. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/4vq8f (2021).

  113. Hofman, J. M. et al. Integrating explanation and prediction in computational social science. Nature 595, 181–188 (2021). This paper highlights the advantages and disadvantages of explanatory versus predictive approaches to modelling, with a focus on applications to computational social science.

    Article  CAS  PubMed  ADS  Google Scholar 

  114. Shmueli, G. To explain or to predict? Stat. Sci. 25, 289–310 (2010).

    Article  MathSciNet  Google Scholar 

  115. Hofman, J. M., Sharma, A. & Watts, D. J. Prediction and explanation in social systems. Science 355, 486–488 (2017).

    Article  CAS  PubMed  ADS  Google Scholar 

  116. Logg, J. M., Minson, J. A. & Moore, D. A. Algorithm appreciation: people prefer algorithmic to human judgment. Organ. Behav. Hum. Decis. Process. 151, 90–103 (2019).

    Article  Google Scholar 

  117. Nguyen, C. T. Cognitive islands and runaway echo chambers: problems for epistemic dependence on experts. Synthese 197, 2803–2821 (2020).

    Article  Google Scholar 

  118. Breiman, L. Statistical modeling: the two cultures. Stat. Sci. 16, 199–215 (2001).

    Article  MathSciNet  Google Scholar 

  119. Gao, J. & Wang, D. Quantifying the benefit of artificial intelligence for scientific research. Preprint at arxiv.org/abs/2304.10578 (2023).

  120. Hanson, B. et al. Garbage in, garbage out: mitigating risks and maximizing benefits of AI in research. Nature 623, 28–31 (2023).

    Article  CAS  PubMed  ADS  Google Scholar 

  121. Kleinberg, J. & Raghavan, M. Algorithmic monoculture and social welfare. Proc. Natl Acad. Sci. USA 118, e2018340118 (2021). This paper uses formal modelling methods to demonstrate that when companies all rely on the same algorithm to make decisions (an algorithmic monoculture), the overall quality of those decisions is reduced because valuable options can slip through the cracks, even when the algorithm performs accurately for individual companies.

    Article  MathSciNet  CAS  PubMed  PubMed Central  Google Scholar 

  122. Hofstra, B. et al. The diversity–innovation paradox in science. Proc. Natl Acad. Sci. USA 117, 9284–9291 (2020).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  123. Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Natl Acad. Sci. USA 101, 16385–16389 (2004).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  124. Page, S. E. Where diversity comes from and why it matters? Eur. J. Soc. Psychol. 44, 267–279 (2014). This article reviews research demonstrating the benefits of cognitive diversity and diversity in methodological approaches for problem solving and innovation.

    Article  Google Scholar 

  125. Clarke, A. E. & Fujimura, J. H. (eds) The Right Tools for the Job: At Work in Twentieth-Century Life Sciences (Princeton Univ. Press, 2014).

  126. Silva, V. J., Bonacelli, M. B. M. & Pacheco, C. A. Framing the effects of machine learning on science. AI Soc. https://doi.org/10.1007/s00146-022-01515-x (2022).

  127. Sassenberg, K. & Ditrich, L. Research in social psychology changed between 2011 and 2016: larger sample sizes, more self-report measures, and more online studies. Adv. Methods Pract. Psychol. Sci. 2, 107–114 (2019).

    Article  Google Scholar 

  128. Simon, A. F. & Wilder, D. Methods and measures in social and personality psychology: a comparison of JPSP publications in 1982 and 2016. J. Soc. Psychol. https://doi.org/10.1080/00224545.2022.2135088 (2022).

    Article  PubMed  Google Scholar 

  129. Anderson, C. A. et al. The MTurkification of social and personality psychology. Pers. Soc. Psychol. Bull. 45, 842–850 (2019).

    Article  PubMed  Google Scholar 

  130. Latour, B. in The Social After Gabriel Tarde: Debates and Assessments (ed. Candea, M.) 145–162 (Routledge, 2010).

  131. Porter, T. M. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life (Princeton Univ. Press, 1996).

  132. Lazer, D. et al. Meaningful measures of human society in the twenty-first century. Nature 595, 189–196 (2021).

    Article  CAS  PubMed  ADS  Google Scholar 

  133. Knox, D., Lucas, C. & Cho, W. K. T. Testing causal theories with learned proxies. Annu. Rev. Polit. Sci. 25, 419–441 (2022).

    Article  Google Scholar 

  134. Barberá, P. Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data. Polit. Anal. 23, 76–91 (2015).

    Article  Google Scholar 

  135. Brady, W. J., McLoughlin, K., Doan, T. N. & Crockett, M. J. How social learning amplifies moral outrage expression in online social networks. Sci. Adv. 7, eabe5641 (2021).

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  136. Barnes, J., Klinger, R. & im Walde, S. S. Assessing state-of-the-art sentiment models on state-of-the-art sentiment datasets. In Proc. 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (eds Balahur, A. et al.) 2–12 (Association for Computational Linguistics, 2017).

  137. Gitelman, L. (ed.) “Raw Data” is an Oxymoron (MIT Press, 2013).

  138. Breznau, N. et al. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proc. Natl Acad. Sci. USA 119, e2203150119 (2022). This study demonstrates how 73 research teams analysing the same dataset reached different conclusions about the relationship between immigration and public support for social policies, highlighting the subjectivity and uncertainty involved in analysing complex datasets.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  139. Gillespie, T. in Media Technologies: Essays on Communication, Materiality, and Society (eds Gillespie, T. et al.) 167–194 (MIT Press, 2014).

  140. Leonelli, S. Data-Centric Biology: A Philosophical Study (Univ. Chicago Press, 2016).

  141. Wang, A., Kapoor, S., Barocas, S. & Narayanan, A. Against predictive optimization: on the legitimacy of decision-making algorithms that optimize predictive accuracy. ACM J. Responsib. Comput., https://doi.org/10.1145/3636509 (2023).

  142. Athey, S. Beyond prediction: using big data for policy problems. Science 355, 483–485 (2017).

    Article  CAS  PubMed  ADS  Google Scholar 

  143. del Rosario Martínez-Ordaz, R. Scientific understanding through big data: from ignorance to insights to understanding. Possibility Stud. Soc. 1, 279–299 (2023).

    Article  Google Scholar 

  144. Nussberger, A.-M., Luo, L., Celis, L. E. & Crockett, M. J. Public attitudes value interpretability but prioritize accuracy in artificial intelligence. Nat. Commun. 13, 5821 (2022).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  145. Zittrain, J. in The Cambridge Handbook of Responsible Artificial Intelligence: Interdisciplinary Perspectives (eds. Voeneky, S. et al.) 176–184 (Cambridge Univ. Press, 2022). This article articulates the epistemic risks of prioritizing predictive accuracy over explanatory understanding when AI tools are interacting in complex systems.

  146. Shumailov, I. et al. The curse of recursion: training on generated data makes models forget. Preprint at arxiv.org/abs/2305.17493 (2023).

  147. Latour, B. Science In Action: How to Follow Scientists and Engineers Through Society (Harvard Univ. Press, 1987). This book provides strategies and approaches for thinking about science as a social endeavour.

  148. Franklin, S. Science as culture, cultures of science. Annu. Rev. Anthropol. 24, 163–184 (1995).

    Article  Google Scholar 

  149. Haraway, D. Situated knowledges: the science question in feminism and the privilege of partial perspective. Fem. Stud. 14, 575–599 (1988). This article acknowledges that the objective ‘view from nowhere’ is unobtainable: knowledge, it argues, is always situated.

    Article  Google Scholar 

  150. Harding, S. Objectivity and Diversity: Another Logic of Scientific Research (Univ. Chicago Press, 2015).

  151. Longino, H. E. Science as Social Knowledge: Values and Objectivity in Scientific Inquiry (Princeton Univ. Press, 1990).

  152. Daston, L. & Galison, P. Objectivity (Princeton Univ. Press, 2007). This book is a historical analysis of the shifting modes of ‘objectivity’ that scientists have pursued, arguing that objectivity is not a universal concept but that it shifts alongside scientific techniques and ambitions.

  153. Prescod-Weinstein, C. Making Black women scientists under white empiricism: the racialization of epistemology in physics. Signs J. Women Cult. Soc. 45, 421–447 (2020).

    Article  Google Scholar 

  154. Mavhunga, C. What Do Science, Technology, and Innovation Mean From Africa? (MIT Press, 2017).

  155. Schiebinger, L. The Mind Has No Sex? Women in the Origins of Modern Science (Harvard Univ. Press, 1991).

  156. Martin, E. The egg and the sperm: how science has constructed a romance based on stereotypical male–female roles. Signs J. Women Cult. Soc. 16, 485–501 (1991). This case study shows how assumptions about gender affect scientific theories, sometimes delaying the articulation of what might be considered to be more accurate descriptions of scientific phenomena.

    Article  Google Scholar 

  157. Harding, S. Rethinking standpoint epistemology: What is “strong objectivity”? Centen. Rev. 36, 437–470 (1992). In this article, Harding outlines her position on ‘strong objectivity’, by which clearly articulating one’s standpoint can lead to more robust knowledge claims.

    Google Scholar 

  158. Oreskes, N. Why Trust Science? (Princeton Univ. Press, 2019). This book introduces the reader to 20 years of scholarship in science and technology studies, arguing that the tools the discipline has for understanding science can help to reinstate public trust in the institution.

  159. Rolin, K., Koskinen, I., Kuorikoski, J. & Reijula, S. Social and cognitive diversity in science: introduction. Synthese 202, 36 (2023).

    Article  Google Scholar 

  160. Hong, L. & Page, S. E. Problem solving by heterogeneous agents. J. Econ. Theory 97, 123–163 (2001).

    Article  MathSciNet  Google Scholar 

  161. Sulik, J., Bahrami, B. & Deroy, O. The diversity gap: when diversity matters for knowledge. Perspect. Psychol. Sci. 17, 752–767 (2022).

    Article  PubMed  Google Scholar 

  162. Lungeanu, A., Whalen, R., Wu, Y. J., DeChurch, L. A. & Contractor, N. S. Diversity, networks, and innovation: a text analytic approach to measuring expertise diversity. Netw. Sci. 11, 36–64 (2023).

    Article  Google Scholar 

  163. AlShebli, B. K., Rahwan, T. & Woon, W. L. The preeminence of ethnic diversity in scientific collaboration. Nat. Commun. 9, 5163 (2018).

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  164. Campbell, L. G., Mehtani, S., Dozier, M. E. & Rinehart, J. Gender-heterogeneous working groups produce higher quality science. PLoS ONE 8, e79147 (2013).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  165. Nielsen, M. W., Bloch, C. W. & Schiebinger, L. Making gender diversity work for scientific discovery and innovation. Nat. Hum. Behav. 2, 726–734 (2018).

    Article  PubMed  Google Scholar 

  166. Yang, Y., Tian, T. Y., Woodruff, T. K., Jones, B. F. & Uzzi, B. Gender-diverse teams produce more novel and higher-impact scientific ideas. Proc. Natl Acad. Sci. USA 119, e2200841119 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  167. Kozlowski, D., Larivière, V., Sugimoto, C. R. & Monroe-White, T. Intersectional inequalities in science. Proc. Natl Acad. Sci. USA 119, e2113067119 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  168. Fehr, C. & Jones, J. M. Culture, exploitation, and epistemic approaches to diversity. Synthese 200, 465 (2022).

    Article  MathSciNet  Google Scholar 

  169. Nakadai, R., Nakawake, Y. & Shibasaki, S. AI language tools risk scientific diversity and innovation. Nat. Hum. Behav. 7, 1804–1805 (2023).

    Article  PubMed  Google Scholar 

  170. National Academies of Sciences, Engineering, and Medicine et al. Advancing Antiracism, Diversity, Equity, and Inclusion in STEMM Organizations: Beyond Broadening Participation (National Academies Press, 2023).

  171. Winner, L. Do artifacts have politics? Daedalus 109, 121–136 (1980).

    Google Scholar 

  172. Eubanks, V. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor (St. Martin’s Press, 2018).

  173. Littmann, M. et al. Validity of machine learning in biology and medicine increased through collaborations across fields of expertise. Nat. Mach. Intell. 2, 18–24 (2020).

    Article  Google Scholar 

  174. Carusi, A. et al. Medical artificial intelligence is as much social as it is technological. Nat. Mach. Intell. 5, 98–100 (2023).

    Article  Google Scholar 

  175. Raghu, M. & Schmidt, E. A survey of deep learning for scientific discovery. Preprint at arxiv.org/abs/2003.11755 (2020).

  176. Bishop, C. AI4Science to empower the fifth paradigm of scientific discovery. Microsoft Research Blog www.microsoft.com/en-us/research/blog/ai4science-to-empower-the-fifth-paradigm-of-scientific-discovery/ (2022).

  177. Whittaker, M. The steep cost of capture. Interactions 28, 50–55 (2021).

    Article  Google Scholar 

  178. Liesenfeld, A., Lopez, A. & Dingemanse, M. Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators. In Proc. 5th International Conference on Conversational User Interfaces 1–6 (Association for Computing Machinery, 2023).

  179. Chu, J. S. G. & Evans, J. A. Slowed canonical progress in large fields of science. Proc. Natl Acad. Sci. USA 118, e2021636118 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  180. Park, M., Leahey, E. & Funk, R. J. Papers and patents are becoming less disruptive over time. Nature 613, 138–144 (2023).

    Article  CAS  PubMed  ADS  Google Scholar 

  181. Frith, U. Fast lane to slow science. Trends Cogn. Sci. 24, 1–2 (2020). This article explains the epistemic risks of a hyperfocus on scientific productivity and explores possible avenues for incentivizing the production of higher-quality science on a slower timescale.

    Article  PubMed  Google Scholar 

  182. Stengers, I. Another Science is Possible: A Manifesto for Slow Science (Wiley, 2018).

  183. Lake, B. M. & Baroni, M. Human-like systematic generalization through a meta-learning neural network. Nature 623, 115–121 (2023).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  184. Feinman, R. & Lake, B. M. Learning task-general representations with generative neuro-symbolic modeling. Preprint at arxiv.org/abs/2006.14448 (2021).

  185. Schölkopf, B. et al. Toward causal representation learning. Proc. IEEE 109, 612–634 (2021).

    Article  Google Scholar 

  186. Mitchell, M. AI’s challenge of understanding the world. Science 382, eadm8175 (2023).

    Article  PubMed  Google Scholar 

  187. Sartori, L. & Bocca, G. Minding the gap(s): public perceptions of AI and socio-technical imaginaries. AI Soc. 38, 443–458 (2023).

    Article  Google Scholar 

Download references

Acknowledgements

We thank D. S. Bassett, W. J. Brady, S. Helmreich, S. Kapoor, T. Lombrozo, A. Narayanan, M. Salganik and A. J. te Velthuis for comments. We also thank C. Buckner and P. Winter for their feedback and suggestions.

Author information

Author notes

  1. These authors contributed equally: Lisa Messeri, M. J. Crockett

Authors and Affiliations

  1. Department of Anthropology, Yale University, New Haven, CT, USA

    Lisa Messeri

  2. Department of Psychology, Princeton University, Princeton, NJ, USA

    M. J. Crockett

  3. University Center for Human Values, Princeton University, Princeton, NJ, USA

    M. J. Crockett

Authors
  1. Lisa Messeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. J. Crockett
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to the research and writing of the paper.

Corresponding authors

Correspondence to Lisa Messeri or M. J. Crockett.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature thanks Cameron Buckner, Peter Winter and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Messeri, L., Crockett, M.J. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024). https://doi.org/10.1038/s41586-024-07146-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-024-07146-0

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing