Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

The Turing test is not a good benchmark for thought in LLMs

Nature Human Behaviour volume 7, pages 1806–1807 (2023)Cite this article

Subjects

Named in honour of its originator Alan Turing, the test asks whether an interrogator can distinguish a human being from a machine on the basis of their respective answers to a series of questions, the content of which is determined by the interrogator. A machine passes the test if, after a sequence of such games, the interrogator is unable to reliably distinguish the machine from the human player. Turing did not claim that we should treat his test as a necessary condition for machine thought, but he did suggest that it would be reasonable to regard any machine that could fool an interrogator into thinking it was human as a thinker.

The question of whether LLMs have passed the Turing test is moot, as Turing left many of the game’s parameters unspecified — including how many questions the interrogator is allowed to ask (50? 100? 1,000?), what kind of background knowledge the interrogator is allowed, whether the interrogator is aware that one of the players is a machine (and, if so, what kind of machine it is), what kind of background knowledge the human player is allowed, and whether the human player knows that their competitor is a machine (and, if so, what kind of machine it is).

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Turing, A. Mind LIX, 433–460 (1950).

    Article  Google Scholar 

  2. BIG-bench collaboration. Beyond the imitation game: measuring and extrapolating the capabilities of language models. github.com, https://github.com/google/BIG-bench (2021).

  3. Block, N. Philos. Rev. 90, 5–43 (1981).

    Article  Google Scholar 

  4. Dennett, D. in Alan Turing: Life and Legacy of a Great Thinker (ed. Teuscher, C.) 295–316 (Springer, 2004).

  5. Gröndahl, T., Guo, Y. & Asokan, N. Preprint at ArXiv, https://doi.org/10.48550/arXiv.2203.00162 (2022).

  6. Mitchell, M. & Krakauer, D. C. Proc. Natl Acad. Sci. USA 120, e2215907120 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Butlin, P. Erkenntnis 88, 3079–3095 (2023).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Philosophy, School of Philosophical, Historical and International Studies, Monash University, Melbourne, Victoria, Australia

    Tim Bayne & Iwan Williams

  2. Brain, Mind and Consciousness Program, Canadian Institute for Advanced Research (CIFAR), Toronto, Ontario, Canada

    Tim Bayne

  3. Monash Centre for Consciousness & Contemplative Studies (M3CS), Monash University, Melbourne, Victoria, Australia

    Iwan Williams

Authors
  1. Tim Bayne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Iwan Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Bayne.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bayne, T., Williams, I. The Turing test is not a good benchmark for thought in LLMs. Nat Hum Behav 7, 1806–1807 (2023). https://doi.org/10.1038/s41562-023-01710-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-023-01710-w

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing