Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

Assessment of the clinical knowledge of ChatGPT-4 in neonatal-perinatal medicine: a comparative analysis with ChatGPT-3.5

Journal of Perinatology (2024)Cite this article

Subjects

Large language models (LLMs) have demonstrated promising performance on clinical knowledge tasks, including the United States Medical Licensing Examination and subspecialty board examinations [1,2,3,4]. Previous work from our group assessed the performance of ChatGPT-3.5 on practice questions for the neonatal-perinatal medicine board examination and found that it performed below a passing rate [3]. Recent results, however, have shown that a new version, GPT-4, performs substantially better than GPT-3.5 on board questions in other medical fields [1]. Therefore, we conducted a comparative analysis of the performance of GPT-4 and GPT-3.5 on practice questions for the neonatal-perinatal medicine board examination.

We compiled questions from a neonatal-perinatal medicine board examination preparation book and excluded questions that were non-multiple-choice format or had figures as GPT-3.5 does not support visual inputs [5]. This yielded 926 questions, which was sufficient to detect an effect size of 20% on paired t-test (α = 0.05, β = 0.2). Each eligible question was entered into the application program interface using the same prompt for both versions. We instructed the LLM that it is the neonatal expert in the specific domain of the question and asked it to “take a deep breath and solve the following question.” We used standard settings on both versions, including a temperature of zero.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Mihalache A, Huang RS, Popovic MM, Muni RH. ChatGPT-4: an assessment of an upgraded artificial intelligence chatbot in the United States Medical Licensing Examination. Med Teach. 2023. https://doi.org/10.1080/0142159X.2023.2249588.

    Article  PubMed  Google Scholar 

  2. Beam K, Sharma P, Levy P, Beam AL. Artificial intelligence in the neonatal intensive care unit: the time is now. J Perinatol; https://doi.org/10.1227/neu.0000000000002551.

  3. Beam K, Sharma P, Kumar B, Wang C, Brodsky D, Martin CR, et al. Performance of a large language model on practice questions for the neonatal board examination. JAMA Pediatr. 2023;177:977–9.

    Article  PubMed  Google Scholar 

  4. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2:e0000198.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Morton S, Ehret D, Ghanta S, Sajti E, Walsh B. Neonatology review: Q&A. 3rd ed. Morrisville (US): Lulu; 2015.

Download references

Acknowledgements

Use of LLMs: Given the aim of this investigation, we used LLMs, specifically ChatGPT-3.5 and ChatGPT-4, in the data collection and analysis. The details of their use are summarized in the text. However, no LLMs were used in the writing or editing of the report.

Funding

This work was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (5T32HD098061).

Author information

Author notes

  1. These authors contributed equally: Puneet Sharma, Guangze Luo, Andrew Beam, Kristyn Beam.

Authors and Affiliations

  1. Division of Newborn Medicine, Boston Children’s Hospital, Boston, MA, USA

    Puneet Sharma

  2. Department of Epidemiology, Harvard T.H. Chan School of Public Health, Cambridge, MA, USA

    Guangze Luo & Andrew Beam

  3. Department of Statistics, Harvard University, Cambridge, MA, USA

    Cindy Wang

  4. Department of Neonatology, Beth Israel Deaconess Medical Center, Boston, MA, USA

    Dara Brodsky & Kristyn Beam

  5. Division of Neonatology, Weill Cornell Medicine, New York City, NY, USA

    Camilia R. Martin

Authors
  1. Puneet Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangze Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cindy Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dara Brodsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Camilia R. Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andrew Beam
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kristyn Beam
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

PS, CW, AB, and KB conceived of the project. PS, CW, GL, DB, and CM were responsible for data collection and analysis. PS drafted initial manuscript. All authors reviewed the final manuscript. PS, GL, AB, and KB contributed equally to this project.

Corresponding author

Correspondence to Puneet Sharma.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, P., Luo, G., Wang, C. et al. Assessment of the clinical knowledge of ChatGPT-4 in neonatal-perinatal medicine: a comparative analysis with ChatGPT-3.5. J Perinatol (2024). https://doi.org/10.1038/s41372-024-01912-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41372-024-01912-8

Search

Advanced search

Quick links