Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

clinical

Quality of information and appropriateness of Open AI outputs for prostate cancer

Prostate Cancer and Prostatic Diseases (2024)Cite this article

Subjects

Abstract

Chat-GPT, a natural language processing (NLP) tool created by Open-AI, can potentially be used as a quick source for obtaining information related to prostate cancer. This study aims to analyze the quality and appropriateness of Chat-GPT’s responses to inquiries related to prostate cancer compared to those of the European Urology Association’s (EAU) 2023 prostate cancer guidelines. Overall, 195 questions were prepared according to the recommendations gathered in the prostate cancer section of the EAU 2023 Guideline. All questions were systematically presented to Chat-GPT’s August 3 Version, and two expert urologists independently assessed and assigned scores ranging from 1 to 4 to each response (1: completely correct, 2: correct but inadequate, 3: a mix of correct and misleading information, and 4: completely incorrect). Sub-analysis per chapter and per grade of recommendation were performed. Overall, 195 recommendations were evaluated. Overall, 50/195 (26%) were completely correct, 51/195 (26%) correct but inadequate, 47/195 (24%) a mix of correct and misleading and 47/195 (24%) incorrect. When looking at different chapters Open AI was particularly accurate in answering questions on follow-up and QoL. Worst performance was recorded for the diagnosis and treatment chapters with respectively 19% and 30% of the answers completely incorrect. When looking at the strength of recommendation, no differences in terms of accuracy were recorded when comparing weak and strong recommendations (p > 0,05). Chat-GPT has a poor accuracy when answering questions on the PCa EAU guidelines recommendations. Future studies should assess its performance after adequate training.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Data availability

Data are available upon request.

Material availability

Material are available upon request.

References

  1. Culp MBB, Soerjomataram I, Efstathiou JA, Bray F, Jemal A. Recent global patterns in prostate cancer incidence and mortality rates. Eur Urol. 2020;77:38–52.

  2. Hamdy FC, Donovan JL, Lane JA, Metcalfe C, Davis M, Turner EL, et al. Fifteen-Year outcomes after monitoring, surgery, or radiotherapy for prostate cancer. N Engl J Med. 2023;388:1547–58.

    Article  PubMed  Google Scholar 

  3. Lombardo R, De Nunzio C. Nomograms in PCa: where do we stand. Prostate Cancer Prostatic Dis. 2023;26:447–8.

    Article  PubMed  Google Scholar 

  4. Checcucci E, Rosati S, De Cillis S, Vagni M, Giordano N, Piana A, et al. Artificial intelligence for target prostate biopsy outcomes prediction the potential application of fuzzy logic. Prostate Cancer Prostatic Dis. 2022;25:359–62.

    Article  PubMed  Google Scholar 

  5. Ditonno F, Franco A, Manfredi C, Veccia A, Valerio M, Bukavina L, et al. Novel non-MRI imaging techniques for primary diagnosis of prostate cancer: micro-ultrasound, contrast-enhanced ultrasound, elastography, multiparametric ultrasound, and PSMA PET/CT. Prostate Cancer Prostatic Dis. 2023. https://doi.org/10.1038/s41391-023-00708-9. Epub ahead of print.

  6. Eppler M, Ganjavi C, Ramacciotti LS, Piazza P, Rodler S, Checcucci E, et al. Awareness and use of ChatGPT and large language models: a prospective cross-sectional global survey in urology. Eur Urol. 2023. https://linkinghub.elsevier.com/retrieve/pii/S0302283823032116

  7. Cocci A, Pezzoli M, Lo Re M, Russo GI, Asmundo MG, Fode M, et al. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer Prostatic Dis. 2023. https://doi.org/10.1038/s41391-023-00754-3. Epub ahead of print.

  8. Lim DYZ, Tan YB, Koh JTE, Tung JYM, Sng GGR, Tan DMY, et al. ChatGPT on guidelines: providing contextual knowledge to GPT allows it to provide advice on appropriate colonoscopy intervals. J Gastroenterol Hepatol. 2023. https://doi.org/10.1111/jgh.16375. Epub ahead of print.

  9. Adhikari K, Naik N, Hameed BZ, Raghunath SK, Somani BK. Exploring the ethical, legal, and social implications of ChatGPT in urology. Curr Urol Rep. 2024;25:1–8.

  10. Daungsupawong H, Wiwanitkit V. Social determinants of health into evaluations of quality and appropriateness of AI assistant ChatGPT. Prostate Cancer Prostatic Dis. 2023. https://doi.org/10.1038/s41391-023-00735-6. Epub ahead of print.

  11. Lombardo R, Cicione A, Santoro G, De Nunzio C. ChatGPT in prostate cancer: myth or reality? Prostate Cancer Prostatic Dis. 2023. Available from: https://www.nature.com/articles/s41391-023-00750-7

  12. Morozov A, Taratkin M, Bazarkin A, Rivas JG, Puliatti S, Checcucci E, et al. A systematic review and meta-analysis of artificial intelligence diagnostic accuracy in prostate cancer histology identification and grading. Prostate Cancer Prostatic Dis. 2023;26:681–92.

    Article  PubMed  Google Scholar 

  13. EAU-EANM-ESTRO-ESUR-ISUP-SIOG-Guidelines-on-Prostate-Cancer-2023. EAU Guidelines. Edn. presented at the EAU Annual Congress Milan 2023.

  14. Cakir H, Caglar U, Yildiz O, Meric A, Ayranci A, Ozgor F. Evaluating the performance of ChatGPT in answering questions related to urolithiasis. Int Urol Nephrol. 2023;56:17–21.

  15. Baydoun A, Jia AY, Zaorsky NG, Kashani R, Rao S, Shoag JE, et al. Artificial intelligence applications in prostate cancer. Prostate Cancer Prostatic Dis. 2023. https://doi.org/10.1038/s41391-023-00684-0. In Press.

  16. Di H, Wen Y. Will generalist medical artificial intelligence be the future path for health-related natural language processing models? Prostate Cancer Prostatic Dis. 2023. https://doi.org/10.1038/s41391-023-00719-6. In Press.

  17. Drazen JM, Kohane IS, Leong T-Y, Lee P, Bubeck S, Petro J, et al. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. N Eng J Med. 2023. https://doi.org/10.1056/NEJMc2305286. In Press.

  18. Nicoletti R, Nicoletti G, Giannini V, Teoh JYC. Developers-Doctor-patients: the artificial intelligence’s trifecta. Prostate Cancer Prostatic Dis. 2023. https://doi.org/10.1038/s41391-023-00718-7. In Press.

  19. Coskun B, Ocakoglu G, Yetemen M, Kaygisiz O. Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer? Urology. 2023;180:35–58.

    Article  PubMed  Google Scholar 

  20. Whiles BB, Bird VG, Canales BK, DiBianco JM, Terry RS. Caution! AI Bot has entered the patient chat: ChatGPT has limitations in providing accurate urologic healthcare advice. Urology. 2023;180:278–84.

    Article  PubMed  Google Scholar 

  21. Goodman RS, Patrinely JR, Stone CA, Zimmerman E, Donald RR, Chang SS, et al. Accuracy and reliability of chatbot responses to physician questions. JAMA Netw Open. 2023;6:e2336483.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Musheyev D, Pan A, Loeb S, Kabarriti AE. How well do artificial intelligence chatbots respond to the top search queries about urological malignancies? Eur Urol. 2023;85:13–16.

    Article  PubMed  Google Scholar 

  23. Wang H, Xia Z, Xu Y, Sun J, Wu J. The predictive value of machine learning and nomograms for lymph node metastasis of prostate cancer: a systematic review and meta-analysis. Prostate Cancer Prostatic Dis. 2023;26:602–13.

    Article  PubMed  Google Scholar 

  24. Manolitsis I, Feretzakis G, Tzelves L, Kalles D, Katsimperis S, Angelopoulos P, et al. Training ChatGPT models in assisting urologists in daily practice. Stud Health Technol Inform. 2023;305:576–9.

Download references

Author information

Authors and Affiliations

  1. Department of Urology, ‘Sapienza’ University of Rome, Rome, Italy

    Riccardo Lombardo, Giacomo Gallo, Jordi Stira, Beatrice Turchi, Giuseppe Santoro, Sara Riolo, Matteo Romagnoli, Antonio Cicione, Giorgia Tema, Antonio Pastore, Yazan Al Salhi, Andrea Fuschi, Giorgio Franco, Antonio Nacchia, Andrea Tubaro & Cosimo De Nunzio

Authors
  1. Riccardo Lombardo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giacomo Gallo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jordi Stira
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Beatrice Turchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Giuseppe Santoro
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sara Riolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Matteo Romagnoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Antonio Cicione
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Giorgia Tema
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Antonio Pastore
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Yazan Al Salhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Andrea Fuschi
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Giorgio Franco
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Antonio Nacchia
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Andrea Tubaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Cosimo De Nunzio
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CDN: Protocol/project development, Data collection or management, Data analysis, Manuscript writing/editing. AC: Data collection or management, Manuscript writing/editing. JS: Data collection or management, Data analysis, Manuscript writing/editing. GS: Data collection or management, Manuscript writing/editing. RL Data collection or management, Manuscript writing/editing, Data analysis. SR: Data collection or management, Manuscript writing/editing, Data analysis. MR: Data collection or management, Manuscript writing/editing, Data analysis. BT: Data collection or management, Manuscript writing/editing. GT: Data collection or management, Manuscript writing/editing. YAS: Data collection or management, Manuscript writing/editing. AF: Data collection or management, Manuscript writing/editing. AP: Protocol/project development, Data collection or management, Data analysis, Manuscript writing/editing. GG: Data collection or management, Manuscript writing/editing, Data analysis. AN: Data collection or management, Manuscript writing/editing. AT Protocol/project development, Data collection or management, Data analysis, Manuscript writing/editing. All authors reviewed and approved the final version of the manuscript.

Corresponding author

Correspondence to Riccardo Lombardo.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethical approval

The study was approved by a local ethical committee and was conducted in accordance with the principles of the Declaration of Helsinki.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lombardo, R., Gallo, G., Stira, J. et al. Quality of information and appropriateness of Open AI outputs for prostate cancer. Prostate Cancer Prostatic Dis (2024). https://doi.org/10.1038/s41391-024-00789-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41391-024-00789-0

Search

Advanced search

Quick links