Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

External validation of a deep learning detection system for glaucomatous optic neuropathy: a real-world multicentre study

Eye volume 37, pages 3813–3818 (2023)Cite this article

Subjects

Abstract

Objectives

To conduct an external validation of an automated artificial intelligence (AI) diagnostic system using fundus photographs from a real-life multicentre cohort.

Methods

We designed external validation in multiple scenarios, consisting of 3049 images from Qilu Hospital of Shandong University in China (QHSDU, validation dataset 1), 7495 images from three other hospitals in China (validation dataset 2), and 516 images from high myopia (HM) population of QHSDU (validation dataset 3). The corresponding sensitivity, specificity and accuracy of this AI diagnostic system to identify glaucomatous optic neuropathy (GON) were calculated.

Results

In validation datasets 1 and 2, the algorithm yielded accuracy of 93.18% and 91.40%, area under the receiver operating curves (AUC) of 95.17% and 96.64%, and significantly higher sensitivity of 91.75% and 91.41%, respectively, compared to manual graders. On the subsets complicated with retinal comorbidities, such as diabetic retinopathy or age-related macular degeneration, in validation datasets 1 and 2, the algorithm achieved accuracy of 87.54% and 93.81%, and AUC of 97.02% and 97.46%, respectively. In validation dataset 3, the algorithm achieved comparable accuracy of 81.98% and AUC of 87.49%, with a sensitivity of 83.61% and specificity of 81.76% on GON recognition specifically in the HM population.

Conclusions

With acceptable generalization capability across varying levels of image quality, different clinical centres, or certain retinal comorbidities, such as HM, the automatic AI diagnostic system had the potential to provide expert-level glaucoma detection.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The receiver operating characteristic analysis of GON classification performance in validation datasets.

Similar content being viewed by others

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

  1. Jonas JB, Aung T, Bourne RR, Bron AM, Ritch R, Panda-Jonas S. Glaucoma. Lancet. 2017;390:2183–93.

    Article  PubMed  Google Scholar 

  2. Zhang Y, Wang N, Liu H. Applications of Artificial Intelligence in the Screening of Glaucoma in China. J Med Syst. 2020;44:124.

    Article  PubMed  Google Scholar 

  3. Tham YC, Li X, Wong TY, Quigley HA, Aung T, Cheng CY. Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. Ophthalmology. 2014;121:2081–90.

    Article  PubMed  Google Scholar 

  4. Li JO, Liu H, Ting DSJ, Jeon S, Chan RVP, Kim JE, et al. Digital technology, tele-medicine and artificial intelligence in ophthalmology: a global perspective. Prog Retin Eye Res. 2021;82:100900.

    Article  CAS  PubMed  Google Scholar 

  5. Liang YB, Friedman DS, Zhou Q, Yang X, Sun LP, Guo LX, et al. Prevalence of primary open angle glaucoma in a rural adult Chinese population: the Handan eye study. Invest Ophthalmol Vis Sci. 2011;52:8250–7.

    Article  PubMed  Google Scholar 

  6. Song W, Shan L, Cheng F, Fan P, Zhang L, Qu W, et al. Prevalence of glaucoma in a rural northern china adult population: a population-based survey in kailu county, inner mongolia. Ophthalmology. 2011;118:1982–8.

    Article  PubMed  Google Scholar 

  7. Kwon S, Kim SH, Khang D, Lee JY. Potential therapeutic usage of nanomedicine for glaucoma treatment. Int J Nanomed. 2020;15:5745–65.

    Article  CAS  Google Scholar 

  8. Mayro EL, Wang M, Elze T, Pasquale LR. The impact of artificial intelligence in the diagnosis and management of glaucoma. Eye (Lond). 2020;34:1–11.

    Article  PubMed  Google Scholar 

  9. Li Z, He Y, Keel S, Meng W, Chang RT, He M. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology. 2018;125:1199–206.

    Article  PubMed  Google Scholar 

  10. Liu H, Li L, Wormstone IM, Qiao C, Zhang C, Liu P, et al. Development and validation of a deep learning system to detect glaucomatous optic neuropathy using fundus photographs. JAMA Ophthalmol. 2019;137:1353–60.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Jammal AA, Thompson AC, Mariottoni EB, Berchuck SI, Urata CN, Estrela T, et al. Human versus machine: comparing a deep learning algorithm to human gradings for detecting glaucoma on fundus photographs. Am J Ophthalmol. 2020;211:123–31.

    Article  PubMed  Google Scholar 

  12. Mehta P, Petersen CA, Wen JC, Banitt MR, Chen PP, Bojikian KD, et al. Automated detection of glaucoma with interpretable machine learning using clinical data and multimodal retinal images. Am J Ophthalmol. 2021;231:154–69.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Sudhan MB, Sinthuja M, Pravinth Raja S, Amutharaj J, Charlyn Pushpa Latha G, Sheeba Rachel S, et al. Segmentation and classification of glaucoma using u-net with deep learning model. J Health Eng. 2022;2022:1601354.

    Article  CAS  Google Scholar 

  14. Gomez-Valverde JJ, Anton A, Fatti G, Liefers B, Herranz A, Santos A, et al. Automatic glaucoma classification using color fundus images based on convolutional neural networks and transfer learning. Biomed Opt Express. 2019;10:892–913.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Diaz-Pinto A, Morales S, Naranjo V, Kohler T, Mossi JM, Navea A. CNNs for automatic glaucoma assessment using fundus images: an extensive validation. Biomed Eng Online. 2019;18:29.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Birkenbihl C, Emon MA, Vrooman H, Westwood S, Lovestone S, AddNeuroMed C, et al. Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice. EPMA J. 2020;11:367–76.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Zhang Y, Y S, Ma K, Chu C, Zhang L, Pang R, et al. The application of artificial intelligence multi-task deep learning model of optic disc area in the classification of glaucoma. Chin J Ophthalmologic Med (Electron Ed). 2020;10:6.

    Google Scholar 

  18. Phene S, Dunn RC, Hammel N, Liu Y, Krause J, Kitade N, et al. Deep learning and glaucoma specialists: the relative importance of optic disc features to predict glaucoma referral in fundus photographs. Ophthalmology. 2019;126:1627–39.

    Article  PubMed  Google Scholar 

  19. Tatham AJ, Medeiros FA, Zangwill LM, Weinreb RN. Strategies to improve early diagnosis in glaucoma. Prog Brain Res. 2015;221:103–33.

    Article  PubMed  Google Scholar 

  20. Jonas JB, Weber P, Nagaoka N, Ohno-Matsui K. Glaucoma in high myopia and parapapillary delta zone. PLoS One. 2017;12:e0175120.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Muramatsu C. Diagnosis of glaucoma on retinal fundus images using deep learning: detection of nerve fiber layer defect and optic disc analysis. Adv Exp Med Biol. 2020;1213:121–32.

    Article  PubMed  Google Scholar 

  22. Devalla SK, Liang Z, Pham TH, Boote C, Strouthidis NG, Thiery AH, et al. Glaucoma management in the era of artificial intelligence. Br J Ophthalmol. 2020;104:301–11.

    Article  PubMed  Google Scholar 

  23. Grytz R, Yang H, Hua Y, Samuels BC, Sigal IA. Connective tissue remodeling in myopia and its potential role in increasing risk of glaucoma. Curr Opin Biomed Eng. 2020;15:40–50.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Chon B, Qiu M, Lin SC. Myopia and glaucoma in the South Korean population. Invest Ophthalmol Vis Sci. 2013;54:6570–7.

    Article  PubMed  Google Scholar 

  25. Ha A, Kim CY, Shim SR, Chang IB, Kim YK. Degree of myopia and glaucoma risk: a dose-response meta-analysis. Am J Ophthalmol. 2022;236:107–19.

    Article  PubMed  Google Scholar 

  26. Wang YX, Yang H, Wei CC, Xu L, Wei WB, Jonas JB. High myopia as risk factor for the 10-year incidence of open-angle glaucoma in the Beijing Eye Study. Br J Ophthalmol 2022: bjophthalmol-2021-320644.

  27. Yang HK, Kim YJ, Sung JY, Kim DH, Kim KG, Hwang JM. Efficacy for differentiating nonglaucomatous versus glaucomatous optic neuropathy using deep learning systems. Am J Ophthalmol. 2020;216:140–6.

    Article  PubMed  Google Scholar 

  28. Jonas JB, Wang YX, Dong L, Panda-Jonas S. High myopia and glaucoma-like optic neuropathy. Asia Pac J Ophthalmol (Philos). 2020;9:234–8.

    Article  Google Scholar 

  29. Wang YX, Panda-Jonas S, Jonas JB. Optic nerve head anatomy in myopia and glaucoma, including parapapillary zones alpha, beta, gamma and delta: Histology and clinical features. Prog Retin Eye Res. 2021;83:100933.

    Article  PubMed  Google Scholar 

  30. Xu T, Wang B, Liu H, Wang H, Yin P, Dong W, et al. Prevalence and causes of vision loss in China from 1990 to 2019: findings from the Global Burden of Disease Study 2019. Lancet Public Health. 2020;5:e682–91.

    Article  PubMed  Google Scholar 

  31. Craig JE, Han X, Qassim A, Hassall M, Cooke Bailey JN, Kinzy TG, et al. Multitrait analysis of glaucoma identifies new risk loci and enables polygenic prediction of disease susceptibility and progression. Nat Genet. 2020;52:160–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Cho HK, Kee C. Population-based glaucoma prevalence studies in Asians. Surv Ophthalmol. 2014;59:434–47.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors express their sincere gratitude to the patients who participated in the trial.

Author information

Authors and Affiliations

  1. Department of Geriatric Medicine, Qilu Hospital of Shandong University, No. 107, Wenhuaxi Road, Jinan, 250012, China

    Xu Qian, Song Xian, Cheng Mei & Qu Yi

  2. Key Laboratory of Cardiovascular Proteomics of Shandong Province, Jinan, China

    Xu Qian, Song Xian, Cheng Mei & Qu Yi

  3. Jinan Clinical Research Center for Geriatric Medicine (202132001), Jinan, China

    Xu Qian, Song Xian, Cheng Mei & Qu Yi

  4. Global Health Research Center, Duke Kunshan University, No. 8 Duke Avenue, Kunshan, Jiangsu Province, 215316, China

    Su Yifei

  5. Lunan Eye Hospital, Linyi, 276000, China

    Guo Wei

  6. Beijing Tongren Hospital, Capital Medical University, Beijing, 100730, China

    Hanruo Liu

  7. School of Information and Electronics, Beijing Institute of Technology, Beijing, 100730, China

    Hanruo Liu

  8. School of Computer Science and Technology, Shandong Jianzhu University, Jinan, 250101, China

    Xi Xiaoming

  9. Tencent Healthcare, Shenzhen, 51800, China

    Chunyan Chu, Yu Shuang & Ma Kai

  10. School of Software, Shandong University, Jinan, 250101, China

    Yin Yilong

Authors
  1. Xu Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Song Xian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Su Yifei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guo Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hanruo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xi Xiaoming
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chunyan Chu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yin Yilong
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yu Shuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ma Kai
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Cheng Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Qu Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

QY was responsible for the concept and design. All of the authors were responsible for the acquisition, analysis, or interpretation of data. XQ and SX wrote the original manuscript draft. All authors contributed to the critical revision of the manuscript. XQ and SY were responsible for statistical analysis. QY supervised the project. QY had full access to all of the data in the study and took responsibility for the integrity of the data and the accuracy of the data analysis.

Corresponding author

Correspondence to Qu Yi.

Ethics declarations

Competing interests

The authors declare no coompeting interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qian, X., Xian, S., Yifei, S. et al. External validation of a deep learning detection system for glaucomatous optic neuropathy: a real-world multicentre study. Eye 37, 3813–3818 (2023). https://doi.org/10.1038/s41433-023-02622-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41433-023-02622-9

Search

Advanced search

Quick links