Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Clinical evaluation of deep learning systems for assisting in the diagnosis of the epiretinal membrane grade in general ophthalmologists

Eye volume 38, pages 730–736 (2024)Cite this article

Subjects

Abstract

Background

Epiretinal membrane (ERM) is a common age-related retinal disease detected by optical coherence tomography (OCT), with a prevalence of 34.1% among people over 60 years old. This study aims to develop artificial intelligence (AI) systems to assist in the diagnosis of ERM grade using OCT images and to clinically evaluate the potential benefits and risks of our AI systems with a comparative experiment.

Methods

A segmentation deep learning (DL) model that segments retinal features associated with ERM severity and a classification DL model that grades the severity of ERM were developed based on an OCT dataset obtained from three hospitals. A comparative experiment was conducted to compare the performance of four general ophthalmologists with and without assistance from the AI in diagnosing ERM severity.

Results

The segmentation network had a pixel accuracy (PA) of 0.980 and a mean intersection over union (MIoU) of 0.873, while the six-classification network had a total accuracy of 81.3%. The diagnostic accuracy scores of the four ophthalmologists increased with AI assistance from 81.7%, 80.7%, 78.0%, and 80.7% to 87.7%, 86.7%, 89.0%, and 91.3%, respectively, while the corresponding time expenditures were reduced. The specific results of the study as well as the misinterpretations of the AI systems were analysed.

Conclusion

Through our comparative experiment, the AI systems proved to be valuable references for medical diagnosis and demonstrated the potential to accelerate clinical workflows. Systematic efforts are needed to ensure the safe and rapid integration of AI systems into ophthalmic practice.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Performance of our artificial intelligence (AI) systems.
Fig. 2: Illustration and results of our comparative experiment.
Fig. 3: Typical cases and results of our comparative experiment.

Similar content being viewed by others

Data availability

The de-identified individual participant data can be requested from the correspondence author, who will evaluate such requests on a case-by-case basis.

References

  1. Bu S-C, Kuijer R, Li X-R, Hooymans JMM, Los LI. Idiopathic epiretinal membrane. Retina. 2014;34:2317–35.

    Article  CAS  PubMed  Google Scholar 

  2. Meuer SM, Myers CE, Klein BEK, Swift MK, Huang Y, Gangaputra S, et al. The epidemiology of vitreoretinal interface abnormalities as detected by spectral-domain optical coherence tomography: the beaver dam eye study. Ophthalmology. 2015;122:787–95.

    Article  PubMed  Google Scholar 

  3. Fung AT, Galvin J, Tran T. Epiretinal membrane: a review. Clin Exp Ophthalmol. 2021;49:289–308.

    Article  PubMed  Google Scholar 

  4. Delyfer M-N, Legout P, Le Goff M, Blaizeau M, Rougier M-B, Schweitzer C, et al. Prevalence of epiretinal membranes in the ageing population using retinal colour images and SD-OCT: the Alienor sudy. Acta ophthalmol. 2020;98:e830–8.

    Article  PubMed  Google Scholar 

  5. Hwang J-U, Sohn J, Moon BG, Joe SG, Lee JY, Kim J-G, et al. Assessment of macular function for idiopathic epiretinal membranes classified by spectral-domain optical coherence tomography. Invest Ophthalmol Vis Sci. 2012;53:3562–9.

    Article  PubMed  Google Scholar 

  6. Stevenson W, Prospero Ponce CM, Agarwal DR, Gelman R, Christoforidis JB. Epiretinal membrane: optical coherence tomography-based diagnosis and classification. Clin Ophthalmol. 2016;10:527–34.

    Article  PubMed Central  PubMed  Google Scholar 

  7. Govetto A, Lalane RA, Sarraf D, Figueroa MS, Hubschman JP. Insights into epiretinal membranes: presence of ectopic inner foveal layers and a new optical coherence tomography staging scheme. Am. J Ophthalmol. 2017;175:99–113.

  8. Lu W, Tong Y, Yu Y, Xing Y, Chen C, Shen Y. Deep learning-based automated classification of multi-categorical abnormalities from optical coherence tomography images. Transl Vis Sci Technol. 2018;7:41.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  9. Sonobe T, Tabuchi H, Ohsugi H, Masumoto H, Ishitobi N, Morita S, et al. Comparison between support vector machine and deep learning, machine-learning technologies for detecting epiretinal membrane using 3D-OCT. Int Ophthalmol. 2019;39:1871–7.

    Article  PubMed  Google Scholar 

  10. Lo Y-C, Lin K-H, Bair H, Sheu WH-H, Chang C-S, Shen Y-C, et al. Epiretinal membrane detection at the ophthalmologist level using deep learning of optical coherence tomography. Sci Rep. 2020;10:8424.

    Article  ADS  CAS  PubMed Central  PubMed  Google Scholar 

  11. Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol. 2017;135:1170–6.

    Article  PubMed Central  PubMed  Google Scholar 

  12. Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, et al. identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 2018;172:1122–31.

  13. De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24:1342–50.

    Article  PubMed  Google Scholar 

  14. Ting DSW, Liu Y, Burlina P, Xu X, Bressler NM, Wong TY. AI for medical imaging goes deep. Nat Med. 2018;24:539–40.

    Article  CAS  PubMed  Google Scholar 

  15. Abràmoff MD, Tobey D, Char DS. Lessons learned about autonomous AI: finding a safe, efficacious, and ethical path through the development process. Am J Ophthalmol. 2020;214:134–42.

    Article  PubMed  Google Scholar 

  16. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44–56.

    Article  CAS  PubMed  Google Scholar 

  17. Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell. 2017;39:2481–95.

    Article  PubMed  Google Scholar 

  18. He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. In Proc. IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR). 2016:90:770–8.

  19. Jiang XY, Ge ZQ. Data augmentation classifier for imbalanced fault classification. IEEE Trans Autom Sci Eng. 2021;18:1206–17.

    Article  Google Scholar 

  20. Doguizi S, Sekeroglu MA, Ozkoyuncu D, Omay AE, Yilmazbas P. Clinical significance of ectopic inner foveal layers in patients with idiopathic epiretinal membranes. Eye (Lond). 2018;32:1652–60.

    Article  PubMed  Google Scholar 

  21. Alkabes M, Fogagnolo P, Vujosevic S, Rossetti L, Casini G, De Cillà S. Correlation between new OCT parameters and metamorphopsia in advanced stages of epiretinal membranes. Acta Ophthalmol. 2020;98:780–6.

    Article  PubMed  Google Scholar 

  22. Flaxel CJ, Adelman RA, Bailey ST, Fawzi A, Lim JI, Vemulakonda GA, et al. Idiopathic epiretinal membrane and vitreomacular traction preferred practice pattern®. Ophthalmology. 2020;127:P145–83.

    Article  PubMed  Google Scholar 

  23. Burlina P, Paul W, Mathew P, Joshi N, Pacheco KD, Bressler NM. Low-shot deep learning of diabetic retinopathy with potential applications to address artificial intelligence bias in retinal diagnostics and rare ophthalmic diseases. JAMA Ophthalmol. 2020;138:1070–7.

    Article  PubMed Central  PubMed  Google Scholar 

  24. Burlina P, Paul W, Liu TYA, Bressler NM. Detecting anomalies in retinal diseases using generative, discriminative, and self-supervised deep learning. JAMA Ophthalmol. 2022;140:185–9.

    Article  PubMed  Google Scholar 

  25. Lehman CD, Wellman RD, Buist DSM, Kerlikowske K, Tosteson ANA, Miglioretti DL. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern Med. 2015;175:1828–37.

    Article  PubMed Central  PubMed  Google Scholar 

  26. Haibe-Kains B, Adam GA, Hosny A, Khodakarami F, Waldron L, Wang B, et al. Transparency and reproducibility in artificial intelligence. Nature. 2020;586:E14–6.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  27. Daneshjou R, Smith MP, Sun MD, Rotemberg V, Zou J. Lack of transparency and potential bias in artificial intelligence data sets and algorithms: a scoping review. JAMA Dermatol. 2021;157:1362–9.

    Article  PubMed Central  PubMed  Google Scholar 

  28. González-Gonzalo C, Thee EF, Klaver CCW, Lee AY, Schlingemann RO, Tufail A, et al. Trustworthy AI: Closing the gap between development and integration of AI systems in ophthalmic practice. Prog Retin Eye Res. 2021:90:101034.

Download references

Acknowledgements

This work was financially supported by the National Natural Science Foundation Regional Innovation and Development Joint Fund (U20A20386), the National Key Research and Development Program of China (grant number 2019YFC0118400), Zhejiang Provincial Key Research and Development Plan (grant number 2019C03020), Natural Science Foundation of Zhejiang Province (grant number LQ21H120002), the Natural Science Foundation of China (grant number 81670888), and the Clinical Medical Research Center for Eye Diseases of Zhejiang Province (2021E50007).

Author information

Authors and Affiliations

  1. Eye Center, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, 310009, China

    Yan Yan, Xiaoling Huang, Zhiyuan Gao, Xindi Liu, Kai Jin & Juan Ye

  2. College of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, China

    Xiaoyu Jiang

Authors
  1. Yan Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoling Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoyu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiyuan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xindi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kai Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Juan Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YY, XH, KJ and JY conceived and designed the experiments. YY, ZG, XL and KJ collected and provided the data. XJ preprocessed the data and developed the deep learning systems. YY and XH analysed the results. All authors reviewed the manuscript.

Corresponding authors

Correspondence to Kai Jin or Juan Ye.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, Y., Huang, X., Jiang, X. et al. Clinical evaluation of deep learning systems for assisting in the diagnosis of the epiretinal membrane grade in general ophthalmologists. Eye 38, 730–736 (2024). https://doi.org/10.1038/s41433-023-02765-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41433-023-02765-9

Search

Advanced search

Quick links