Article | Published:

Evaluation of an AI system for the automated detection of glaucoma from stereoscopic optic disc photographs: the European Optic Disc Assessment Study

Abstract

Objectives

To evaluate the performance of a deep learning based Artificial Intelligence (AI) software for detection of glaucoma from stereoscopic optic disc photographs, and to compare this performance to the performance of a large cohort of ophthalmologists and optometrists.

Methods

A retrospective study evaluating the diagnostic performance of an AI software (Pegasus v1.0, Visulytix Ltd., London UK) and comparing it with that of 243 European ophthalmologists and 208 British optometrists, as determined in previous studies, for the detection of glaucomatous optic neuropathy from 94 scanned stereoscopic photographic slides scanned into digital format.

Results

Pegasus was able to detect glaucomatous optic neuropathy with an accuracy of 83.4% (95% CI: 77.5–89.2). This is comparable to an average ophthalmologist accuracy of 80.5% (95% CI: 67.2–93.8) and average optometrist accuracy of 80% (95% CI: 67–88) on the same images. In addition, the AI system had an intra-observer agreement (Cohen’s Kappa, κ) of 0.74 (95% CI: 0.63–0.85), compared with 0.70 (range: −0.13–1.00; 95% CI: 0.67–0.73) and 0.71 (range: 0.08–1.00) for ophthalmologists and optometrists, respectively. There was no statistically significant difference between the performance of the deep learning system and ophthalmologists or optometrists.

Conclusion

The AI system obtained a diagnostic performance and repeatability comparable to that of the ophthalmologists and optometrists. We conclude that deep learning based AI systems, such as Pegasus, demonstrate significant promise in the assisted detection of glaucomatous optic neuropathy.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Quigley HA, Broman AT. The number of people with glaucoma worldwide in 2010 and 2020. Br J Ophthalmol 2006;90:262–7.

  2. 2.

    Reus NJ, Lemij HG, Garway-Heath DF, Airaksinen PJ, Anton A, Bron AM, et al. Clinical assessment of stereoscopic optic disc photographs for glaucoma: the European Optic Disc Assessment Trial. Ophthalmology. 2010;117:717–23.

  3. 3.

    Chan HH, Ong DN, Kong YX, O’Neill EC, Pandav SS, Coote MA, et al. Glaucomatous optic neuropathy evaluation (GONE) project: the effect of monoscopic versus stereoscopic viewing conditions on optic nerve evaluation. Am J Ophthalmol. 2014;157:936–44.

  4. 4.

    Grigera DE, Mello PA, Barbosa WL, Casiraghi JF, Grossmann RP, Peyret A. Level of agreement among Latin American glaucoma subspecialists on the diagnosis and treatment of glaucoma: results of an online survey. Arq Bras Oftalmol. 2013;76:163–9.

  5. 5.

    Batterbury M. Agreement between ophthalmologists and optometrists in optic disc assessment: training implications for glaucoma co-management. Graefes Arch Clin Exp Ophthalmol. 2001;239:342–59.

  6. 6.

    Teitelbaum BA, Haefs R, Connor D. Interobserver variability in the estimation of the cup/disk ratio among observers of differing educational background. Optometry 2001;72:729–32.

  7. 7.

    Lockwood AJ, Kirwan JF, Ashleigh Z. Optometrists referrals for glaucoma assessment: a prospective survey of clinical data and outcomes. Eye. 2010;24:1515.

  8. 8.

    Standards for Virtual Clinics in Glaucoma Care in the NHS Hospital Eye Service, November 2016. Available at http://www.rcophth.ac.uk/standards-publications-research. Accessed 1st July 2018.

  9. 9.

    Trikha S, Macgregor C, Jeffery M, Kirwan J. The Portsmouth-based glaucoma refinement scheme: a role for virtual clinics in the future? Eye. 2012;26:1288.

  10. 10.

    Ratnarajan G. Evaluating and optimising glaucoma referral refinement pathways with specific reference to the changes scheme. Doctoral dissertation, University College London. 2016.

  11. 11.

    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–10.

  12. 12.

    Yang GG, Minasyan A, Lacey M, Gordon J, Knoll T. HMGA2 is a reliable immunohistochemical marker for separating melanoma from Nevi. Nature. 2014;94:145A.

  13. 13.

    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436.

  14. 14.

    Seo E, Jaccard N, Trikha S, Pasquale LR, Song BJ. Automated evaluation of optic disc images for manifest glaucoma detection using a deep-learning, neural network-based algorithm. Invest Ophthalmol Vis Sci 2018;59:2080–80.

  15. 15.

    Hadwin SE, Redmond T, Garway‐Heath DF, Lemij HG, Reus NJ, Ward G, et al. Assessment of optic disc photographs for glaucoma by UK optometrists: the Moorfields Optic Disc Assessment Study (MODAS). Ophthalmic Physiol Opt. 2013;33:618–24.

  16. 16.

    Fu H, Cheng J, Xu Y, Zhang C, Wong DW, Liu J, et al. Disc-aware ensemble network for glaucoma screening from fundus image. IEEE Trans. Med. Imaging. 2018;11:2493–2501.

  17. 17.

    Chen X, Xu Y, Wong DW, Wong TY, Liu J Glaucoma detection based on deep convolutional neural network. In IEEE International Conference of the Engineering in Medicine and Biology Society (EMBC). 2015 (pp. 715–8).

  18. 18.

    Ting DS, Cheung CY, Lim G, Tan GS, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318:2211–23.

  19. 19.

    He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770–78).

  20. 20.

    De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 2018;24:1342–50.

Download references

Acknowledgements

We acknowledge Visulytix Ltd. for supply of hardware and funding.

Funding

Funding for the analysis of the EODAT images was provided by Visulytix Ltd. Support: all equipment and funding for this work was provided by Visulytix Ltd.

Author information

Correspondence to Thomas W. Rogers.

Ethics declarations

Conflict of interest

T.W. Rogers and N. Jaccard are employed by Visulytix Ltd. S. Trikha owns stock in and has received honoraria from Visulytix Ltd. F. Carbonaro owns stock in Visulytix Ltd. N.J. Reus, K.A. Vermeer and H.G. Lemij declare no potential conflicts of interest.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark
Fig. 1
Fig. 2
Fig. 3