Evaluation of an AI system for the automated detection of glaucoma from stereoscopic optic disc photographs: the European Optic Disc Assessment Study



To evaluate the performance of a deep learning based Artificial Intelligence (AI) software for detection of glaucoma from stereoscopic optic disc photographs, and to compare this performance to the performance of a large cohort of ophthalmologists and optometrists.


A retrospective study evaluating the diagnostic performance of an AI software (Pegasus v1.0, Visulytix Ltd., London UK) and comparing it with that of 243 European ophthalmologists and 208 British optometrists, as determined in previous studies, for the detection of glaucomatous optic neuropathy from 94 scanned stereoscopic photographic slides scanned into digital format.


Pegasus was able to detect glaucomatous optic neuropathy with an accuracy of 83.4% (95% CI: 77.5–89.2). This is comparable to an average ophthalmologist accuracy of 80.5% (95% CI: 67.2–93.8) and average optometrist accuracy of 80% (95% CI: 67–88) on the same images. In addition, the AI system had an intra-observer agreement (Cohen’s Kappa, κ) of 0.74 (95% CI: 0.63–0.85), compared with 0.70 (range: −0.13–1.00; 95% CI: 0.67–0.73) and 0.71 (range: 0.08–1.00) for ophthalmologists and optometrists, respectively. There was no statistically significant difference between the performance of the deep learning system and ophthalmologists or optometrists.


The AI system obtained a diagnostic performance and repeatability comparable to that of the ophthalmologists and optometrists. We conclude that deep learning based AI systems, such as Pegasus, demonstrate significant promise in the assisted detection of glaucomatous optic neuropathy.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Fig. 1
Fig. 2
Fig. 3


  1. 1.

    Quigley HA, Broman AT. The number of people with glaucoma worldwide in 2010 and 2020. Br J Ophthalmol 2006;90:262–7.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Reus NJ, Lemij HG, Garway-Heath DF, Airaksinen PJ, Anton A, Bron AM, et al. Clinical assessment of stereoscopic optic disc photographs for glaucoma: the European Optic Disc Assessment Trial. Ophthalmology. 2010;117:717–23.

    Article  PubMed  Google Scholar 

  3. 3.

    Chan HH, Ong DN, Kong YX, O’Neill EC, Pandav SS, Coote MA, et al. Glaucomatous optic neuropathy evaluation (GONE) project: the effect of monoscopic versus stereoscopic viewing conditions on optic nerve evaluation. Am J Ophthalmol. 2014;157:936–44.

    Article  PubMed  Google Scholar 

  4. 4.

    Grigera DE, Mello PA, Barbosa WL, Casiraghi JF, Grossmann RP, Peyret A. Level of agreement among Latin American glaucoma subspecialists on the diagnosis and treatment of glaucoma: results of an online survey. Arq Bras Oftalmol. 2013;76:163–9.

    Article  PubMed  Google Scholar 

  5. 5.

    Batterbury M. Agreement between ophthalmologists and optometrists in optic disc assessment: training implications for glaucoma co-management. Graefes Arch Clin Exp Ophthalmol. 2001;239:342–59.

    Article  PubMed  Google Scholar 

  6. 6.

    Teitelbaum BA, Haefs R, Connor D. Interobserver variability in the estimation of the cup/disk ratio among observers of differing educational background. Optometry 2001;72:729–32.

    CAS  PubMed  Google Scholar 

  7. 7.

    Lockwood AJ, Kirwan JF, Ashleigh Z. Optometrists referrals for glaucoma assessment: a prospective survey of clinical data and outcomes. Eye. 2010;24:1515.

    CAS  Article  PubMed  Google Scholar 

  8. 8.

    Standards for Virtual Clinics in Glaucoma Care in the NHS Hospital Eye Service, November 2016. Available at http://www.rcophth.ac.uk/standards-publications-research. Accessed 1st July 2018.

  9. 9.

    Trikha S, Macgregor C, Jeffery M, Kirwan J. The Portsmouth-based glaucoma refinement scheme: a role for virtual clinics in the future? Eye. 2012;26:1288.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Ratnarajan G. Evaluating and optimising glaucoma referral refinement pathways with specific reference to the changes scheme. Doctoral dissertation, University College London. 2016.

  11. 11.

    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–10.

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Yang GG, Minasyan A, Lacey M, Gordon J, Knoll T. HMGA2 is a reliable immunohistochemical marker for separating melanoma from Nevi. Nature. 2014;94:145A.

    Google Scholar 

  13. 13.

    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436.

    CAS  Article  Google Scholar 

  14. 14.

    Seo E, Jaccard N, Trikha S, Pasquale LR, Song BJ. Automated evaluation of optic disc images for manifest glaucoma detection using a deep-learning, neural network-based algorithm. Invest Ophthalmol Vis Sci 2018;59:2080–80.

    Article  Google Scholar 

  15. 15.

    Hadwin SE, Redmond T, Garway‐Heath DF, Lemij HG, Reus NJ, Ward G, et al. Assessment of optic disc photographs for glaucoma by UK optometrists: the Moorfields Optic Disc Assessment Study (MODAS). Ophthalmic Physiol Opt. 2013;33:618–24.

    Article  PubMed  Google Scholar 

  16. 16.

    Fu H, Cheng J, Xu Y, Zhang C, Wong DW, Liu J, et al. Disc-aware ensemble network for glaucoma screening from fundus image. IEEE Trans. Med. Imaging. 2018;11:2493–2501.

    Article  PubMed  Google Scholar 

  17. 17.

    Chen X, Xu Y, Wong DW, Wong TY, Liu J Glaucoma detection based on deep convolutional neural network. In IEEE International Conference of the Engineering in Medicine and Biology Society (EMBC). 2015 (pp. 715–8).

  18. 18.

    Ting DS, Cheung CY, Lim G, Tan GS, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318:2211–23.

    Article  PubMed  PubMed Central  Google Scholar 

  19. 19.

    He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770–78).

  20. 20.

    De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 2018;24:1342–50.

    CAS  Article  PubMed  Google Scholar 

Download references


We acknowledge Visulytix Ltd. for supply of hardware and funding.


Funding for the analysis of the EODAT images was provided by Visulytix Ltd. Support: all equipment and funding for this work was provided by Visulytix Ltd.

Author information



Corresponding author

Correspondence to Thomas W. Rogers.

Ethics declarations

Conflict of interest

T.W. Rogers and N. Jaccard are employed by Visulytix Ltd. S. Trikha owns stock in and has received honoraria from Visulytix Ltd. F. Carbonaro owns stock in Visulytix Ltd. N.J. Reus, K.A. Vermeer and H.G. Lemij declare no potential conflicts of interest.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rogers, T.W., Jaccard, N., Carbonaro, F. et al. Evaluation of an AI system for the automated detection of glaucoma from stereoscopic optic disc photographs: the European Optic Disc Assessment Study. Eye 33, 1791–1797 (2019). https://doi.org/10.1038/s41433-019-0510-3

Download citation

Further reading