Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Identifying facial phenotypes of genetic disorders using deep learning

Abstract

Syndromic genetic conditions, in aggregate, affect 8% of the population1. Many syndromes have recognizable facial features2 that are highly informative to clinical geneticists3,4,5. Recent studies show that facial analysis technologies measured up to the capabilities of expert clinicians in syndrome identification6,7,8,9. However, these technologies identified only a few disease phenotypes, limiting their role in clinical settings, where hundreds of diagnoses must be considered. Here we present a facial image analysis framework, DeepGestalt, using computer vision and deep-learning algorithms, that quantifies similarities to hundreds of syndromes. DeepGestalt outperformed clinicians in three initial experiments, two with the goal of distinguishing subjects with a target syndrome from other syndromes, and one of separating different genetic subtypes in Noonan syndrome. On the final experiment reflecting a real clinical setting problem, DeepGestalt achieved 91% top-10 accuracy in identifying the correct syndrome on 502 different images. The model was trained on a dataset of over 17,000 images representing more than 200 syndromes, curated through a community-driven phenotyping platform. DeepGestalt potentially adds considerable value to phenotypic evaluations in clinical genetics, genetic testing, research and precision medicine.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: DeepGestalt: high-level flow and network architecture.
Fig. 2: Composite photos and test set results of the Specialized Gestalt Model.

Data availability

The data that support the findings of this study are divided into two groups, published data and restricted data. Published data are available from the reported references and also in Supplementary Table 6. Restricted data are curated from Face2Gene users under a license and cannot be published, to protect patient privacy.

References

  1. 1.

    Baird, P. A., Anderson, T., Newcombe, H. & Lowry, R. Genetic disorders in children and young adults: a population study. Am. J. Hum. Genet. 42, 677–693 (1988).

    CAS  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Hart, T. & Hart, P. Genetic studies of craniofacial anomalies: clinical implications and applications. Orthod. Craniofac. Res. 12, 212–220 (2009).

    CAS  Article  Google Scholar 

  3. 3.

    Ferry, Q. et al. Diagnostically relevant facial gestalt information from ordinary photos. eLife 3, e02020 (2014).

    Article  Google Scholar 

  4. 4.

    Basel-Vanagaite, L. et al. Recognition of the Cornelia de Lange syndrome phenotype with facial dysmorphology novel analysis. Clin. Genet. 89, 557–563 (2016).

    CAS  Article  Google Scholar 

  5. 5.

    Rai, M. C. E., Werghi, N., Al Muhairi, H. & Alsafar, H. Using facial images for the diagnosis of genetic syndromes: a survey. In 2015 International Conference on Communications, Signal Processing, and their Applications (ICCSPA) (2015).

  6. 6.

    Shukla, P., Gupta, T., Saini, A., Singh, P. & Balasubramanian, R. A deep learning frame-work for recognizing developmental disorders. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2017).

  7. 7.

    Hadj-Rabia, S. et al. Automatic recognition of the XLHED phenotype from facial images. Am. J. Med. Genet. A. 173, 2408–2414 (2017).

    CAS  Article  Google Scholar 

  8. 8.

    Valentine, M. et al. Computer-aided recognition of facial attributes for Fetal Alcohol Spectrum disorders. Pediatrics 140, e20162028 (2017).

    Article  Google Scholar 

  9. 9.

    Gripp, K. W., Baker, L., Telegrafi, A. & Monaghan, K. G. The role of objective facial analysis using FDNA in making diagnoses following whole exome analysis. Report of two patients with mutations in the BAF complex genes. Am. J. Med. Genet. A. 170, 1754–1762 (2016).

    CAS  Article  Google Scholar 

  10. 10.

    Delgadillo, V., Maria del Mar, O., Gort, L., Coll, M. J. & Pineda, M. Natural history of Sanfilippo syndrome in Spain. Orphanet J. Rare Dis. 8, 189 (2013).

    Article  Google Scholar 

  11. 11.

    Kole, A. et al. The Voice of 12,000 Patients: experiences and expectations of rare disease patients on diagnosis and care in Europe. Eurordis http://www.eurordis.org/IMG/pdf/voice_12000_patients/EURORDISCARE_FULLBOOKr.pdf (2009).

  12. 12.

    Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014 1701–1708 (IEEE, 2014).

  13. 13.

    Huang, G. B., Ramesh, M., Berg, T. & Learned-Miller, E. Labeled faces in the Wild: a database for studying face recognition in unconstrained environments. In Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition (2008).

  14. 14.

    Yi, D., Lei, Z., Liao, S. & Li, S. Z. Learning face representation from scratch. Preprint at https://arxiv.org/abs/1411.7923 (2014).

  15. 15.

    Schroff, F., Kalenichenko, D. & Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 815–823 (IEEE,2015).

  16. 16.

    Li, H., Lin, Z., Shen, X., Brandt, J. & Hua, G. A convolutional neural network cascade for face detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 5325–5334 (IEEE, 2015).

  17. 17.

    Karlinsky, L. & Ullman, S. Using linking features in learning non-parametric part models. Computer Vision–ECCV 2012, 326–339 (2012).

    Google Scholar 

  18. 18.

    Rohatgi, S. et al. Facial diagnosis of mild and variant CdLS: Insights from a dysmorphologist survey. Am. J. Med. Genet. A. 152, 1641–1653 (2010).

    Article  Google Scholar 

  19. 19.

    Bird, L. M., Tan, W. H. & Wolf, L. The role of computer-aided facial recognition technology in accelerating the identification of Angelman syndrome. In 35th Annual David W Smith Workshop (2014).

  20. 20.

    Allanson, J. E. et al. The face of Noonan syndrome: does phenotype predict genotype. Am. J. Med. Genet. A. 152, 1960–1966 (2010).

    Article  Google Scholar 

  21. 21.

    Gulec, E. Y., Ocak, Z., Candan, S., Ataman, E. & Yarar, C. Novel mutations in PTPN11 gene in two girls with Noonan syndrome phenotype. Int. J. Cardiol. 186, 13–15 (2015).

    Article  Google Scholar 

  22. 22.

    Zenker, M. et al. SOS1 is the second most common Noonan gene but plays no major role in cardio-facio-cutaneous syndrome. J. Med. Genet. 44, 651–656 (2007).

    CAS  Article  Google Scholar 

  23. 23.

    Rusu, C., Idriceanu, J., Bodescu, I., Anton, M. & Vulpoi, C. Genotype-phenotype correlations in Noonan Syndrome. Acta Endocrinologica 10, 463–476 (2014).

    Article  Google Scholar 

  24. 24.

    Cavé, H. et al. Mutations in RIT1 cause Noonan syndrome with possible juvenile myelomonocytic leukemia but are not involved in acute lymphoblastic leukemia. Eur. J. Hum. Genet. 24, 1124–1131 (2016).

    Article  Google Scholar 

  25. 25.

    Kouz, K. et al. Genotype and phenotype in patients with Noonan syndrome and a RIT1 mutation. Genet. Med. 18, 1226–1234 (2016).

    CAS  Article  Google Scholar 

  26. 26.

    Winter, R. M. & Baraitser The London Dysmorphology Database. J. Med. Genet. 24, 509–510 (1987).

    CAS  Article  Google Scholar 

  27. 27.

    Robinson, P. N. & Mundlos, S. The human phenotype ontology. Clin. Genet. 77, 525–534 (2010).

    CAS  Article  Google Scholar 

  28. 28.

    Köhler, S. et al. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. A. J. Hum. Genet. 85, 457–464 (2009).

    Article  Google Scholar 

  29. 29.

    Zarate, Y. A. et al. Natural history and genotype-phenotype correlations in 72 individuals with SATB2-associated syndrome. Am. J. Med. Genet. A. 176, 925–935 (2018).

    Article  Google Scholar 

  30. 30.

    Liehr, T. et al. Next generation phenotyping in Emanuel and Pallister Killian Syndrome using computer-aided facial dysmorphology analysis of 2D photos. Clin. Genet. 93, 378–381 (2017).

    Article  Google Scholar 

  31. 31.

    Hennekam, R. & Biesecker, L. G. Next-generation sequencing demands next-generation phenotyping. Hum. Mutat. 33, 884–886 (2012).

    CAS  Article  Google Scholar 

  32. 32.

    Huang, G., Mattar, M., Lee, H. & Learned-Miller, E. G. Learning to align from scratch. In Advances in Neural Information Processing Systems 2012 764–772 (2012).

  33. 33.

    Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 2014 3320–3328 (2014).

  34. 34.

    Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Web-scale training for face identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 2746–2754 (IEEE, 2015).

  35. 35.

    Zhou, E., Cao, Z. & Yin, Q. Naive-deep face recognition: touching the limit of LFW benchmark or not? Preprint at https://arxiv.org/abs/1501.04690 (2015).

  36. 36.

    Liu, J., Deng, Y., Bai, T., Wei, Z. & Huang, C. Targeting ultimate accuracy: face recognition via deep embedding. Preprint at https://arxiv.org/abs/1506.07310 (2015).

  37. 37.

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).

  38. 38.

    Parkhi, O. M., Vedaldi, A. & Zisserman, A. Deep face recognition. In Proceedings of the British Machine Vision Conference 1, 6 (2015).

  39. 39.

    Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. International Conference on Machine Learning 2015 448–456 (2015).

  40. 40.

    Chollet, F. et al. Keras. http://keras.io (2015).

  41. 41.

    Abadi, M. et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467 (2016).

  42. 42.

    He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proc. IEEE International Conference on Computer Vision 1026–1034 (IEEE, 2015).

  43. 43.

    Kingma, D. and Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  44. 44.

    Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international Conference on Artificial Intelligence and Statistics 249–256 (2010).

  45. 45.

    Efron, B. Bootstrap methods: another look at the jackknife. In Breakthroughs in Statistics 569–593 (Springer, New York, 1992).

Download references

Acknowledgements

The authors thank the patients and their families, as well as Face2Gene users worldwide who contribute with their knowledge and dedication to the improvement of this and other tools for the ultimate benefit of better healthcare.

Author information

Affiliations

Authors

Contributions

Y.G., Y.H. and O.B. initiated the project. Y.G., Y.H. and D.G. developed the DeepGestalt framework. N.F., L.B.-S., P.M.K., S.B.K., M.Z., L.M.B. and K.W.G. designed and conducted the clinical experiments. O.B. and G.N. finalized the dataset, computed statistics and contributed to the software engineering. Y.G., Y.H., O.B., P.M.K. and K.W.G. contributed to the writing of the manuscript.

Corresponding author

Correspondence to Yaron Gurovich.

Ethics declarations

Competing interests

Y.G., Y.H., O.B., G.N., N.F. and D.G. are employees of FDNA; L.B.-S., P.M.K. and K.W.G. are advisors of FDNA; L.B.-S., P.M.K., M.Z., L.M.B. and K.W.G. are members of the scientific advisory board of FDNA.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1 and 2 and Supplementary Tables 1–6

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gurovich, Y., Hanani, Y., Bar, O. et al. Identifying facial phenotypes of genetic disorders using deep learning. Nat Med 25, 60–64 (2019). https://doi.org/10.1038/s41591-018-0279-0

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing