Syndromic genetic conditions, in aggregate, affect 8% of the population1. Many syndromes have recognizable facial features2 that are highly informative to clinical geneticists3,4,5. Recent studies show that facial analysis technologies measured up to the capabilities of expert clinicians in syndrome identification6,7,8,9. However, these technologies identified only a few disease phenotypes, limiting their role in clinical settings, where hundreds of diagnoses must be considered. Here we present a facial image analysis framework, DeepGestalt, using computer vision and deep-learning algorithms, that quantifies similarities to hundreds of syndromes. DeepGestalt outperformed clinicians in three initial experiments, two with the goal of distinguishing subjects with a target syndrome from other syndromes, and one of separating different genetic subtypes in Noonan syndrome. On the final experiment reflecting a real clinical setting problem, DeepGestalt achieved 91% top-10 accuracy in identifying the correct syndrome on 502 different images. The model was trained on a dataset of over 17,000 images representing more than 200 syndromes, curated through a community-driven phenotyping platform. DeepGestalt potentially adds considerable value to phenotypic evaluations in clinical genetics, genetic testing, research and precision medicine.
Subscribe to Journal
Get full journal access for 1 year
only $4.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The data that support the findings of this study are divided into two groups, published data and restricted data. Published data are available from the reported references and also in Supplementary Table 6. Restricted data are curated from Face2Gene users under a license and cannot be published, to protect patient privacy.
Baird, P. A., Anderson, T., Newcombe, H. & Lowry, R. Genetic disorders in children and young adults: a population study. Am. J. Hum. Genet. 42, 677–693 (1988).
Hart, T. & Hart, P. Genetic studies of craniofacial anomalies: clinical implications and applications. Orthod. Craniofac. Res. 12, 212–220 (2009).
Ferry, Q. et al. Diagnostically relevant facial gestalt information from ordinary photos. eLife 3, e02020 (2014).
Basel-Vanagaite, L. et al. Recognition of the Cornelia de Lange syndrome phenotype with facial dysmorphology novel analysis. Clin. Genet. 89, 557–563 (2016).
Rai, M. C. E., Werghi, N., Al Muhairi, H. & Alsafar, H. Using facial images for the diagnosis of genetic syndromes: a survey. In 2015 International Conference on Communications, Signal Processing, and their Applications (ICCSPA) (2015).
Shukla, P., Gupta, T., Saini, A., Singh, P. & Balasubramanian, R. A deep learning frame-work for recognizing developmental disorders. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2017).
Hadj-Rabia, S. et al. Automatic recognition of the XLHED phenotype from facial images. Am. J. Med. Genet. A. 173, 2408–2414 (2017).
Valentine, M. et al. Computer-aided recognition of facial attributes for Fetal Alcohol Spectrum disorders. Pediatrics 140, e20162028 (2017).
Gripp, K. W., Baker, L., Telegrafi, A. & Monaghan, K. G. The role of objective facial analysis using FDNA in making diagnoses following whole exome analysis. Report of two patients with mutations in the BAF complex genes. Am. J. Med. Genet. A. 170, 1754–1762 (2016).
Delgadillo, V., Maria del Mar, O., Gort, L., Coll, M. J. & Pineda, M. Natural history of Sanfilippo syndrome in Spain. Orphanet J. Rare Dis. 8, 189 (2013).
Kole, A. et al. The Voice of 12,000 Patients: experiences and expectations of rare disease patients on diagnosis and care in Europe. Eurordis http://www.eurordis.org/IMG/pdf/voice_12000_patients/EURORDISCARE_FULLBOOKr.pdf (2009).
Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014 1701–1708 (IEEE, 2014).
Huang, G. B., Ramesh, M., Berg, T. & Learned-Miller, E. Labeled faces in the Wild: a database for studying face recognition in unconstrained environments. In Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition (2008).
Yi, D., Lei, Z., Liao, S. & Li, S. Z. Learning face representation from scratch. Preprint at https://arxiv.org/abs/1411.7923 (2014).
Schroff, F., Kalenichenko, D. & Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 815–823 (IEEE,2015).
Li, H., Lin, Z., Shen, X., Brandt, J. & Hua, G. A convolutional neural network cascade for face detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 5325–5334 (IEEE, 2015).
Karlinsky, L. & Ullman, S. Using linking features in learning non-parametric part models. Computer Vision–ECCV 2012, 326–339 (2012).
Rohatgi, S. et al. Facial diagnosis of mild and variant CdLS: Insights from a dysmorphologist survey. Am. J. Med. Genet. A. 152, 1641–1653 (2010).
Bird, L. M., Tan, W. H. & Wolf, L. The role of computer-aided facial recognition technology in accelerating the identification of Angelman syndrome. In 35th Annual David W Smith Workshop (2014).
Allanson, J. E. et al. The face of Noonan syndrome: does phenotype predict genotype. Am. J. Med. Genet. A. 152, 1960–1966 (2010).
Gulec, E. Y., Ocak, Z., Candan, S., Ataman, E. & Yarar, C. Novel mutations in PTPN11 gene in two girls with Noonan syndrome phenotype. Int. J. Cardiol. 186, 13–15 (2015).
Zenker, M. et al. SOS1 is the second most common Noonan gene but plays no major role in cardio-facio-cutaneous syndrome. J. Med. Genet. 44, 651–656 (2007).
Rusu, C., Idriceanu, J., Bodescu, I., Anton, M. & Vulpoi, C. Genotype-phenotype correlations in Noonan Syndrome. Acta Endocrinologica 10, 463–476 (2014).
Cavé, H. et al. Mutations in RIT1 cause Noonan syndrome with possible juvenile myelomonocytic leukemia but are not involved in acute lymphoblastic leukemia. Eur. J. Hum. Genet. 24, 1124–1131 (2016).
Kouz, K. et al. Genotype and phenotype in patients with Noonan syndrome and a RIT1 mutation. Genet. Med. 18, 1226–1234 (2016).
Winter, R. M. & Baraitser The London Dysmorphology Database. J. Med. Genet. 24, 509–510 (1987).
Robinson, P. N. & Mundlos, S. The human phenotype ontology. Clin. Genet. 77, 525–534 (2010).
Köhler, S. et al. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. A. J. Hum. Genet. 85, 457–464 (2009).
Zarate, Y. A. et al. Natural history and genotype-phenotype correlations in 72 individuals with SATB2-associated syndrome. Am. J. Med. Genet. A. 176, 925–935 (2018).
Liehr, T. et al. Next generation phenotyping in Emanuel and Pallister Killian Syndrome using computer-aided facial dysmorphology analysis of 2D photos. Clin. Genet. 93, 378–381 (2017).
Hennekam, R. & Biesecker, L. G. Next-generation sequencing demands next-generation phenotyping. Hum. Mutat. 33, 884–886 (2012).
Huang, G., Mattar, M., Lee, H. & Learned-Miller, E. G. Learning to align from scratch. In Advances in Neural Information Processing Systems 2012 764–772 (2012).
Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 2014 3320–3328 (2014).
Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Web-scale training for face identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 2746–2754 (IEEE, 2015).
Zhou, E., Cao, Z. & Yin, Q. Naive-deep face recognition: touching the limit of LFW benchmark or not? Preprint at https://arxiv.org/abs/1501.04690 (2015).
Liu, J., Deng, Y., Bai, T., Wei, Z. & Huang, C. Targeting ultimate accuracy: face recognition via deep embedding. Preprint at https://arxiv.org/abs/1506.07310 (2015).
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).
Parkhi, O. M., Vedaldi, A. & Zisserman, A. Deep face recognition. In Proceedings of the British Machine Vision Conference 1, 6 (2015).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. International Conference on Machine Learning 2015 448–456 (2015).
Chollet, F. et al. Keras. http://keras.io (2015).
Abadi, M. et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467 (2016).
He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proc. IEEE International Conference on Computer Vision 1026–1034 (IEEE, 2015).
Kingma, D. and Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).
Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international Conference on Artificial Intelligence and Statistics 249–256 (2010).
Efron, B. Bootstrap methods: another look at the jackknife. In Breakthroughs in Statistics 569–593 (Springer, New York, 1992).
The authors thank the patients and their families, as well as Face2Gene users worldwide who contribute with their knowledge and dedication to the improvement of this and other tools for the ultimate benefit of better healthcare.
Y.G., Y.H., O.B., G.N., N.F. and D.G. are employees of FDNA; L.B.-S., P.M.K. and K.W.G. are advisors of FDNA; L.B.-S., P.M.K., M.Z., L.M.B. and K.W.G. are members of the scientific advisory board of FDNA.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Gurovich, Y., Hanani, Y., Bar, O. et al. Identifying facial phenotypes of genetic disorders using deep learning. Nat Med 25, 60–64 (2019). https://doi.org/10.1038/s41591-018-0279-0
Evidence-Based Complementary and Alternative Medicine (2021)
“Did you know this camera tracks your mood?”: Understanding Privacy Expectations and Preferences in the Age of Video Analytics
Proceedings on Privacy Enhancing Technologies (2021)
Big Data Research (2021)
Clinical and Translational Science (2021)