Abstract

Syndromic genetic conditions, in aggregate, affect 8% of the population1. Many syndromes have recognizable facial features2 that are highly informative to clinical geneticists3,4,5. Recent studies show that facial analysis technologies measured up to the capabilities of expert clinicians in syndrome identification6,7,8,9. However, these technologies identified only a few disease phenotypes, limiting their role in clinical settings, where hundreds of diagnoses must be considered. Here we present a facial image analysis framework, DeepGestalt, using computer vision and deep-learning algorithms, that quantifies similarities to hundreds of syndromes. DeepGestalt outperformed clinicians in three initial experiments, two with the goal of distinguishing subjects with a target syndrome from other syndromes, and one of separating different genetic subtypes in Noonan syndrome. On the final experiment reflecting a real clinical setting problem, DeepGestalt achieved 91% top-10 accuracy in identifying the correct syndrome on 502 different images. The model was trained on a dataset of over 17,000 images representing more than 200 syndromes, curated through a community-driven phenotyping platform. DeepGestalt potentially adds considerable value to phenotypic evaluations in clinical genetics, genetic testing, research and precision medicine.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Data availability

The data that support the findings of this study are divided into two groups, published data and restricted data. Published data are available from the reported references and also in Supplementary Table 6. Restricted data are curated from Face2Gene users under a license and cannot be published, to protect patient privacy.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Baird, P. A., Anderson, T., Newcombe, H. & Lowry, R. Genetic disorders in children and young adults: a population study. Am. J. Hum. Genet. 42, 677–693 (1988).

  2. 2.

    Hart, T. & Hart, P. Genetic studies of craniofacial anomalies: clinical implications and applications. Orthod. Craniofac. Res. 12, 212–220 (2009).

  3. 3.

    Ferry, Q. et al. Diagnostically relevant facial gestalt information from ordinary photos. eLife 3, e02020 (2014).

  4. 4.

    Basel-Vanagaite, L. et al. Recognition of the Cornelia de Lange syndrome phenotype with facial dysmorphology novel analysis. Clin. Genet. 89, 557–563 (2016).

  5. 5.

    Rai, M. C. E., Werghi, N., Al Muhairi, H. & Alsafar, H. Using facial images for the diagnosis of genetic syndromes: a survey. In 2015 International Conference on Communications, Signal Processing, and their Applications (ICCSPA) (2015).

  6. 6.

    Shukla, P., Gupta, T., Saini, A., Singh, P. & Balasubramanian, R. A deep learning frame-work for recognizing developmental disorders. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2017).

  7. 7.

    Hadj-Rabia, S. et al. Automatic recognition of the XLHED phenotype from facial images. Am. J. Med. Genet. A. 173, 2408–2414 (2017).

  8. 8.

    Valentine, M. et al. Computer-aided recognition of facial attributes for Fetal Alcohol Spectrum disorders. Pediatrics 140, e20162028 (2017).

  9. 9.

    Gripp, K. W., Baker, L., Telegrafi, A. & Monaghan, K. G. The role of objective facial analysis using FDNA in making diagnoses following whole exome analysis. Report of two patients with mutations in the BAF complex genes. Am. J. Med. Genet. A. 170, 1754–1762 (2016).

  10. 10.

    Delgadillo, V., Maria del Mar, O., Gort, L., Coll, M. J. & Pineda, M. Natural history of Sanfilippo syndrome in Spain. Orphanet J. Rare Dis. 8, 189 (2013).

  11. 11.

    Kole, A. et al. The Voice of 12,000 Patients: experiences and expectations of rare disease patients on diagnosis and care in Europe. Eurordis http://www.eurordis.org/IMG/pdf/voice_12000_patients/EURORDISCARE_FULLBOOKr.pdf (2009).

  12. 12.

    Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014 1701–1708 (IEEE, 2014).

  13. 13.

    Huang, G. B., Ramesh, M., Berg, T. & Learned-Miller, E. Labeled faces in the Wild: a database for studying face recognition in unconstrained environments. In Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition (2008).

  14. 14.

    Yi, D., Lei, Z., Liao, S. & Li, S. Z. Learning face representation from scratch. Preprint at https://arxiv.org/abs/1411.7923 (2014).

  15. 15.

    Schroff, F., Kalenichenko, D. & Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 815–823 (IEEE,2015).

  16. 16.

    Li, H., Lin, Z., Shen, X., Brandt, J. & Hua, G. A convolutional neural network cascade for face detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 5325–5334 (IEEE, 2015).

  17. 17.

    Karlinsky, L. & Ullman, S. Using linking features in learning non-parametric part models. Computer Vision–ECCV 2012, 326–339 (2012).

  18. 18.

    Rohatgi, S. et al. Facial diagnosis of mild and variant CdLS: Insights from a dysmorphologist survey. Am. J. Med. Genet. A. 152, 1641–1653 (2010).

  19. 19.

    Bird, L. M., Tan, W. H. & Wolf, L. The role of computer-aided facial recognition technology in accelerating the identification of Angelman syndrome. In 35th Annual David W Smith Workshop (2014).

  20. 20.

    Allanson, J. E. et al. The face of Noonan syndrome: does phenotype predict genotype. Am. J. Med. Genet. A. 152, 1960–1966 (2010).

  21. 21.

    Gulec, E. Y., Ocak, Z., Candan, S., Ataman, E. & Yarar, C. Novel mutations in PTPN11 gene in two girls with Noonan syndrome phenotype. Int. J. Cardiol. 186, 13–15 (2015).

  22. 22.

    Zenker, M. et al. SOS1 is the second most common Noonan gene but plays no major role in cardio-facio-cutaneous syndrome. J. Med. Genet. 44, 651–656 (2007).

  23. 23.

    Rusu, C., Idriceanu, J., Bodescu, I., Anton, M. & Vulpoi, C. Genotype-phenotype correlations in Noonan Syndrome. Acta Endocrinologica 10, 463–476 (2014).

  24. 24.

    Cavé, H. et al. Mutations in RIT1 cause Noonan syndrome with possible juvenile myelomonocytic leukemia but are not involved in acute lymphoblastic leukemia. Eur. J. Hum. Genet. 24, 1124–1131 (2016).

  25. 25.

    Kouz, K. et al. Genotype and phenotype in patients with Noonan syndrome and a RIT1 mutation. Genet. Med. 18, 1226–1234 (2016).

  26. 26.

    Winter, R. M. & Baraitser The London Dysmorphology Database. J. Med. Genet. 24, 509–510 (1987).

  27. 27.

    Robinson, P. N. & Mundlos, S. The human phenotype ontology. Clin. Genet. 77, 525–534 (2010).

  28. 28.

    Köhler, S. et al. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. A. J. Hum. Genet. 85, 457–464 (2009).

  29. 29.

    Zarate, Y. A. et al. Natural history and genotype-phenotype correlations in 72 individuals with SATB2-associated syndrome. Am. J. Med. Genet. A. 176, 925–935 (2018).

  30. 30.

    Liehr, T. et al. Next generation phenotyping in Emanuel and Pallister Killian Syndrome using computer-aided facial dysmorphology analysis of 2D photos. Clin. Genet. 93, 378–381 (2017).

  31. 31.

    Hennekam, R. & Biesecker, L. G. Next-generation sequencing demands next-generation phenotyping. Hum. Mutat. 33, 884–886 (2012).

  32. 32.

    Huang, G., Mattar, M., Lee, H. & Learned-Miller, E. G. Learning to align from scratch. In Advances in Neural Information Processing Systems 2012 764–772 (2012).

  33. 33.

    Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 2014 3320–3328 (2014).

  34. 34.

    Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Web-scale training for face identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 2746–2754 (IEEE, 2015).

  35. 35.

    Zhou, E., Cao, Z. & Yin, Q. Naive-deep face recognition: touching the limit of LFW benchmark or not? Preprint at https://arxiv.org/abs/1501.04690 (2015).

  36. 36.

    Liu, J., Deng, Y., Bai, T., Wei, Z. & Huang, C. Targeting ultimate accuracy: face recognition via deep embedding. Preprint at https://arxiv.org/abs/1506.07310 (2015).

  37. 37.

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).

  38. 38.

    Parkhi, O. M., Vedaldi, A. & Zisserman, A. Deep face recognition. In Proceedings of the British Machine Vision Conference 1, 6 (2015).

  39. 39.

    Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. International Conference on Machine Learning 2015 448–456 (2015).

  40. 40.

    Chollet, F. et al. Keras. http://keras.io (2015).

  41. 41.

    Abadi, M. et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467 (2016).

  42. 42.

    He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proc. IEEE International Conference on Computer Vision 1026–1034 (IEEE, 2015).

  43. 43.

    Kingma, D. and Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  44. 44.

    Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international Conference on Artificial Intelligence and Statistics 249–256 (2010).

  45. 45.

    Efron, B. Bootstrap methods: another look at the jackknife. In Breakthroughs in Statistics 569–593 (Springer, New York, 1992).

Download references

Acknowledgements

The authors thank the patients and their families, as well as Face2Gene users worldwide who contribute with their knowledge and dedication to the improvement of this and other tools for the ultimate benefit of better healthcare.

Author information

Affiliations

  1. FDNA Inc., Boston, MA, USA

    • Yaron Gurovich
    • , Yair Hanani
    • , Omri Bar
    • , Guy Nadav
    • , Nicole Fleischer
    •  & Dekel Gelbman
  2. Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel

    • Lina Basel-Salmon
  3. Recanati Genetic Institute, Rabin Medical Center & Schneider Children’s Medical Center, Petah Tikva, Israel

    • Lina Basel-Salmon
  4. Institute for Genomic Statistic and Bioinformatics, University Hospital Bonn, Rheinische-Friedrich-Wilhelms University, Bonn, Germany

    • Peter M. Krawitz
  5. Institute of Human Genetics, University Hospital Magdeburg, Magdeburg, Germany

    • Susanne B. Kamphausen
    •  & Martin Zenker
  6. Department of Pediatrics, University of California San Diego, San Diego, CA, USA

    • Lynne M. Bird
  7. Division of Genetics/Dysmorphology, Rady Children’s Hospital San Diego, San Diego, CA, USA

    • Lynne M. Bird
  8. Division of Medical Genetics, A. I. du Pont Hospital for Children/Nemours, Wilmington, DE, USA

    • Karen W. Gripp

Authors

  1. Search for Yaron Gurovich in:

  2. Search for Yair Hanani in:

  3. Search for Omri Bar in:

  4. Search for Guy Nadav in:

  5. Search for Nicole Fleischer in:

  6. Search for Dekel Gelbman in:

  7. Search for Lina Basel-Salmon in:

  8. Search for Peter M. Krawitz in:

  9. Search for Susanne B. Kamphausen in:

  10. Search for Martin Zenker in:

  11. Search for Lynne M. Bird in:

  12. Search for Karen W. Gripp in:

Contributions

Y.G., Y.H. and O.B. initiated the project. Y.G., Y.H. and D.G. developed the DeepGestalt framework. N.F., L.B.-S., P.M.K., S.B.K., M.Z., L.M.B. and K.W.G. designed and conducted the clinical experiments. O.B. and G.N. finalized the dataset, computed statistics and contributed to the software engineering. Y.G., Y.H., O.B., P.M.K. and K.W.G. contributed to the writing of the manuscript.

Competing interests

Y.G., Y.H., O.B., G.N., N.F. and D.G. are employees of FDNA; L.B.-S., P.M.K. and K.W.G. are advisors of FDNA; L.B.-S., P.M.K., M.Z., L.M.B. and K.W.G. are members of the scientific advisory board of FDNA.

Corresponding author

Correspondence to Yaron Gurovich.

Supplementary information

  1. Supplementary Text and Figures

    Supplementary Figures 1 and 2 and Supplementary Tables 1–6

  2. Reporting Summary

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41591-018-0279-0