Deep learning is an emerging transformative tool in diagnostic medicine, yet limited access and the interpretability of learned parameters hinders widespread adoption. Here we have generated a diverse repository of 838,644 histopathologic images and used them to optimize and discretize learned representations into 512-dimensional feature vectors. Importantly, we show that individual machine-engineered features correlate with salient human-derived morphologic constructs and ontological relationships. Deciphering the overlap between human and machine reasoning may aid in eliminating biases and improving automation and accountability for artificial intelligence-assisted medicine.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The full image training datasets used to generate the DLFVs are available on Bitbucket (https://bitbucket.org/diamandislabii/faust-feature-vectors-2019) (https://doi.org/10.5281/zenodo.3234829). External testing images generated by the TCGA Research Network (http://cancergenome.nih.gov/) are also publicly available without restrictions.
Code and both the general and 1p19q-codeletion trained CNN models are available on Bitbucket (https://bitbucket.org/diamandislabii/faust-feature-vectors-2019 and https://doi.org/10.5281/zenodo.3234829). A stand-alone computing capsule is available on Code Ocean for Feature Vector Extraction and Clustering (https://doi.org/10.24433/CO.3573560.v2) and for feature activation map (FAM) generation (https://doi.org/10.24433/CO.7749421.v1).
Lotz, J. M., Primack, J. & Madau, P. A new nonparametric approach to galaxy morphological classification. Astron. J. 128, 163–182 (2004).
Murphy, W. J. et al. Molecular phylogenetics and the origins of placental mammals. Nature 409, 614–618 (2001).
Louis, D. N. et al. The 2016 World Health Organization classification of tumors of the central nervous system: a summary. Acta Neuropathol. 131, 803–820 (2016).
Nicholson, A. G. et al. Inter-observer variation between pathologists in diffuse parenchymal lung disease. Thorax 59, 500–505 (2004).
Gilles, F. H. et al. Pathologist interobserver variability of histologic features in childhood brain tumors: results from the CCG-945 study. Pediatr. Dev. Pathol. 11, 108–117 (2008).
Allard, F. D. et al. Intraobserver and interobserver variability in the assessment of dysplasia in ampullary mucosal biopsies. Am. J. Surg. Pathol. 42, 1095–1100 (2018).
Naghavi, M. et al. Global, regional and national age-sex specific mortality for 264 causes of death, 1980–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 390, 1151–1210 (2017).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
Servick, K. Brain scientists dive into deep neural networks. Science 361, 1177 (2018).
Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at http://arxiv.org/abs/1409.1556 (2014).
Holzinger, A. et al. Causability and explainabilty of artificial intelligence in medicine. WIRES Data Min. Knowl. Discov. 9, e1312 (2019).
Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. Preprint at http://arxiv.org/abs/1702.08608(2017).
Samek, W., Wiegand, T. & Müller, K.-R. Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. Preprint at http://arxiv.org/abs/1708.08296 (2017).
Faust, K. et al. Visualizing histopathologic deep learning classification and anomaly detection using nonlinear feature space dimensionality reduction. BMC Bioinformatics 19, 173 (2018).
Cancer Genome Atlas Research Network et al.Comprehensive, integrative genomic analysis of diffuse lower-grade gliomas. N. Engl. J. Med. 372, 2481–2498 (2015).
Funding support for this research and trainees is provided by the Princess Margaret Cancer Foundation, an Adam Coules Research Grant, the Department of Laboratory Medicine and Pathology at the University Health Network, the Brain Tumour Foundation of Canada, the American Society of Clinical Oncology Career Development Award (ASCO-CDA) and The Brain Tumour Charity Expanding Theories Research Grant Program (GN-000560). We thank P. Boutros for critical feedback and suggestions.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Faust, K., Bala, S., van Ommeren, R. et al. Intelligent feature engineering and ontological mapping of brain tumour histomorphologies by deep learning. Nat Mach Intell 1, 316–321 (2019). https://doi.org/10.1038/s42256-019-0068-6