Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Augmenting vascular disease diagnosis by vasculature-aware unsupervised learning

Nature Machine Intelligence volume 2, pages 337–346 (2020)Cite this article

Subjects

An Author Correction to this article was published on 08 January 2021

This article has been updated

A preprint version of the article is available at bioRxiv.

Abstract

Vascular disease is one of the leading causes of death and threatens human health worldwide. Imaging examination of vascular pathology with reduced invasiveness is challenging due to the intrinsic vasculature complexity and non-uniform scattering from bio-tissues. Here, we report VasNet, a vasculature-aware unsupervised learning algorithm that augments pathovascular recognition from small sets of unlabelled fluorescence and digital subtraction angiography images. VasNet adopts a multi-scale fusion strategy with a domain adversarial neural network loss function that induces biased pattern reconstruction by strengthening features relevant to the retinal vasculature reference while weakening irrelevant features. VasNet delivers the outputs ‘Structure + X’ (where X refers to multi-dimensional features such as blood flows, the distinguishment of blood dilation and its suspicious counterparts, and the dependence of new pattern emergence on disease progression). Therefore, explainable imaging output from VasNet and other algorithm extensions holds the promise to augment medical diagnosis, as it improves performance while reducing the cost of human expertise, equipment and time consumption.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The augmentation principle of vascular disease diagnosis.
Fig. 2: VasNet architecture.
Fig. 3: Performance evaluation of VasNet and other existing techniques.
Fig. 4: Augmenting the diagnosis of mouse cerebral vascular disease.
Fig. 5: Augmenting the diagnosis of bleeding from dynamic DSA data.
Fig. 6: Augmenting the diagnosis of bleeding from static DSA data in the human abdomen.
Fig. 7: Discovering a new diagnosis of mouse bowel vascular disease.

Similar content being viewed by others

Data availability

A pretrained model and all experimental images appearing in the Article and the Supplementary Information are publicly available at https://doi.org/10.6084/m9.figshare.11986962. Other detailed data that support the findings of this study are available from the corresponding author upon reasonable request.

Code availability

Our PyTorch implementation of VasNet, training data and test samples are available from GitHub (https://github.com/mjiUST/VasNet) or https://doi.org/10.5281/zenodo.3762820.

Change history

  • 08 January 2021

    A Correction to this paper has been published: https://doi.org/10.1038/s42256-020-0188-z

References

  1. Tortora, G. J. & Derrickson, B. H. Principles of Anatomy and Physiology (Wiley, 2018).

  2. Field, T. S. & Hill, M. D. Cerebral venous thrombosis: we should ask the right questions to get better answers. Stroke 50, 1598–1604 (2019).

    Article  Google Scholar 

  3. Portegies, M., Koudstaal, P. & Ikram, M. Cerebrovascular disease. Handbook Clin. Neurol. 138, 239–261 (2016).

    Article  Google Scholar 

  4. Hu, X., De Silva, T. M., Chen, J. & Faraci, F. M. Cerebral vascular disease and neurovascular injury in ischemic stroke. Circ. Res. 120, 449–471 (2017).

    Article  Google Scholar 

  5. Brown, R. D. Jr & Broderick, J. P. Unruptured intracranial aneurysms: epidemiology, natural history, management options and familial screening. Lancet Neurol. 13, 393–404 (2014).

    Article  Google Scholar 

  6. Kisler, K., Nelson, A. R., Montagne, A. & Zlokovic, B. V. Cerebral blood flow regulation and neurovascular dysfunction in Alzheimer disease. Nat. Rev. Neurosci. 18, 419–434 (2017).

    Article  Google Scholar 

  7. Sweeney, M. D., Kisler, K., Montagne, A., Toga, A. W. & Zlokovic, B. V. The role of brain vasculature in neurodegenerative disorders. Nat. Neurosci. 21, 1318–1331 (2018).

    Article  Google Scholar 

  8. Franks, I. Gut microbes might promote intestinal angiogenesis. Nat. Rev. Gastroenterol. Hepatol. 10, 3 (2013).

    Article  Google Scholar 

  9. Hanauer, S. B. Update on the etiology, pathogenesis and diagnosis of ulcerative colitis. Nat. Clin. Pract. Gastroenterol. Hepatol. 1, 26–31 (2004).

    Article  Google Scholar 

  10. Torres, J., Mehandru, S., Colombel, J.-F. & Peyrin-Biroulet, L. Crohn’s disease. Lancet 389, 1741–1755 (2017).

    Article  Google Scholar 

  11. Lopera, J. E. Embolization in trauma: principles and techniques. Semin. Intervent. Radiol. 27, 014–028 (2010).

    Article  Google Scholar 

  12. Burke, C. T. & Mauro, M. A. Bronchial artery embolization. Semin. Intervent. Radiol. 21, 43–48 (2004).

    Article  Google Scholar 

  13. Rilling, W. S. & Chen, G. W. Preoperative embolization. Semin. Intervent. Radiol. 21, 3–9 (2004).

    Article  Google Scholar 

  14. Hong, G., Antaris, A. L. & Dai, H. Near-infrared fluorophores for biomedical imaging. Nat. Biomed. Eng. 1, 0010 (2017).

    Article  Google Scholar 

  15. Wan, H. et al. A bright organic NIR-II nanofluorophore for three-dimensional imaging into biological tissues. Nat. Commun. 9, 1171 (2018).

    Article  Google Scholar 

  16. Hong, G. et al. Through-skull fluorescence imaging of the brain in a new near-infrared window. Nat. Photon. 8, 723–730 (2014).

    Article  Google Scholar 

  17. Martínez-Corral, I. et al. In vivo imaging of lymphatic vessels in development, wound healing, inflammation and tumor metastasis. Proc. Natl Acad. Sci. USA 109, 6223–6228 (2012).

    Article  Google Scholar 

  18. Jia, Y. et al. Quantitative optical coherence tomography angiography of vascular abnormalities in the living human eye. Proc. Natl Acad. Sci. USA 112, E2395–E2402 (2015).

    Article  Google Scholar 

  19. Zhang, W. et al. High-resolution, in vivo multimodal photoacoustic microscopy, optical coherence tomography and fluorescence microscopy imaging of rabbit retinal neovascularization. Light Sci. Appl. 7, 103 (2018).

    Article  Google Scholar 

  20. Sakadžić, S. & Wang, L. V. High-resolution ultrasound-modulated optical tomography in biological tissues. Opt. Lett. 29, 2770–2772 (2004).

    Article  Google Scholar 

  21. Yao, J., Maslov, K. I., Shi, Y., Taber, L. A. & Wang, L. V. In vivo photoacoustic imaging of transverse blood flow by using Doppler broadening of bandwidth. Opt. Lett. 35, 1419–1421 (2010).

    Article  Google Scholar 

  22. Huang, C.-H. et al. High-resolution structural and functional assessments of cerebral microvasculature using 3D Gas ΔR2*-mMRA. PLoS ONE 8, e78186 (2013).

    Article  Google Scholar 

  23. Wang, L. V. & Yao, J. A practical guide to photoacoustic tomography in the life sciences. Nat. Methods 13, 627–638 (2016).

    Article  Google Scholar 

  24. Nasiriavanaki, M. et al. High-resolution photoacoustic tomography of resting-state functional connectivity in the mouse brain. Proc. Natl Acad. Sci. USA 111, 21–26 (2014).

    Article  Google Scholar 

  25. Meijering, E. H., Niessen, W. J. & Viegever, M. Retrospective motion correction in digital subtraction angiography: a review. IEEE Trans. Med. Imaging 18, 2–21 (1999).

    Article  Google Scholar 

  26. Jeans, W. The development and use of digital subtraction angiography. Br. J. Radiol. 63, 161–168 (1990).

    Article  Google Scholar 

  27. Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).

    Article  Google Scholar 

  28. Wang, Y. et al. Accurate quantification of astrocyte and neurotransmitter fluorescence dynamics for single-cell and population-level physiology. Nat. Neurosci. 22, 1936–1944 (2019).

    Article  Google Scholar 

  29. Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).

    Article  Google Scholar 

  30. Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131 (2018).

    Article  Google Scholar 

  31. Cole, J. H. et al. Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. NeuroImage 163, 115–124 (2017).

    Article  Google Scholar 

  32. Li, S., Deng, M., Lee, J., Sinha, A. & Barbastathis, G. Imaging through glass diffusers using densely connected convolutional networks. Optica 5, 803–813 (2018).

    Article  Google Scholar 

  33. Li, Y., Xue, Y. & Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181–1190 (2018).

    Article  Google Scholar 

  34. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019).

    Article  Google Scholar 

  35. Lee, H. et al. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat. Biomed. Eng. 3, 173–182 (2019).

    Article  Google Scholar 

  36. Mahapatra, D., Bozorgtabar, B., Hewavitharanage, S. & Garnavi, R. Image super resolution using generative adversarial networks and local saliency maps for retinal image analysis. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention 382–390 (Springer, 2017).

  37. Son, J., Park, S. J. & Jung, K.-H. Retinal vessel segmentation in fundoscopic images with generative adversarial networks. Preprint at https://arxiv.org/abs/1706.09318 (2017).

  38. Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision 2223–2232 (IEEE, 2017).

  39. Murray, C. D. The physiological principle of minimum work: I. The vascular system and the cost of blood volume. Proc. Natl Acad. Sci. USA 12, 207–214 (1926).

    Article  Google Scholar 

  40. West, G. B., Brown, J. H. & Enquist, B. J. A general model for the origin of allometric scaling laws in biology. Science 276, 122–126 (1997).

    Article  Google Scholar 

  41. Ganin, Y. et al. Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17, 2096–2030 (2016).

    MathSciNet  Google Scholar 

  42. Dey, N. et al. Richardson–Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution. Microsc. Res. Tech. 69, 260–266 (2006).

    Article  Google Scholar 

  43. Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).

    Article  Google Scholar 

  44. Lu, B., Chen, J.-C. & Chellappa, R. Unsupervised domain-specific deblurring via disentangled representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 10225–10234 (IEEE, 2019).

  45. Dice, L. R. Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945).

    Article  Google Scholar 

  46. Gegundez-Arias, M. E., Aquino, A., Bravo, J. M. & Marin, D. A function for quality evaluation of retinal vessel segmentations. IEEE Trans. Med. Imaging 31, 231–239 (2012).

    Article  Google Scholar 

  47. Laroui, H. et al. Dextran sodium sulfate (DSS) induces colitis in mice by forming nano-lipocomplexes with medium-chain-length fatty acids in the colon. PLoS ONE 7, e32084 (2012).

    Article  Google Scholar 

  48. Chassaing, B., Aitken, J. D., Malleshappa, M. & Vijay‐Kumar, M. Dextran sulfate sodium (DSS)‐induced colitis in mice. Curr. Protoc. Immunol. 104, 15–25 (2014).

    Google Scholar 

  49. van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

    MATH  Google Scholar 

  50. O’Donnell, T. F. Jr, Rasmussen, J. C. & Sevick-Muraca, E. M. New diagnostic modalities in the evaluation of lymphedema. J. Vasc. Surg, Venous Lymphat. Disord. 5, 261–273 (2017).

    Article  Google Scholar 

  51. Greives, M. R., Aldrich, M. B., Sevick-Muraca, E. M. & Rasmussen, J. C. Near-infrared fluorescence lymphatic imaging of a toddler with congenital lymphedema. Pediatrics 139, e20154456 (2017).

    Article  Google Scholar 

  52. Tzeng, E., Hoffman, J., Darrell, T. & Saenko, K. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision 4068–4076 (IEEE, 2015).

  53. Fraz, M. M. et al. Blood vessel segmentation methodologies in retinal images—a survey. Comput. Methods Programs Biomed. 108, 407–433 (2012).

    Article  Google Scholar 

  54. Staal, J., Abràmoff, M. D., Niemeijer, M., Viergever, M. A. & Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23, 501–509 (2004).

    Article  Google Scholar 

  55. Watts, L. T., Zheng, W., Garling, R. J., Frohlich, V. C. & Lechleiter, J. D. Rose Bengal photothrombosis by confocal optical imaging in vivo: a model of single vessel stroke. J. Visualized Exp. 23, e52794 (2015).

    Google Scholar 

Download references

Acknowledgements

We thank H. Fan for insightful suggestions regarding the optics experiments, X. Yuan for camera troubleshooting, J. Cai for fruitful discussions about data processing in the bowel vasculature section and H. Zhao for drawing illustrations. The work was supported by the National Natural Science Foundation of China (grants 61722209, 61860206003, 61971255 and 81872368), funds from the Shenzhen Science and Technology Innovation Committee (grants KQJSCX20180327143623167 and JCYJ20180508152130899) and a fund from Shenzhen Development and Reform Commission Subject Construction Project [2017]143.

Author information

Author notes

  1. These authors contributed equally: Yong Wang, Mengqi Ji, Shengwei Jiang.

Authors and Affiliations

  1. Tsinghua–Berkeley Shenzhen Institute (TBSI), Tsinghua Shenzhen International Graduate School (SIGS), Tsinghua University, Shenzhen, China

    Yong Wang, Mengqi Ji, Shengwei Jiang, Laiqiang Huang, Shaohua Ma, Lu Fang & Qionghai Dai

  2. Department of Automation, Tsinghua University, Beijing, China

    Mengqi Ji, Xukang Wang, Jiamin Wu, Jingtao Fan & Qionghai Dai

  3. Interventional Radiology Department, The First Medicine Centre, Chinese PLA General Hospital, Beijing, China

    Feng Duan

  4. Institute of Brain and Cognitive Sciences, Tsinghua University, Beijing, China

    Jingtao Fan, Shaohua Ma, Lu Fang & Qionghai Dai

Authors
  1. Yong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mengqi Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengwei Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xukang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiamin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Feng Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jingtao Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Laiqiang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Shaohua Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lu Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Qionghai Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

L.F., S.M., Q.D. and L.H. conceived this project. S.M., L.F., Y.W., S.J. and M.J. designed the experiments. Y.W. and S.J. performed animal experiments. M.J. innovated and implemented the algorithm. M.J., X.W., Y.W. and J.F. tested the algorithm and performed the data analysis. F.D. contributed the X-ray and DSA data under ethical approval from the PLA General Hospital. Q.D. oversaw the experimental design and progress. S.M., L.F., M.J., Y.W. and J.W. prepared the manuscript. All authors reviewed the paper.

Corresponding authors

Correspondence to Shaohua Ma, Lu Fang or Qionghai Dai.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–14, Discussion and Table 1.

Reporting Summary

Supplementary Video 1

In vivo fluorescence imaging of cerebral vasculature of a healthy mouse at a 25 Hz frame acquisition rate. As ICG is injected, VasNet recovers the dynamic changes of vascular structure of a healthy cerebrum.

Supplementary Video 2

In vivo fluorescence imaging of cerebral vasculature of a mouse with thrombosis model at a 25 Hz frame acquisition rate. As ICG is injected, VasNet recovers the dynamic changes of vascular structure of a diseased cerebrum.

Supplementary Video 3

Augmenting diagnosis of bleeding from dynamic DSA vascular images acquired from human. VasNet outputs image with vasculature reconstructed. The bleeding features are augmented along with the dynamics of dilation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Ji, M., Jiang, S. et al. Augmenting vascular disease diagnosis by vasculature-aware unsupervised learning. Nat Mach Intell 2, 337–346 (2020). https://doi.org/10.1038/s42256-020-0188-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-020-0188-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing