Vascular disease is one of the leading causes of death and threatens human health worldwide. Imaging examination of vascular pathology with reduced invasiveness is challenging due to the intrinsic vasculature complexity and non-uniform scattering from bio-tissues. Here, we report VasNet, a vasculature-aware unsupervised learning algorithm that augments pathovascular recognition from small sets of unlabelled fluorescence and digital subtraction angiography images. VasNet adopts a multi-scale fusion strategy with a domain adversarial neural network loss function that induces biased pattern reconstruction by strengthening features relevant to the retinal vasculature reference while weakening irrelevant features. VasNet delivers the outputs ‘Structure + X’ (where X refers to multi-dimensional features such as blood flows, the distinguishment of blood dilation and its suspicious counterparts, and the dependence of new pattern emergence on disease progression). Therefore, explainable imaging output from VasNet and other algorithm extensions holds the promise to augment medical diagnosis, as it improves performance while reducing the cost of human expertise, equipment and time consumption.
Human organs are infiltrated with blood vessels that supply oxygen and nutrients and remove metabolic byproducts from living tissue cells1. Vascular abnormalities, both structural and dynamic, have strong pathologic indications in a wide range of diseases. Common cerebral vascular diseases, including atherosclerosis, thrombosis and aneurysm, can cause stroke2,3,4,5 and many forms of neurological dysfunction and degeneration6,7. Inflammatory bowel diseases (IBDs) and bowel microbial infections are associated with enriched vasculature and microbleeds along the digestive tract8,9,10. Internal bleeding is one major factor leading to mortality after trauma11 and haemoptysis12. Hypervascular tumours13 feature enriched angiogenesis. Therefore, probing abnormal vasculature has significant impacts for a range of biomedical examinations.
Vascular imaging with reduced invasiveness requires zero or minimal incision of patients or experimental animals, but introduces some challenges to imaging-based diagnosis. Vascular networks are covered by biological structures such as skin and organ tissue, introducing substantial light scattering in optical investigations such as fluorescence imaging14,15,16, optical coherence tomography (OCT)17,18 and photoacoustic angiography19,20,21. Topologies are impacted by heavy and inhomogeneous noise that disrupts the recognition of pathovascular features. Also, when investigating internal bleeding using X-ray computed tomography (CT) and digital subtraction angiography (DSA)22,23,24,25,26, similar pathological features in solid organs, such as vessel bleeding, stricture, aneurysm and tortuous blood vessels, are difficult to distinguish due to the artefacts introduced by organ motion and vascular network complexity.
Deconvolution is a relatively common solution to extracting information from noise-polluted signals, and usually requires estimating or measuring the point spread function (PSF) of the system based on theoretical models. However, spatially inhomogeneous and individually different characteristics prevent non-blind deconvolution from being implementable in biomedical data. Learning-based techniques have recently been utilized to extract high-quality physiological and pathological features and perform recognition across a wide range of scales, including cells27,28, histology tissue slides29 and deep-tissue features acquired from OCT, CT and magnetic resonance imaging (MRI)30,31. Meanwhile, deep learning has proven to be versatile in extracting features from highly scattered patterns through synthetic diffusers32,33. Unfortunately, such learning-based descattering methods are supervised, and a large number of labelled and per-pixel registered data are necessary for training the neural networks. Moreover, a model trained on one location of the diffuser can hardly be extended to other regions due to the assumption of a spatial-invariant PSF for deconvolution. There is thus an essential conflict between limited data availability and the broad diversity of pathology features. On the other hand, the ‘black box’ operations of deep learning make decisions without explanations. This has accelerated advances in deep learning in medical diagnosis through establishing algorithms to ‘unbox’ learning and generate explainable outputs to render decision-making transparent34,35.
Several learning-based approaches have been reported that use gold-standard vascular images to extract explicit vasculature from original fundus images36,37. These models require large datasets of pixel-level aligned image pairs to train the generator network. The recent seminal unsupervised image-to-image method, ‘cycle-consistent generative adversarial networks’ (CycleGAN)38, has demonstrated powerful performance on transfer learning between two sets of images. These approaches treat the vascular information and degradation patterns equally and thus are not feasible for eliminating non-uniform noises induced by the heterogeneous diffusibility of skin and organ tissue, uncontrolled organ motion and non-uniform background illumination.
Here, we report VasNet, an unsupervised transfer learning technique specialized for vascular feature recognition. Vasculature in different organs evolves following a general physiological principle, Murray’s Law39,40, and shares an intrinsic commonality in appearance. Therefore, the widely accessible retinal vasculature in binary formats is chosen as the target domain of vascular-aware transfer learning, with the diffusive organ vasculature acquired from fluorescence or X-ray imaging as the source domain. This eliminates the dependence on large datasets of image registration and manual labelling of ground truths in supervised methods. The algorithm outputs explainable images with multi-dimensional information about blood flows, including the vascular structure, flow rates and the prescreened examination of suspicions. In this work, we have performed diagnosis augmentation of thrombosis and internal bleeding and proved the concept of establishing new diagnostics for ulcerative colitis in animal models.
Vascular diagnosis augmentation
The concept of augmenting vascular diagnosis operates by blind vasculature extraction from an unsupervised deep learning algorithm without using labelled data. It is, intrinsically, an image-to-image translation problem. The principle of diagnosis augmentation is outlined in Fig. 1. A domain-transferred image is consistently generated by the neural network from an input image containing vascular information and noise overlay, until the generated image becomes indistinguishable from the expected vasculature. To generate explicit vascular images using the unsupervised transfer learning technique, we chose a publicly available set of binary retinal vascular images as the target domain reference (Supplementary Fig. 1). The retinal images present high-contrast topologies of blood vessels against their surroundings, with enriched complexity, including vessel branching, size (diameter) variation, knots and endings.
VasNet utilizes the cycle consistency loss and learns the mapping between two image domains in the absence of labelled blood vessels (Fig. 2). A domain adversarial neural network (DANN)41 loss function is embedded in the algorithm to create bias in feature selection to improve vasculature extraction from inhomogeneous backgrounds. We feed the segmented retinal vasculature into VasNet as the target image domain (A domain) and the raw images acquired from fluorescence or X-ray imaging and overlaid with heterogeneous noises as the source image domain (B domain). The problem of blind vasculature extraction is then translated as disentanglement of the vascular structure zB from the degraded observations (B domain).
VasNet eliminates the requirement for large datasets of labelled images as ground truths to train neural networks. Instead of presenting an outcome with a binary decision to overlap or validate the human diagnosis, our strategy aims to augment human diagnosis by presenting the end-users with explainable images with enriched information. VasNet is vasculature-aware and delivers the results of ‘Structure + X’. X refers to multi-dimensional vascular characteristics such as blood flow, the examined blood dilation and its suspicious counterparts, and disease-induced subtle morphological changes, which might open a route towards alternative, yet more effective, disease diagnostic measures by examining new structure emergence (Fig. 1). In the following studies, we implemented VasNet to translate images of cerebral, bowel and internal solid organ vessels, overlaid with heterogeneous noise, into retinal-like topologies with explainable characteristics.
Unsupervised vasculature extraction
To recognize vascular features through intact diffusers such as the skin and solid organs that enclose blood vessels, we eliminated imposed noise by modelling the non-uniform descattering as an unsupervised mutual information disentanglement problem. Successful descattering was attributed to the adaptation and modification of the unsupervised domain-transfer network and the selection of a standard dataset of the retinal vascular network as reference for imaging generation (Supplementary Fig. 1). Determined by the binary retinal vascular network as the training reference, noise-corrupted images were disentangled into vascular-like features (defined as reference-relevant features) and noise (defined as reference-irrelevant features). VasNet discriminated the two types of content after training. Therefore, the algorithm testing performed unbalanced selectivity on the vascular-relevant and -irrelevant features.
In this study, as we were targeting vasculature reconstruction, the algorithm picked vascular-like features (such as regional features with high aspect ratios) and tended to disregard non-vascular features in the reconstruction. This resulted in significant differences in the output images on using VasNet compared with traditional schemes such as deconvolution42, the embedded vessel analysis tool in Fiji43 and learning-based methods, including CycleGAN38 and UDSD44 (Fig. 3). The performance of different algorithms was compared quantitatively and evaluated by the dice similarity coefficient (F1 score)45 and connectivity–area–length (CAL)46 metrics, in reference to the ground truths generated by human expert segmentation (Supplementary Fig. 2 and Supplementary Table 1). The visual presentation of CycleGAN outputs shows its weakness, that is, the reconstructed ‘vessels’ being discontinuous. This might be attributed to its lack of vascular-awareness in feature extraction. Similarly, CycleGAN tends to over-interpret the shadows in the images in Fig. 3d,e into ‘vessels’ featuring irregular dashes. The intrinsic awareness of vascular feature extraction in VasNet endows it with enhanced performance on images acquired across modalities, organ types and scales.
A major challenge in imaging-based vascular diagnosis is the loss of structural precision, especially the transverse dimensions of blood flows that indicate vascular abnormalities such as blood vessel stricture, dilation and breakage. The VasNet algorithm induces biased structural pattern reconstruction and recapitulates the topology of the vessels hidden in heterogeneous and complex noise. The versatility of VasNet with different imaging techniques and variations in vascular structure is demonstrated in Fig. 3 and Table 1. The reference retinal vascular image had enriched variation in vessel topology and size, but the distribution of vessel dimensions in the testing images was distinguishable from the reference images (Supplementary Fig. 3).
Augmenting the diagnosis of mouse cerebral vascular disease
Indocyanine green (ICG) is an FDA-approved dye molecule for clinical use that emits in the near-infrared (NIR) window (peak emission at ~820 nm). Specific binding of ICG to lipoproteins in the bloodstream prolongs the circulation lifetime of ICG in blood14. Light scattering in skin weakens with increased wavelength, but the images acquired in our experiments in the NIR-I region (800–900 nm) for feature recognition were highly disrupted with zero or minimal tissue incisions. The unlabelled diffused images were then input into the transfer learning network and translated into explainable images that showed both precise vascular structures and spatial flow dynamics in the mouse cerebrum.
The cerebral vasculature is covered by a translucent skull and intact scalp skin (~0.6 mm thick). It has a stem vein along the superior longitudinal sinus, with branched vessels distributed on the two sides. A stroke occurs when the blood supply is occluded. Thrombosis is a common type of occlusion resulting from blood clot formation inside vessels. In this study, fluorescence image sequences of mice cerebral vasculature were recorded through the intact scalp skin and the skull after intravenous injection of ICG. Both healthy mice and mice with thrombosis were imaged following the same procedures (see Methods for details).
The acquired image sequences contained both spatial- and temporal-dependent illumination of ICG circulating in the blood, from which blood flow rates could be derived. Owing to light scattering through the scalp and skull, only diffused vascular topology of healthy and diseased mice could be acquired (Fig. 4a,c). We applied VasNet to eliminate irrelevant information from the vasculature, including noise and non-vascular topologies. The algorithm-generated images exhibited high-contrast vascular features (Fig. 4b,d and Supplementary Videos 1 and 2) and were validated by the patterns obtained after removing the scalp (Supplementary Fig. 9). The global cerebral vasculature was constructed from the illumination sequence.
The blood flow rates throughout the network were derived from the illumination sequence and encoded in colours, based on the principle of the continuum of mass for incompressible fluids, that is, the blood (Fig. 4e). The colour coding shows drops in regional flow rates resulting from blood occlusion in the mice with thrombosis when compared with their healthy counterparts. The flow rates and their variations in different parts of the vascular network were found to be consistent in at least three healthy and three diseased mice (Fig. 4e–h and Supplementary Figs. 10 and 11). This proved that our algorithm was not only reliable in probing the structural abnormalities, but also the change in flow rates with high spatial signatures in the cerebrum. The structural-dynamic dual check augmented both the diagnosis efficiency and accuracy in non-invasive examinations.
Augmenting the diagnosis of human internal vascular disease
DSA is among the most important examination methods in the diagnosis and treatment of internal bleeding and hypervascular tumours via intervention embolization25,26. However, subject motion, respiration and the skeleton create severe artefacts and impose noise onto the vascular features, which may result in delayed diagnosis or a high rate of misdiagnosis. For example, bleeding, tortuous vessels and aneurysm are all characterized by dilation of the radiocontrast agents, and the overlaid noise can lead to the pathological vascular features and their normal counterparts being indistinguishable.
Although dilation of the radiocontrast agent (iodixanol) was observed in the DSA images (Fig. 5a,c), the artefacts caused by respiration-induced organ motion reduced the recognition of suspicious features or features hidden within the complex vascular background. Furthermore, the intervention embolization required a high-rate, or nearly real-time, diagnosis of diseased features with precision spatial resolution, as it was performed under the real-time guidance of DSA imaging.
The limitation was overcome by generating explainable images from the VasNet algorithm (Fig. 5b,d,e). The global vascular network was reconstructed with significantly improved accuracy and contrast against the background noise. The ROIs framed in black were interpreted with colour coding to exhibit the dilation in the recording. Dilation recognition in the raw input and output images was compared (Fig. 5f–i and Supplementary Video 3). The intensity contrast of the circulating agent in the dilation track, which contributed positively to diagnosis ability, was less distinctive in the DSA raw images than in the learning output images. The feature augmentation was performed in 40 groups of DSA image sequences, each acquired from an individual subject. Three examples are shown in Fig. 5 and Supplementary Fig. 12.
VasNet interpretation also augmented diagnosis from single static images. VasNet enhanced the vascular structure against the non-uniform degradation and succeeded in delivering outputs with ‘Structure + X’ in the DSA modality, where X helped to distinguish the blood dilation. To evaluate the effectiveness of augmenting the bleeding diagnosis quantitatively, we trained a basic classifier to determine whether the bleeding occurred or not, tested on the augmented vasculature of the suspicious patches. More specifically, data from 40 subjects were collected, half of which were adopted for training and the rest for testing.
The correctly classified positive (red) and negative (blue) samples are depicted in Fig. 6a,b. By interpreting the bleeding recognition results, we conclude that the classifier prefers to group together the augmented samples with narrow-neck connected expansion as positive predictions. For the case shown in Fig. 6c, which counted the bleeding and non-bleeding features of 20 patients, the bleeding diagnosis performance by the neural network, quantified by the area under the receiver operating characteristic (ROC) curve (AUC), reached 97%. The classification accuracy reached 88% when the cutoff probability was 0.4.
Discovering a new diagnosis of mouse bowel vascular disease
IBDs (predominantly ulcerative colitis (UC) and Crohn’s disease) involve mucosal inflammation and feature an increased density of abnormal vessels, such as strictures, ulcers, bleeding along the tract, as well as mucosal angiogenesis9,10. Mouse colitis models were established by watering BALB/c mice with dextran sulfate sodium (DSS, 5.0 wt% in drinking water). DSS is toxic to the colonic epithelia and induces erosion that ultimately compromises barrier integrity, resulting in increased colonic epithelial permeability47. Here, disease induction occurred within three to seven days following DSS administration and appeared as severe colonic bleeding. It mimicked the superficial inflammation seen in UC, which features enriched angiogenesis in the mucosa, but with nothing reported on the bowel outer surface48.
The bowel vasculature was covered by abdominal skin (~0.3 mm thick), which scattered emission from the circulating ICG in the vessels along the outer surface of the tract. Fluorescence image sequences of the mice bowel vasculature were recorded through the intact abdominal skin (see Methods for details). Both healthy (day 0) and DSS-administrated mice were imaged for control. The ground truths were accessed by cutting off the abdominal skin and the peritoneum and presenting the vascular features to the camera (Fig. 7a,b).
The explicit vascular topology extracted from the blurry raw images (Fig. 7c–f) was interpreted with various diversification methods, including the large vessel fraction and the vessel length and width (Fig. 7g,h). Figure 7g shows the statistics of vessel diversification of mice at days 0, 2, 4 and 6 after being administrated with DSS, by measuring the vessel connectivity, the mean branch length and the mean width of the reconstructed vessels. The connectivity was quantified by evaluating the fraction of connected vessels higher than a certain value in area (150 pixels in this case), indexed as ‘area percentage of large vessels’. The percentage was counted as the coverage fraction of large vessels over all vessels in each image. The mean branch lengths and widths were quantified by Fiji43. The quantification is presented in the corresponding boxplots in Fig. 7g. For better visualization of the multiple indices, we performed the t-SNE algorithm49 to present the diversification in two dimensions. The statistics show the distinct distribution of mice groups in different health conditions, as suggested by their disease progression, indexed by the number of days after DSS administration (Fig. 7h). However, the distinction was not always the same, as the separation by counting the area fraction of large vessels was the most significant, but least significant in the width statistics. The outlier at day 2 in the connectivity plot, indicated by a black arrow, was suggested to be heavily diseased. This was verified by skinless observation of the bowel condition (Supplementary Fig. 13). This study extends the promise of our unsupervised transfer learning technique in discovering new diagnostic characteristics.
In this work, we have reported an alternative implementation of deep learning techniques in augmenting biomedical diagnosis. We have established VasNet, an unsupervised transfer learning network, to overcome the challenges set by the limited data accessibility in biomedical imaging, the broad diversity of pathological features and the labour-intensive image registration and ground-truth labelling for training a neural network. We have embedded a multi-scale domain adversarial training scheme in the algorithm that simplifies disentanglement of the vascular structure and the modality-specific noise representation. This proved versatile for the interpretation of images acquired from different modalities and carrying structures with varied dimensions, size distributions and heterogeneous diffusion levels. Rather than providing a binary decision-making outcome, VasNet unboxes the feasibility of learning on disease diagnosis, and generates explainable images with multi-dimensional information, including vascular topology, flow rates, prescreened suspicions and statistics of blood vessel features. With this, we anticipate that the acceptance of AI decision-making might be boosted. Doctors can either accept a binary decision or make a decision based on the enriched information output from the neural network.
We have also proposed the idea of using deep learning techniques to discover new diagnostic characteristics. Unknown characteristics are usually hidden under heterogeneous noise, as in the vascular patterns shown in Fig. 7c,e. Although the distinction between the fluorescence images from healthy mice (Fig. 7c) and mice with colitis (Fig. 7e) is observable, it requires clear and amplified differences in pattern evolution with disease progress to distinguish the features in healthy, pseudo-healthy and diseased conditions. Our algorithm-dependent examination amplified this distinction and thus validated the diagnosis of colitis by the appearance of the vasculature on the exterior surfaces of the bowel. The colitis progresses with angiogenesis, which initiates at the early stage of disease occurrence.
We note that VasNet has limitations in extracting vascular features from sources with extremely low signal-to-noise ratios. It may interpret noises into hallucinations, an example of which is shown in Supplementary Fig. 14. However, because end-users tend to focus on areas of interest and play down fringe areas, the chances of hallucinations resulting in misdiagnosis are expected to be low.
The technology could be extended to the diagnosis of other types of disease, depending on feature classification and diversification. For example, lymphatic disorders, such as lymphoedema (LED), are examined by lymphoscintigraphy (LSG), but this technique is limited in its ability to identify pathology and guide therapy. Our VasNet-based vessel imaging augmentation might be able to assist in the rapid diagnosis of LED and also guide therapeutic intervention50,51.
Our diagnosis augmentation strategy reduces the dependence of imaging-based diagnosis on high-end equipment, and takes over some of the load on doctors by reducing the dependence on personal experience and repetitive labour-intensive practice and confirmation. With the advent of new diagnostic technologies, such as portable, easily affordable and domestic diagnostic instruments, professional users such as doctors can make faster and more accurate decisions about patients, while non-professional users, such as patients, medical interns and general practitioners, can perform pseudo-professional diagnoses.
To perform the training and testing of biomedical data with limited volume and enriched diversity, we innovated the unsupervised neural network to emphasize feature reconstruction for vascular disease diagnosis (VasNet). Its success is attributed to the adaptation and modification of the domain-transfer network and selection of the standard dataset of a retinal vascular network as the reference for imaging generation. The incorporation of a multi-scale fusion strategy with a DANN loss function creates a biased reconstruction on vascular-relevant patterns. Consecutively, VasNet improves vasculature extraction and reduces misinterpretation of heterogeneous noise.
Network structure. As shown in Fig. 2, VasNet decouples the structural information zA and zB from the target domain (A) and source domain (B) and the irrelevant content zBn. These embeddings are the outputs of encoders EA2z, EB2z and EB2zn, respectively. The target domain (A) is expected to contain the vascular structure of interest. Besides the vasculature content, the source domain (B) suffers from various types of degradation, such as non-uniform background and non-uniform degree of scattering, decoupled by the decoder EB2zn. To effectively disentangle the vasculature content, that is, the small gap between the statistics of the mutual structural features zA and zB, we propose to penalize the distribution shift between zA and zB by the domain adversarial loss proposed in the DANN41. Because the mutual information between zA and zB encodes the vasculature properties, the domain adversarial training scheme ensures that the inference network (in the blue dashed rectangle in Fig. 2) tends to disregard the vascular-irrelevant features, such as non-uniform background and scattering noise.
Biased feature reconstruction by DANN. Inspired by UDSD44, which achieved state-of-the-art domain-specific deblurring results based on disentangled representations, VasNet pays special attention to the content differences between the source domain and the target domain. Although UDSD tends to decompose the shared content from the input domains, the gradient signal of the structural encoders may vanish through a long path propagation through a stack of encoders and decoders, that is, from the rightmost outputs in Fig. 2a to the leftmost structural embeddings, which causes UDSD to fail to decompose the shared content. In contrast, VasNet extracts the shared content from both A and B domains by introducing the DANN loss to explicitly encourage the emergence of the statistically mutual features by penalizing the domain gap between embeddings zA and zB. The DANN loss function generates mutual features that are indiscriminable and intractable from their parental domain. In this way, VasNet effectively avoids the information loss due to the explicit guidance for structure extraction and reduces the structural domain shift between the source and target domains. The oscillation in the training process often existed because the DANN loss may lead to a small gap between two unfixed feature distributions, as mentioned by Tzeng and others52. To deal with this, we defined the domain adversarial loss using a cross-entropy loss function against random guess probability. Such domain adversarial loss encourages the emergence of indiscriminative structural information from both source and target domains.
Training details. Considering the existence of various widths of blood vessels, we further utilized a coarse-to-fine scheme by fusing the predictions of the input pyramid with different scales for both wide and narrow vessels. Training the algorithm with multiple vascular networks of different scales for each input was found to significantly slow down the training process and require extraordinarily large data volume. In contrast, by taking advantage of the fact that various scales of the vasculature share the similar topology and width distribution, our multi-scale fusion strategy led to considerably complete and accurate vasculature reconstruction. The proposed VasNet generated a 512 × 512 output within 0.1 s, working on a TITAN Xp GPU.
The publicly available retinal vasculature dataset CHASE_DB153 and DRIVE54 with manual segmentation were selected as the target domain images. Because the vascular networks of different scales shared similar topology, the algorithm only required inputting small patches (128 × 128) cropped from the raw large images. Similarly, the source domain images, which were from different modalities and had various degrees of scattering, were also cropped into small patches. The criterion used to determine the patch scale was to make the histograms of average vascular branch lengths from the two domains resemble each other.
The preparation of the source domain images is presented in the following. All animal studies and use of clinical data were conducted under the ethics regulations of Tsinghua University and the PLA General Hospital, China.
For the fluorescence imaging set-up, in both imaging experiments through the scalp and abdomen, the excitation light was provided by a 785 nm laser source (Changchun New Industries) coupled to an 8-mm collimator. The light was expanded and adjusted by an iris to illuminate the entire mouse head or abdomen area. The excitation power density at the imaging plane was ~29 mW cm−2 in the experiments. The emitted fluorescence was filtered by an 810–890 nm bandpass filter (Thorlabs, FBH850-40) and captured by a CMOS camera (Blackfly, BFS-U3-120S4M-CS) through a lens with a focal length of 25 mm. The camera was set to expose continuously at a frame rate of 25 frames per second to the fluorescence illumination process.
All animal experiments were performed under the National Institutes of Health Guide for the Care and Use of Laboratory Animals and approved by the Institutional Animal Care and Use Committee of Tsinghua University. BALB/c mice were purchased from Guangdong Medical Laboratory Animal Centre. Adult BALB/c mice (male, 4–6 weeks old) were housed at 22 ± 2 °C with a 50:50 light–dark cycle and were used with randomization. Before imaging, the hair on the heads and abdomens of the mice was removed using human depilatory cream.
In the cerebral thrombosis mouse model, BALB/c mice were anaesthetized by isoflurane and the hair over the scalp was removed. Each mouse was intravenously injected immediately with Rose Bengal at a dosage of 10 mg per kg body weight. A mouse model of thrombosis was created by intravenously injecting Rose Bengal and exposing a region of the cerebrum to 532-nm laser light (0.8 W) for 4 min through the intact scalp skin and the skull55 (Supplementary Fig. 8a). The radius of the laser spot was ~2.5 mm. Rose Bengal irradiated with green excitation light generated the production of reactive oxygen species, which subsequently activated a cascade of coagulation. The induction of the coagulation cascade produced an ischaemic lesion that was pathologically relevant to clinical stroke55. After one day, the mice were used for cerebral vasculature imaging.
For the UC mouse model, BALB/c mice were randomly assigned to the experimental or control groups (n = 5–6). For the experimental group, 5.0 wt% DSS in double-distilled water (DDW) was provided as drinking water for 2, 4 or 6 days. For the control group, DDW was always used as the drinking water. The treated and untreated mice were imaged on the abdomen on the same day.
For vasculature imaging, a BALB/c mouse with the hair removed from the head or abdomen was anaesthetized by isoflurane and fixed under the camera and lens. The area of interest (head or abdomen) was exposed under a 785-nm laser source (Supplementary Fig. 8b), then the mouse was intravenously injected immediately with ICG at a dosage of 8 mg per kg body weight. The ICG was dissolved in phosphate-buffered saline (PBS). The ICG molecules transported rapidly with the circulation, and the vasculature became fluorescent almost instantaneously after injection. Camera acquisition of the fluorescence images started immediately before the intravenous injection to record the illumination process.
To prepare the DSA imaging dataset, we collected an angiography dataset from 40 patients from the Chinese PLA General Hospital, Beijing. For each patient, during the dynamic process, between three and six images containing the vascular information were annotated with the real bleeding point region (accurate label) and two or three normal yet suspicious regions (false labels). To perform the bleeding detection task, we used DSA images of 20 patients for training and images of 20 patients for testing.
Further information on research design is available in the Nature Research Reporting Summary linked to this Article.
A pretrained model and all experimental images appearing in the Article and the Supplementary Information are publicly available at https://doi.org/10.6084/m9.figshare.11986962. Other detailed data that support the findings of this study are available from the corresponding author upon reasonable request.
Tortora, G. J. & Derrickson, B. H. Principles of Anatomy and Physiology (Wiley, 2018).
Field, T. S. & Hill, M. D. Cerebral venous thrombosis: we should ask the right questions to get better answers. Stroke 50, 1598–1604 (2019).
Portegies, M., Koudstaal, P. & Ikram, M. Cerebrovascular disease. Handbook Clin. Neurol. 138, 239–261 (2016).
Hu, X., De Silva, T. M., Chen, J. & Faraci, F. M. Cerebral vascular disease and neurovascular injury in ischemic stroke. Circ. Res. 120, 449–471 (2017).
Brown, R. D. Jr & Broderick, J. P. Unruptured intracranial aneurysms: epidemiology, natural history, management options and familial screening. Lancet Neurol. 13, 393–404 (2014).
Kisler, K., Nelson, A. R., Montagne, A. & Zlokovic, B. V. Cerebral blood flow regulation and neurovascular dysfunction in Alzheimer disease. Nat. Rev. Neurosci. 18, 419–434 (2017).
Sweeney, M. D., Kisler, K., Montagne, A., Toga, A. W. & Zlokovic, B. V. The role of brain vasculature in neurodegenerative disorders. Nat. Neurosci. 21, 1318–1331 (2018).
Franks, I. Gut microbes might promote intestinal angiogenesis. Nat. Rev. Gastroenterol. Hepatol. 10, 3 (2013).
Hanauer, S. B. Update on the etiology, pathogenesis and diagnosis of ulcerative colitis. Nat. Clin. Pract. Gastroenterol. Hepatol. 1, 26–31 (2004).
Torres, J., Mehandru, S., Colombel, J.-F. & Peyrin-Biroulet, L. Crohn’s disease. Lancet 389, 1741–1755 (2017).
Lopera, J. E. Embolization in trauma: principles and techniques. Semin. Intervent. Radiol. 27, 014–028 (2010).
Burke, C. T. & Mauro, M. A. Bronchial artery embolization. Semin. Intervent. Radiol. 21, 43–48 (2004).
Rilling, W. S. & Chen, G. W. Preoperative embolization. Semin. Intervent. Radiol. 21, 3–9 (2004).
Hong, G., Antaris, A. L. & Dai, H. Near-infrared fluorophores for biomedical imaging. Nat. Biomed. Eng. 1, 0010 (2017).
Wan, H. et al. A bright organic NIR-II nanofluorophore for three-dimensional imaging into biological tissues. Nat. Commun. 9, 1171 (2018).
Hong, G. et al. Through-skull fluorescence imaging of the brain in a new near-infrared window. Nat. Photon. 8, 723–730 (2014).
Martínez-Corral, I. et al. In vivo imaging of lymphatic vessels in development, wound healing, inflammation and tumor metastasis. Proc. Natl Acad. Sci. USA 109, 6223–6228 (2012).
Jia, Y. et al. Quantitative optical coherence tomography angiography of vascular abnormalities in the living human eye. Proc. Natl Acad. Sci. USA 112, E2395–E2402 (2015).
Zhang, W. et al. High-resolution, in vivo multimodal photoacoustic microscopy, optical coherence tomography and fluorescence microscopy imaging of rabbit retinal neovascularization. Light Sci. Appl. 7, 103 (2018).
Sakadžić, S. & Wang, L. V. High-resolution ultrasound-modulated optical tomography in biological tissues. Opt. Lett. 29, 2770–2772 (2004).
Yao, J., Maslov, K. I., Shi, Y., Taber, L. A. & Wang, L. V. In vivo photoacoustic imaging of transverse blood flow by using Doppler broadening of bandwidth. Opt. Lett. 35, 1419–1421 (2010).
Huang, C.-H. et al. High-resolution structural and functional assessments of cerebral microvasculature using 3D Gas ΔR2*-mMRA. PLoS ONE 8, e78186 (2013).
Wang, L. V. & Yao, J. A practical guide to photoacoustic tomography in the life sciences. Nat. Methods 13, 627–638 (2016).
Nasiriavanaki, M. et al. High-resolution photoacoustic tomography of resting-state functional connectivity in the mouse brain. Proc. Natl Acad. Sci. USA 111, 21–26 (2014).
Meijering, E. H., Niessen, W. J. & Viegever, M. Retrospective motion correction in digital subtraction angiography: a review. IEEE Trans. Med. Imaging 18, 2–21 (1999).
Jeans, W. The development and use of digital subtraction angiography. Br. J. Radiol. 63, 161–168 (1990).
Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).
Wang, Y. et al. Accurate quantification of astrocyte and neurotransmitter fluorescence dynamics for single-cell and population-level physiology. Nat. Neurosci. 22, 1936–1944 (2019).
Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).
Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131 (2018).
Cole, J. H. et al. Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. NeuroImage 163, 115–124 (2017).
Li, S., Deng, M., Lee, J., Sinha, A. & Barbastathis, G. Imaging through glass diffusers using densely connected convolutional networks. Optica 5, 803–813 (2018).
Li, Y., Xue, Y. & Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181–1190 (2018).
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019).
Lee, H. et al. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat. Biomed. Eng. 3, 173–182 (2019).
Mahapatra, D., Bozorgtabar, B., Hewavitharanage, S. & Garnavi, R. Image super resolution using generative adversarial networks and local saliency maps for retinal image analysis. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention 382–390 (Springer, 2017).
Son, J., Park, S. J. & Jung, K.-H. Retinal vessel segmentation in fundoscopic images with generative adversarial networks. Preprint at https://arxiv.org/abs/1706.09318 (2017).
Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision 2223–2232 (IEEE, 2017).
Murray, C. D. The physiological principle of minimum work: I. The vascular system and the cost of blood volume. Proc. Natl Acad. Sci. USA 12, 207–214 (1926).
West, G. B., Brown, J. H. & Enquist, B. J. A general model for the origin of allometric scaling laws in biology. Science 276, 122–126 (1997).
Ganin, Y. et al. Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17, 2096–2030 (2016).
Dey, N. et al. Richardson–Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution. Microsc. Res. Tech. 69, 260–266 (2006).
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
Lu, B., Chen, J.-C. & Chellappa, R. Unsupervised domain-specific deblurring via disentangled representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 10225–10234 (IEEE, 2019).
Dice, L. R. Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945).
Gegundez-Arias, M. E., Aquino, A., Bravo, J. M. & Marin, D. A function for quality evaluation of retinal vessel segmentations. IEEE Trans. Med. Imaging 31, 231–239 (2012).
Laroui, H. et al. Dextran sodium sulfate (DSS) induces colitis in mice by forming nano-lipocomplexes with medium-chain-length fatty acids in the colon. PLoS ONE 7, e32084 (2012).
Chassaing, B., Aitken, J. D., Malleshappa, M. & Vijay‐Kumar, M. Dextran sulfate sodium (DSS)‐induced colitis in mice. Curr. Protoc. Immunol. 104, 15–25 (2014).
van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
O’Donnell, T. F. Jr, Rasmussen, J. C. & Sevick-Muraca, E. M. New diagnostic modalities in the evaluation of lymphedema. J. Vasc. Surg, Venous Lymphat. Disord. 5, 261–273 (2017).
Greives, M. R., Aldrich, M. B., Sevick-Muraca, E. M. & Rasmussen, J. C. Near-infrared fluorescence lymphatic imaging of a toddler with congenital lymphedema. Pediatrics 139, e20154456 (2017).
Tzeng, E., Hoffman, J., Darrell, T. & Saenko, K. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision 4068–4076 (IEEE, 2015).
Fraz, M. M. et al. Blood vessel segmentation methodologies in retinal images—a survey. Comput. Methods Programs Biomed. 108, 407–433 (2012).
Staal, J., Abràmoff, M. D., Niemeijer, M., Viergever, M. A. & Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23, 501–509 (2004).
Watts, L. T., Zheng, W., Garling, R. J., Frohlich, V. C. & Lechleiter, J. D. Rose Bengal photothrombosis by confocal optical imaging in vivo: a model of single vessel stroke. J. Visualized Exp. 23, e52794 (2015).
We thank H. Fan for insightful suggestions regarding the optics experiments, X. Yuan for camera troubleshooting, J. Cai for fruitful discussions about data processing in the bowel vasculature section and H. Zhao for drawing illustrations. The work was supported by the National Natural Science Foundation of China (grants 61722209, 61860206003, 61971255 and 81872368), funds from the Shenzhen Science and Technology Innovation Committee (grants KQJSCX20180327143623167 and JCYJ20180508152130899) and a fund from Shenzhen Development and Reform Commission Subject Construction Project 143.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Figs. 1–14, Discussion and Table 1.
In vivo fluorescence imaging of cerebral vasculature of a healthy mouse at a 25 Hz frame acquisition rate. As ICG is injected, VasNet recovers the dynamic changes of vascular structure of a healthy cerebrum.
In vivo fluorescence imaging of cerebral vasculature of a mouse with thrombosis model at a 25 Hz frame acquisition rate. As ICG is injected, VasNet recovers the dynamic changes of vascular structure of a diseased cerebrum.
Augmenting diagnosis of bleeding from dynamic DSA vascular images acquired from human. VasNet outputs image with vasculature reconstructed. The bleeding features are augmented along with the dynamics of dilation.
About this article
Cite this article
Wang, Y., Ji, M., Jiang, S. et al. Augmenting vascular disease diagnosis by vasculature-aware unsupervised learning. Nat Mach Intell 2, 337–346 (2020). https://doi.org/10.1038/s42256-020-0188-z
This article is cited by
Nature Communications (2023)
Assessment of blood perfusion quality in laparoscopic colorectal surgery by means of Machine Learning
Scientific Reports (2022)
Nature Communications (2021)