Main

Human organs are infiltrated with blood vessels that supply oxygen and nutrients and remove metabolic byproducts from living tissue cells1. Vascular abnormalities, both structural and dynamic, have strong pathologic indications in a wide range of diseases. Common cerebral vascular diseases, including atherosclerosis, thrombosis and aneurysm, can cause stroke2,3,4,5 and many forms of neurological dysfunction and degeneration6,7. Inflammatory bowel diseases (IBDs) and bowel microbial infections are associated with enriched vasculature and microbleeds along the digestive tract8,9,10. Internal bleeding is one major factor leading to mortality after trauma11 and haemoptysis12. Hypervascular tumours13 feature enriched angiogenesis. Therefore, probing abnormal vasculature has significant impacts for a range of biomedical examinations.

Vascular imaging with reduced invasiveness requires zero or minimal incision of patients or experimental animals, but introduces some challenges to imaging-based diagnosis. Vascular networks are covered by biological structures such as skin and organ tissue, introducing substantial light scattering in optical investigations such as fluorescence imaging14,15,16, optical coherence tomography (OCT)17,18 and photoacoustic angiography19,20,21. Topologies are impacted by heavy and inhomogeneous noise that disrupts the recognition of pathovascular features. Also, when investigating internal bleeding using X-ray computed tomography (CT) and digital subtraction angiography (DSA)22,23,24,25,26, similar pathological features in solid organs, such as vessel bleeding, stricture, aneurysm and tortuous blood vessels, are difficult to distinguish due to the artefacts introduced by organ motion and vascular network complexity.

Deconvolution is a relatively common solution to extracting information from noise-polluted signals, and usually requires estimating or measuring the point spread function (PSF) of the system based on theoretical models. However, spatially inhomogeneous and individually different characteristics prevent non-blind deconvolution from being implementable in biomedical data. Learning-based techniques have recently been utilized to extract high-quality physiological and pathological features and perform recognition across a wide range of scales, including cells27,28, histology tissue slides29 and deep-tissue features acquired from OCT, CT and magnetic resonance imaging (MRI)30,31. Meanwhile, deep learning has proven to be versatile in extracting features from highly scattered patterns through synthetic diffusers32,33. Unfortunately, such learning-based descattering methods are supervised, and a large number of labelled and per-pixel registered data are necessary for training the neural networks. Moreover, a model trained on one location of the diffuser can hardly be extended to other regions due to the assumption of a spatial-invariant PSF for deconvolution. There is thus an essential conflict between limited data availability and the broad diversity of pathology features. On the other hand, the ‘black box’ operations of deep learning make decisions without explanations. This has accelerated advances in deep learning in medical diagnosis through establishing algorithms to ‘unbox’ learning and generate explainable outputs to render decision-making transparent34,35.

Several learning-based approaches have been reported that use gold-standard vascular images to extract explicit vasculature from original fundus images36,37. These models require large datasets of pixel-level aligned image pairs to train the generator network. The recent seminal unsupervised image-to-image method, ‘cycle-consistent generative adversarial networks’ (CycleGAN)38, has demonstrated powerful performance on transfer learning between two sets of images. These approaches treat the vascular information and degradation patterns equally and thus are not feasible for eliminating non-uniform noises induced by the heterogeneous diffusibility of skin and organ tissue, uncontrolled organ motion and non-uniform background illumination.

Here, we report VasNet, an unsupervised transfer learning technique specialized for vascular feature recognition. Vasculature in different organs evolves following a general physiological principle, Murray’s Law39,40, and shares an intrinsic commonality in appearance. Therefore, the widely accessible retinal vasculature in binary formats is chosen as the target domain of vascular-aware transfer learning, with the diffusive organ vasculature acquired from fluorescence or X-ray imaging as the source domain. This eliminates the dependence on large datasets of image registration and manual labelling of ground truths in supervised methods. The algorithm outputs explainable images with multi-dimensional information about blood flows, including the vascular structure, flow rates and the prescreened examination of suspicions. In this work, we have performed diagnosis augmentation of thrombosis and internal bleeding and proved the concept of establishing new diagnostics for ulcerative colitis in animal models.

Vascular diagnosis augmentation

The concept of augmenting vascular diagnosis operates by blind vasculature extraction from an unsupervised deep learning algorithm without using labelled data. It is, intrinsically, an image-to-image translation problem. The principle of diagnosis augmentation is outlined in Fig. 1. A domain-transferred image is consistently generated by the neural network from an input image containing vascular information and noise overlay, until the generated image becomes indistinguishable from the expected vasculature. To generate explicit vascular images using the unsupervised transfer learning technique, we chose a publicly available set of binary retinal vascular images as the target domain reference (Supplementary Fig. 1). The retinal images present high-contrast topologies of blood vessels against their surroundings, with enriched complexity, including vessel branching, size (diameter) variation, knots and endings.

Fig. 1: The augmentation principle of vascular disease diagnosis.
figure 1

VasNet learns the image-to-image mapping between two unpaired image domains: raw vascular observations corrupted by scattering, aberrations or non-uniform noises and the segmentation of retinal vascular images. It extracts the vascular topology, colour-codes the blood flow dynamics and unveils the spatiotemporal illumination of regions of interest, examines the pathological features and presents suspicions in contrasting colours, and discovers new diagnostic features and suggests the probability of a disease occurrence. Retinal vascular images reproduced from ref. 54, IEEE.

Full size image

VasNet utilizes the cycle consistency loss and learns the mapping between two image domains in the absence of labelled blood vessels (Fig. 2). A domain adversarial neural network (DANN)41 loss function is embedded in the algorithm to create bias in feature selection to improve vasculature extraction from inhomogeneous backgrounds. We feed the segmented retinal vasculature into VasNet as the target image domain (A domain) and the raw images acquired from fluorescence or X-ray imaging and overlaid with heterogeneous noises as the source image domain (B domain). The problem of blind vasculature extraction is then translated as disentanglement of the vascular structure zB from the degraded observations (B domain).

Fig. 2: VasNet architecture.
figure 2

a, VasNet solves the heterogeneous descattering problem by extracting the mutual embeddings of structural statistics from the vasculature (domain A) and the scattered observations (domain B). Note that domains A and B do not necessarily come from the same imaging modality. In our work, the retinal vasculature is chosen as the A domain. The proposed training strategy with a DANN loss function directly penalizes dissimilar statistics between the mutual features, that is zA (orange) and zB (green). For inference, only the part in the blue dashed rectangle needs to perform. b, The encoders (trapezoids) and the disentangled embeddings (coloured blocks). Retinal vascular images reproduced from ref. 54, IEEE.

Full size image

VasNet eliminates the requirement for large datasets of labelled images as ground truths to train neural networks. Instead of presenting an outcome with a binary decision to overlap or validate the human diagnosis, our strategy aims to augment human diagnosis by presenting the end-users with explainable images with enriched information. VasNet is vasculature-aware and delivers the results of ‘Structure + X’. X refers to multi-dimensional vascular characteristics such as blood flow, the examined blood dilation and its suspicious counterparts, and disease-induced subtle morphological changes, which might open a route towards alternative, yet more effective, disease diagnostic measures by examining new structure emergence (Fig. 1). In the following studies, we implemented VasNet to translate images of cerebral, bowel and internal solid organ vessels, overlaid with heterogeneous noise, into retinal-like topologies with explainable characteristics.

Unsupervised vasculature extraction

To recognize vascular features through intact diffusers such as the skin and solid organs that enclose blood vessels, we eliminated imposed noise by modelling the non-uniform descattering as an unsupervised mutual information disentanglement problem. Successful descattering was attributed to the adaptation and modification of the unsupervised domain-transfer network and the selection of a standard dataset of the retinal vascular network as reference for imaging generation (Supplementary Fig. 1). Determined by the binary retinal vascular network as the training reference, noise-corrupted images were disentangled into vascular-like features (defined as reference-relevant features) and noise (defined as reference-irrelevant features). VasNet discriminated the two types of content after training. Therefore, the algorithm testing performed unbalanced selectivity on the vascular-relevant and -irrelevant features.

In this study, as we were targeting vasculature reconstruction, the algorithm picked vascular-like features (such as regional features with high aspect ratios) and tended to disregard non-vascular features in the reconstruction. This resulted in significant differences in the output images on using VasNet compared with traditional schemes such as deconvolution42, the embedded vessel analysis tool in Fiji43 and learning-based methods, including CycleGAN38 and UDSD44 (Fig. 3). The performance of different algorithms was compared quantitatively and evaluated by the dice similarity coefficient (F1 score)45 and connectivity–area–length (CAL)46 metrics, in reference to the ground truths generated by human expert segmentation (Supplementary Fig. 2 and Supplementary Table 1). The visual presentation of CycleGAN outputs shows its weakness, that is, the reconstructed ‘vessels’ being discontinuous. This might be attributed to its lack of vascular-awareness in feature extraction. Similarly, CycleGAN tends to over-interpret the shadows in the images in Fig. 3d,e into ‘vessels’ featuring irregular dashes. The intrinsic awareness of vascular feature extraction in VasNet endows it with enhanced performance on images acquired across modalities, organ types and scales.

Fig. 3: Performance evaluation of VasNet and other existing techniques.
figure 3

a–e, Performance evaluation on fluorescence images from mice (a,d,e) and DSA images from humans (b,c), following the imaging modalities in Fig. 1. Input vascular images were reconstructed by traditional schemes including deconvolution, the embedded vessel analysis tool in Fiji and learning-based methods including CycleGAN38, UDSD44 and VasNet. The ground truths were constructed by human expert segmentation on input images. Scale bars, 1 mm (a), 2 cm (b,c) and 5 mm (d,e).

Full size image

A major challenge in imaging-based vascular diagnosis is the loss of structural precision, especially the transverse dimensions of blood flows that indicate vascular abnormalities such as blood vessel stricture, dilation and breakage. The VasNet algorithm induces biased structural pattern reconstruction and recapitulates the topology of the vessels hidden in heterogeneous and complex noise. The versatility of VasNet with different imaging techniques and variations in vascular structure is demonstrated in Fig. 3 and Table 1. The reference retinal vascular image had enriched variation in vessel topology and size, but the distribution of vessel dimensions in the testing images was distinguishable from the reference images (Supplementary Fig. 3).

Table 1 Comparison of the F1 score and CAL metric
Full size table

Augmenting the diagnosis of mouse cerebral vascular disease

Indocyanine green (ICG) is an FDA-approved dye molecule for clinical use that emits in the near-infrared (NIR) window (peak emission at ~820 nm). Specific binding of ICG to lipoproteins in the bloodstream prolongs the circulation lifetime of ICG in blood14. Light scattering in skin weakens with increased wavelength, but the images acquired in our experiments in the NIR-I region (800–900 nm) for feature recognition were highly disrupted with zero or minimal tissue incisions. The unlabelled diffused images were then input into the transfer learning network and translated into explainable images that showed both precise vascular structures and spatial flow dynamics in the mouse cerebrum.

The cerebral vasculature is covered by a translucent skull and intact scalp skin (~0.6 mm thick). It has a stem vein along the superior longitudinal sinus, with branched vessels distributed on the two sides. A stroke occurs when the blood supply is occluded. Thrombosis is a common type of occlusion resulting from blood clot formation inside vessels. In this study, fluorescence image sequences of mice cerebral vasculature were recorded through the intact scalp skin and the skull after intravenous injection of ICG. Both healthy mice and mice with thrombosis were imaged following the same procedures (see Methods for details).

The acquired image sequences contained both spatial- and temporal-dependent illumination of ICG circulating in the blood, from which blood flow rates could be derived. Owing to light scattering through the scalp and skull, only diffused vascular topology of healthy and diseased mice could be acquired (Fig. 4a,c). We applied VasNet to eliminate irrelevant information from the vasculature, including noise and non-vascular topologies. The algorithm-generated images exhibited high-contrast vascular features (Fig. 4b,d and Supplementary Videos 1 and 2) and were validated by the patterns obtained after removing the scalp (Supplementary Fig. 9). The global cerebral vasculature was constructed from the illumination sequence.

Fig. 4: Augmenting the diagnosis of mouse cerebral vascular disease.
figure 4

Fluorescence imaging and interpretation of the mice cerebral vasculature using VasNet to model the diagnosis of human thrombosis. The vascular network was perfused with ICG and imaged in the NIR window (excitation at 785 nm and emission at 810–890 nm). a–d, Acquired images of ICG illuminated vessels (a,c) and the VasNet output images (b,d) in the cerebrum of a healthy mouse (a,b) and a mouse with thrombosis (c,d). e–h, Derived blood flow rates from time-dependent illumination of blood vessels in a healthy cerebrum (e,f) and a diseased cerebrum (g,h). The time-dependent intensity variation at multiple locations, indicated with coloured arrows in e and g, is plotted in f and h. Scale bars in all panels, 1 mm.

Full size image

The blood flow rates throughout the network were derived from the illumination sequence and encoded in colours, based on the principle of the continuum of mass for incompressible fluids, that is, the blood (Fig. 4e). The colour coding shows drops in regional flow rates resulting from blood occlusion in the mice with thrombosis when compared with their healthy counterparts. The flow rates and their variations in different parts of the vascular network were found to be consistent in at least three healthy and three diseased mice (Fig. 4e–h and Supplementary Figs. 10 and 11). This proved that our algorithm was not only reliable in probing the structural abnormalities, but also the change in flow rates with high spatial signatures in the cerebrum. The structural-dynamic dual check augmented both the diagnosis efficiency and accuracy in non-invasive examinations.

Augmenting the diagnosis of human internal vascular disease

DSA is among the most important examination methods in the diagnosis and treatment of internal bleeding and hypervascular tumours via intervention embolization25,26. However, subject motion, respiration and the skeleton create severe artefacts and impose noise onto the vascular features, which may result in delayed diagnosis or a high rate of misdiagnosis. For example, bleeding, tortuous vessels and aneurysm are all characterized by dilation of the radiocontrast agents, and the overlaid noise can lead to the pathological vascular features and their normal counterparts being indistinguishable.

Although dilation of the radiocontrast agent (iodixanol) was observed in the DSA images (Fig. 5a,c), the artefacts caused by respiration-induced organ motion reduced the recognition of suspicious features or features hidden within the complex vascular background. Furthermore, the intervention embolization required a high-rate, or nearly real-time, diagnosis of diseased features with precision spatial resolution, as it was performed under the real-time guidance of DSA imaging.

Fig. 5: Augmenting the diagnosis of bleeding from dynamic DSA data.
figure 5

VasNet interpretation of time-lapse clinical DSA vascular images acquired from the human abdomen. a, A DSA image of the internal vasculature. Images were acquired with a radiocontrast agent (iodixanol) to highlight the blood circulation. b, VasNet output image with the vasculature reconstructed and the bleeding features augmented. c,d, Magnified time-lapse view of the regions of interest (ROIs) outlined in a and b, respectively. The dashed (lines 1 and 1′) and solid (lines 2 and 2′) lines in the image sequence indicate the dilation. e, The dilation trajectory of iodixanol at the bleeding site, derived from the VasNet output sequences in d. The colour coding shows the dynamics of the dilation. f,g, Time-dependent intensities of iodixanol along the dashed and solid lines in c and d, respectively. h,i, Quantification of the iodixanol intensities along the dashed and solid lines in c and d at time points of 1 s (h) and 2 s (i). Scale bars, 2 cm (a), 1 cm (c).

Full size image

The limitation was overcome by generating explainable images from the VasNet algorithm (Fig. 5b,d,e). The global vascular network was reconstructed with significantly improved accuracy and contrast against the background noise. The ROIs framed in black were interpreted with colour coding to exhibit the dilation in the recording. Dilation recognition in the raw input and output images was compared (Fig. 5f–i and Supplementary Video 3). The intensity contrast of the circulating agent in the dilation track, which contributed positively to diagnosis ability, was less distinctive in the DSA raw images than in the learning output images. The feature augmentation was performed in 40 groups of DSA image sequences, each acquired from an individual subject. Three examples are shown in Fig. 5 and Supplementary Fig. 12.

VasNet interpretation also augmented diagnosis from single static images. VasNet enhanced the vascular structure against the non-uniform degradation and succeeded in delivering outputs with ‘Structure + X’ in the DSA modality, where X helped to distinguish the blood dilation. To evaluate the effectiveness of augmenting the bleeding diagnosis quantitatively, we trained a basic classifier to determine whether the bleeding occurred or not, tested on the augmented vasculature of the suspicious patches. More specifically, data from 40 subjects were collected, half of which were adopted for training and the rest for testing.

The correctly classified positive (red) and negative (blue) samples are depicted in Fig. 6a,b. By interpreting the bleeding recognition results, we conclude that the classifier prefers to group together the augmented samples with narrow-neck connected expansion as positive predictions. For the case shown in Fig. 6c, which counted the bleeding and non-bleeding features of 20 patients, the bleeding diagnosis performance by the neural network, quantified by the area under the receiver operating characteristic (ROC) curve (AUC), reached 97%. The classification accuracy reached 88% when the cutoff probability was 0.4.

Fig. 6: Augmenting the diagnosis of bleeding from static DSA data in the human abdomen.
figure 6

a,b, Distinctive feature recognition to distinguish a tortuous vessel (blue, a), a branched vessel (blue, b) and bleeding (red in a and b). The red vessels featured thin necks and expanded domains. c, Counts of bleeding (red) and non-bleeding (blue) suspicions in DSA images (20 patient samples in total), with each count showing the bleeding probability. d, The ROC curve of the bleeding counts. Note that the bleeding regions were marked by clinicians in the Chinese PLA General Hospital and used as the ground truth for accuracy evaluation. Scale bars, 2 cm (a,b).

Full size image

Discovering a new diagnosis of mouse bowel vascular disease

IBDs (predominantly ulcerative colitis (UC) and Crohn’s disease) involve mucosal inflammation and feature an increased density of abnormal vessels, such as strictures, ulcers, bleeding along the tract, as well as mucosal angiogenesis9,10. Mouse colitis models were established by watering BALB/c mice with dextran sulfate sodium (DSS, 5.0 wt% in drinking water). DSS is toxic to the colonic epithelia and induces erosion that ultimately compromises barrier integrity, resulting in increased colonic epithelial permeability47. Here, disease induction occurred within three to seven days following DSS administration and appeared as severe colonic bleeding. It mimicked the superficial inflammation seen in UC, which features enriched angiogenesis in the mucosa, but with nothing reported on the bowel outer surface48.

The bowel vasculature was covered by abdominal skin (~0.3 mm thick), which scattered emission from the circulating ICG in the vessels along the outer surface of the tract. Fluorescence image sequences of the mice bowel vasculature were recorded through the intact abdominal skin (see Methods for details). Both healthy (day 0) and DSS-administrated mice were imaged for control. The ground truths were accessed by cutting off the abdominal skin and the peritoneum and presenting the vascular features to the camera (Fig. 7a,b).

Fig. 7: Discovering a new diagnosis of mouse bowel vascular disease.
figure 7

Fluorescence imaging and interpretation of the mice bowel vasculature to model colitis diagnosis. a, The steps of diagnostic testing, including creating the colitis models by watering mice with DSS, non-invasive imaging of the bowel vasculature, and access to ground truths of the vasculature by sectioning off the abdominal skins. b, Bright-field image of the bowel vasculature of an experimental mouse. c–f, Acquired fluorescence images (c,e) and interpreted images (d,f) of the bowel vasculature of healthy mice (c,d) and mice with colitis (e,f). g, Multi-style interpretation and counts of the bowel vasculature, including the area percentage of large vessels, the mean branch length and the mean width, imaged at day 0 (D0), D2, D4 and D6 after DSS administration. h, Scatter diagram of blood vessels acquired from 22 mice based on quantitative evaluation of the area percentage of large vessels, the mean branch length and the mean width of the vessels. We used the t-distributed stochastic neighbour embedding (t-SNE) algorithm to embed high-dimensional parameters in two-dimensional (2D) space for visualization. Scale bars, 5 mm.

Full size image

The explicit vascular topology extracted from the blurry raw images (Fig. 7c–f) was interpreted with various diversification methods, including the large vessel fraction and the vessel length and width (Fig. 7g,h). Figure 7g shows the statistics of vessel diversification of mice at days 0, 2, 4 and 6 after being administrated with DSS, by measuring the vessel connectivity, the mean branch length and the mean width of the reconstructed vessels. The connectivity was quantified by evaluating the fraction of connected vessels higher than a certain value in area (150 pixels in this case), indexed as ‘area percentage of large vessels’. The percentage was counted as the coverage fraction of large vessels over all vessels in each image. The mean branch lengths and widths were quantified by Fiji43. The quantification is presented in the corresponding boxplots in Fig. 7g. For better visualization of the multiple indices, we performed the t-SNE algorithm49 to present the diversification in two dimensions. The statistics show the distinct distribution of mice groups in different health conditions, as suggested by their disease progression, indexed by the number of days after DSS administration (Fig. 7h). However, the distinction was not always the same, as the separation by counting the area fraction of large vessels was the most significant, but least significant in the width statistics. The outlier at day 2 in the connectivity plot, indicated by a black arrow, was suggested to be heavily diseased. This was verified by skinless observation of the bowel condition (Supplementary Fig. 13). This study extends the promise of our unsupervised transfer learning technique in discovering new diagnostic characteristics.

Discussion

In this work, we have reported an alternative implementation of deep learning techniques in augmenting biomedical diagnosis. We have established VasNet, an unsupervised transfer learning network, to overcome the challenges set by the limited data accessibility in biomedical imaging, the broad diversity of pathological features and the labour-intensive image registration and ground-truth labelling for training a neural network. We have embedded a multi-scale domain adversarial training scheme in the algorithm that simplifies disentanglement of the vascular structure and the modality-specific noise representation. This proved versatile for the interpretation of images acquired from different modalities and carrying structures with varied dimensions, size distributions and heterogeneous diffusion levels. Rather than providing a binary decision-making outcome, VasNet unboxes the feasibility of learning on disease diagnosis, and generates explainable images with multi-dimensional information, including vascular topology, flow rates, prescreened suspicions and statistics of blood vessel features. With this, we anticipate that the acceptance of AI decision-making might be boosted. Doctors can either accept a binary decision or make a decision based on the enriched information output from the neural network.

We have also proposed the idea of using deep learning techniques to discover new diagnostic characteristics. Unknown characteristics are usually hidden under heterogeneous noise, as in the vascular patterns shown in Fig. 7c,e. Although the distinction between the fluorescence images from healthy mice (Fig. 7c) and mice with colitis (Fig. 7e) is observable, it requires clear and amplified differences in pattern evolution with disease progress to distinguish the features in healthy, pseudo-healthy and diseased conditions. Our algorithm-dependent examination amplified this distinction and thus validated the diagnosis of colitis by the appearance of the vasculature on the exterior surfaces of the bowel. The colitis progresses with angiogenesis, which initiates at the early stage of disease occurrence.

We note that VasNet has limitations in extracting vascular features from sources with extremely low signal-to-noise ratios. It may interpret noises into hallucinations, an example of which is shown in Supplementary Fig. 14. However, because end-users tend to focus on areas of interest and play down fringe areas, the chances of hallucinations resulting in misdiagnosis are expected to be low.

The technology could be extended to the diagnosis of other types of disease, depending on feature classification and diversification. For example, lymphatic disorders, such as lymphoedema (LED), are examined by lymphoscintigraphy (LSG), but this technique is limited in its ability to identify pathology and guide therapy. Our VasNet-based vessel imaging augmentation might be able to assist in the rapid diagnosis of LED and also guide therapeutic intervention50,51.

Our diagnosis augmentation strategy reduces the dependence of imaging-based diagnosis on high-end equipment, and takes over some of the load on doctors by reducing the dependence on personal experience and repetitive labour-intensive practice and confirmation. With the advent of new diagnostic technologies, such as portable, easily affordable and domestic diagnostic instruments, professional users such as doctors can make faster and more accurate decisions about patients, while non-professional users, such as patients, medical interns and general practitioners, can perform pseudo-professional diagnoses.

Methods

Learning algorithm

To perform the training and testing of biomedical data with limited volume and enriched diversity, we innovated the unsupervised neural network to emphasize feature reconstruction for vascular disease diagnosis (VasNet). Its success is attributed to the adaptation and modification of the domain-transfer network and selection of the standard dataset of a retinal vascular network as the reference for imaging generation. The incorporation of a multi-scale fusion strategy with a DANN loss function creates a biased reconstruction on vascular-relevant patterns. Consecutively, VasNet improves vasculature extraction and reduces misinterpretation of heterogeneous noise.

  1. 1.

    Network structure. As shown in Fig. 2, VasNet decouples the structural information zA and zB from the target domain (A) and source domain (B) and the irrelevant content zBn. These embeddings are the outputs of encoders EA2z, EB2z and EB2zn, respectively. The target domain (A) is expected to contain the vascular structure of interest. Besides the vasculature content, the source domain (B) suffers from various types of degradation, such as non-uniform background and non-uniform degree of scattering, decoupled by the decoder EB2zn. To effectively disentangle the vasculature content, that is, the small gap between the statistics of the mutual structural features zA and zB, we propose to penalize the distribution shift between zA and zB by the domain adversarial loss proposed in the DANN41. Because the mutual information between zA and zB encodes the vasculature properties, the domain adversarial training scheme ensures that the inference network (in the blue dashed rectangle in Fig. 2) tends to disregard the vascular-irrelevant features, such as non-uniform background and scattering noise.

  2. 2.

    Biased feature reconstruction by DANN. Inspired by UDSD44, which achieved state-of-the-art domain-specific deblurring results based on disentangled representations, VasNet pays special attention to the content differences between the source domain and the target domain. Although UDSD tends to decompose the shared content from the input domains, the gradient signal of the structural encoders may vanish through a long path propagation through a stack of encoders and decoders, that is, from the rightmost outputs in Fig. 2a to the leftmost structural embeddings, which causes UDSD to fail to decompose the shared content. In contrast, VasNet extracts the shared content from both A and B domains by introducing the DANN loss to explicitly encourage the emergence of the statistically mutual features by penalizing the domain gap between embeddings zA and zB. The DANN loss function generates mutual features that are indiscriminable and intractable from their parental domain. In this way, VasNet effectively avoids the information loss due to the explicit guidance for structure extraction and reduces the structural domain shift between the source and target domains. The oscillation in the training process often existed because the DANN loss may lead to a small gap between two unfixed feature distributions, as mentioned by Tzeng and others52. To deal with this, we defined the domain adversarial loss using a cross-entropy loss function against random guess probability. Such domain adversarial loss encourages the emergence of indiscriminative structural information from both source and target domains.

  3. 3.

    Training details. Considering the existence of various widths of blood vessels, we further utilized a coarse-to-fine scheme by fusing the predictions of the input pyramid with different scales for both wide and narrow vessels. Training the algorithm with multiple vascular networks of different scales for each input was found to significantly slow down the training process and require extraordinarily large data volume. In contrast, by taking advantage of the fact that various scales of the vasculature share the similar topology and width distribution, our multi-scale fusion strategy led to considerably complete and accurate vasculature reconstruction. The proposed VasNet generated a 512 × 512 output within 0.1 s, working on a TITAN Xp GPU.

Data preparation

The publicly available retinal vasculature dataset CHASE_DB153 and DRIVE54 with manual segmentation were selected as the target domain images. Because the vascular networks of different scales shared similar topology, the algorithm only required inputting small patches (128 × 128) cropped from the raw large images. Similarly, the source domain images, which were from different modalities and had various degrees of scattering, were also cropped into small patches. The criterion used to determine the patch scale was to make the histograms of average vascular branch lengths from the two domains resemble each other.

The preparation of the source domain images is presented in the following. All animal studies and use of clinical data were conducted under the ethics regulations of Tsinghua University and the PLA General Hospital, China.

  1. 1.

    For the fluorescence imaging set-up, in both imaging experiments through the scalp and abdomen, the excitation light was provided by a 785 nm laser source (Changchun New Industries) coupled to an 8-mm collimator. The light was expanded and adjusted by an iris to illuminate the entire mouse head or abdomen area. The excitation power density at the imaging plane was ~29 mW cm−2 in the experiments. The emitted fluorescence was filtered by an 810–890 nm bandpass filter (Thorlabs, FBH850-40) and captured by a CMOS camera (Blackfly, BFS-U3-120S4M-CS) through a lens with a focal length of 25 mm. The camera was set to expose continuously at a frame rate of 25 frames per second to the fluorescence illumination process.

  2. 2.

    All animal experiments were performed under the National Institutes of Health Guide for the Care and Use of Laboratory Animals and approved by the Institutional Animal Care and Use Committee of Tsinghua University. BALB/c mice were purchased from Guangdong Medical Laboratory Animal Centre. Adult BALB/c mice (male, 4–6 weeks old) were housed at 22 ± 2 °C with a 50:50 light–dark cycle and were used with randomization. Before imaging, the hair on the heads and abdomens of the mice was removed using human depilatory cream.

  3. 3.

    In the cerebral thrombosis mouse model, BALB/c mice were anaesthetized by isoflurane and the hair over the scalp was removed. Each mouse was intravenously injected immediately with Rose Bengal at a dosage of 10 mg per kg body weight. A mouse model of thrombosis was created by intravenously injecting Rose Bengal and exposing a region of the cerebrum to 532-nm laser light (0.8 W) for 4 min through the intact scalp skin and the skull55 (Supplementary Fig. 8a). The radius of the laser spot was ~2.5 mm. Rose Bengal irradiated with green excitation light generated the production of reactive oxygen species, which subsequently activated a cascade of coagulation. The induction of the coagulation cascade produced an ischaemic lesion that was pathologically relevant to clinical stroke55. After one day, the mice were used for cerebral vasculature imaging.

  4. 4.

    For the UC mouse model, BALB/c mice were randomly assigned to the experimental or control groups (n = 5–6). For the experimental group, 5.0 wt% DSS in double-distilled water (DDW) was provided as drinking water for 2, 4 or 6 days. For the control group, DDW was always used as the drinking water. The treated and untreated mice were imaged on the abdomen on the same day.

  5. 5.

    For vasculature imaging, a BALB/c mouse with the hair removed from the head or abdomen was anaesthetized by isoflurane and fixed under the camera and lens. The area of interest (head or abdomen) was exposed under a 785-nm laser source (Supplementary Fig. 8b), then the mouse was intravenously injected immediately with ICG at a dosage of 8 mg per kg body weight. The ICG was dissolved in phosphate-buffered saline (PBS). The ICG molecules transported rapidly with the circulation, and the vasculature became fluorescent almost instantaneously after injection. Camera acquisition of the fluorescence images started immediately before the intravenous injection to record the illumination process.

  6. 6.

    To prepare the DSA imaging dataset, we collected an angiography dataset from 40 patients from the Chinese PLA General Hospital, Beijing. For each patient, during the dynamic process, between three and six images containing the vascular information were annotated with the real bleeding point region (accurate label) and two or three normal yet suspicious regions (false labels). To perform the bleeding detection task, we used DSA images of 20 patients for training and images of 20 patients for testing.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this Article.