Background & Summary

Blood vessel segmentation is often a necessary prerequisite for extracting meaningful analyses from biomedical imaging data. By creating a segmentation, or a mask, that separates vascular from non-vascular pixels, structural information about the vascular system can be acquired, such as diameter, branch order, and blood vessel type. Identification of blood vessels as arterioles, venules, or capillaries can be used to analyze vascular dynamics, such as blood flow and vascular supply. Blood vessel segmentation has clear clinical value. For example, in ischemic stroke studies, vascular segmentation enables detection and quantification of vascular occlusions, which can be helpful in determining therapeutic options1,2. Structural characteristics can also be used as predictors or markers to assist in the diagnosis of diseases, such as Alzheimer’s disease3,4, traumatic brain injury5, brain tumours6, atherosclerosis7, and retinal pathology8,9.

Apart from vascular analyses, blood vessel segmentation is also a necessary preprocessing step for the analysis of cells and pathological entities (Fig. 1). In addition to the endothelial and mural cells that make up the blood vessel proper, various other cell types interact with vascular walls, including astrocyte endfeet processes, perivascular macrophages, and peripheral leukocytes. Such cells and their interactions with vasculature can be identified and analyzed based on distance metrics to vascular walls, a task which is simplified with accurate vascular segmentation masks. Vascular-cellular interactions have been of particular interest in studies focused on diseases. For example, recruitment of peripheral leukocytes to cerebrovasculature has been observed following traumatic brain injury10, middle cerebral artery occlusion11, and in Alzheimer’s disease12. Similar distance metrics can be used to analyze pathological entities, such as perivascular Aβ plaques13 and atherosclerotic plaques14. Thus, segmentation of blood vessels is a necessary preprocessing step that facilitates further vascular and cellular analyses.

Fig. 1
figure 1

General workflow for 2PFM image processing. (a) A cranial window was created on the parietal bone, enabling in vivo 2PFM imaging. Superficial blood vessels can be observed through the cranial window (left) and ocular lens of the microscope (right). (b) Raw image volumes of rodent cerebrovasculature were collected and saved in Olympus image formats (*.oib, *.oir). (c) Segmentations of blood vessels were achieved using a U-Net and manual corrections. In the MiniVess dataset, raw and segmented image volumes are shared as NIfTI files (*.nii.gz). Segmentation of blood vessels enables downstream analyses, such as (d) cell tracking (arrow), (e) vasoconstriction and dilation (arrows), and (f) analysis of pathological entities, such as Aβ plaques (cyan). Blood vessels are shown in magenta. Scale bars = 100 μm in (b,c) and 50 μm in (d,e,f).

Full size image

In the neurosciences, two-photon fluorescence microscopy (2PFM) is currently the technique of choice for intravital microscopy. While the resolution of 2PFM can be on par with confocal microscopy, the risk of phototoxicity and photobleaching of tissues and fluorophores is substantially reduced because the excitation volume is limited to the focal volume of the microscope15. The use of longer wavelengths also results in less scattering by the neural tissue, allowing imaging at deeper depths within the brain. 2PFM has been extensively used to investigate various phenomena, including neural activity using voltage-sensitive dyes16and calcium indicators17, microglial activity using transgenic animal models18, and vascular dynamics19.

Most methods of automated image processing of 2PFM images rely on proprietary software, such as Imaris (Bitplane, United Kingdom) and Volocity (Quorum Technologies, Canada). Each analysis type, such as vascular segmentation and cell tracking, is generally sold as separate modules. Comprehensive analyses of datasets are therefore functionally limited by the modules available and can become prohibitively expensive. Furthermore, while automated modules produce impressive results for images with high SNR, biomedical images, particularly intravital 2PFM images, are inherently noisy. In practice, substantial manual modifications are required. An open-source alternative is FIJI (Fiji Is Just ImageJ)20. However, FIJI plugins often lack extensive documentation, resulting in a ‘black-box’ nature that may deter and limit use.

Deep learning, such as convolutional neural networks (CNNs) and recent transformer-based model architectures21,22, have been extensively used for automated segmentation tasks in biomedical imaging. For example, the U-Net, a fully CNN, achieves impressive performance in segmenting densely packed neurons in electron microscopy images23. Clinically, CNNs have achieved state-of-the-art performances in segmenting brain vasculature in magnetic resonance angiography24 and retinal vasculature in optical coherence tomography angiography25 datasets, which have been used to assist in the identification of pathological features.

A common challenge in the application of deep learning models to the biomedical imaging field is the generalizability of models. Models are often exclusively trained on datasets that were collected from a single site. Such models often fail to perform when evaluated on datasets collected at different sites due to a so-called ‘domain shift’ (see e.g. Ouyang et al. 202126 for an example in medical image segmentation), caused by differences in tissue preparation, scanner or microscope set-up, and/or inter-user variability in defining labels27,28. The problem is compounded by poor reporting of the number of evaluation sites and samples used29. One way to improve the reliability and transparency of ML models is to use diverse samples during training, and independent data cohorts for testing30. However, the availability of such annotated, publicly available biomedical imaging datasets is limited due to ethical and privacy concerns, particularly in clinical studies. Another strategy is to use synthetic datasets or publicly available non-biomedical datasets (e.g. ImageNet) as part of the training process, and then evaluate the trained model on the real dataset, a process known as ‘transfer learning’31,32. For example, using transfer learning, a CNN that was pre-trained on a synthetic dataset of blood vessels resulted in impressive segmentation of real mouse brain vasculature33. However, the availability of real, annotated, field-specific datasets remains to be a need for evaluating the generalizability of models in the biomedical imaging field. In addition, there has been a recent shift in focus from adjusting model parameters to achieve better performance metrics (‘model-centric’), to improving the quality of datasets to improve performance metrics (‘data-centric’), highlighting the importance of high-quality, publicly available datasets.

Public microscopy datasets have been curated by various research groups world-wide. For example, the Human Protein Atlas shows the distribution and expression of proteins and genes across major organ systems34,35,36, the Broad Bioimage Benchmark Collection contains annotated cell datasets37, and the Allen Brain Cell Types Atlas offers electrophysiological, morphological, and transcriptomic data measured from human and mouse brain. However, vascular datasets have not been as extensively documented. The availability of an annotated 2PFM vascular dataset would assist in diversifying the samples used for training a segmentation model, or in evaluating the performance of segmentation models that were trained on other datasets.

We hereby present MiniVess38, an expert-annotated dataset of 70 3D 2PFM image volumes of rodent cerebrovasculature. The dataset can be used for training segmentation networks39,40, fine-tuning pre-trained networks31,32,41, and as an external validation set for assessing a model’s generalizability42. The 3D volumes in this dataset have been curated to only contain clean XYZ imaging in order to ensure correct and consistent annotations, or segmentations, which has been observed to be integral to the evaluation of machine learning models43. Code for image preprocessing and the U-Net workflow are also provided in the MiniVess project Github page. The U-Net code was written using MONAI, a PyTorch-based framework that was built to encourage best practices for AI development in healthcare research. We hope that the availability of the image volumes and code will assist in evaluating the reliability of models built for the analysis of biomedical images.

Methods

Animal preparation

This dataset consists of 2PFM images of the cortical vasculature in adult male and female mice from the C57BL/6 and CD1 strains (20–30 g), and EGFP Wistar rats (Wistar-TgN(CAG-GFP)184ys) (310–630 g)44. All animal procedures were approved and conducted in compliance with the Animal Care Committee guidelines at Sunnybrook Research Institute, Canada.

To allow optical access to the brain, an acute cranial window was created over the parietal bone (Fig. 1). Detailed protocols on cranial window procedures have been published elsewhere45. Briefly, animals were anesthetized using 1.5–2% isoflurane in a mix of medical air and oxygen. Following fur and scalp removal, a 3–4 mm circle (mice) or rectangle (rats) of bone was removed from the parietal bone using a dental drill, and replaced with a glass cover slip. Due to the thickness of the skull in rats, 1% agarose was deposited onto the brain to prevent air bubbles beneath the cover slip. Animal physiology was monitored using a pulse oximeter, and temperature was maintained using a heating pad with a rectal thermistor. To visualize vasculature, Texas Red 70 kDa dextran (dissolved in PBS, 5 mg/kg; Invitrogen, Canada) was injected through a tail vein catheter. Animals were sacrificed under deep anesthesia using cervical dislocation (mice) or euthanol injection (rats) following the end of imaging.

Imaging

Imaging was conducted using a FV1000MPE multiphoton laser scanning microscope (Olympus Corp., Japan) with an InSight DS tunable laser (Spectra-Physics, USA), or a Ti:Sa laser (MaiTai, Spectra-Physics, Germany). A 25× water-immersion objective lens (XLPN25XWMP2, NA 1.05, WD 2 mm, Olympus Corp., Japan) was used to collect 512 × 512 images with a lateral resolution of 0.621–0.994 μm/pixel, an imaging speed of 2–8 μs/pixel, and a step-size of 1–10 μm, for a maximum depth of 700 μm. Excitation wavelengths of 810 or 900 nm were used. Fluorescent emissions were collected with photo-multiplier tubes preceded by a 575–645 nm bandpass filter. Images were saved in Olympus’s 12-bit .oib or .oir file formats. Acquisition settings were set to utilize the full 12-bit dynamic range (intensity values of 0–4095). Image details are listed in Supplementary Table 1.

File conversions

Image volumes were converted to the NIfTI (.nii) file format to make segmentation model protoyping faster, as it is commonly used in neuroimaging and ML frameworks, such as the Medical Open Network for Artificial Intelligence (MONAI, https://monai.io/). In MONAI, users can create dataloaders that are customized for their data formats by using Python libraries [such as tifffile (available in PyPI), python-bioformats (available in PyPI), and pyometiff (https://github.com/filippocastelli/pyometiff). In the future, we plan to develop a dataloader to allow direct use of microscopy formats, skipping the NIfTI conversion. Here, we provide the code to convert Olympus files (.oib and .oir) to NifTI (.nii) format, with metadata encoded in the NifTI1 header format. NifTI files were further compressed as .gz archive files (.nii.gz). The original Olympus files are 12-bit, and the exported NifTI files are saved as 16-bit images, as a 12-bit data type is not available. The code also provides options to export each channel separately in multichannel image volumes, separate time volumes as single volumes, and remove top and bottom slices. Further details can be found in the GitHub repository https://github.com/ctpn/minivess.

Ground-truth annotation

Pre-processing

To create segmented image volumes, images were first preprocessed in Python. Single channel image volumes were individually processed using histogram equalization (to adjust image contrast; scikit-image Contrast Limited Adaptive Histogram Equalization), median filtering (for smoothing; window size = 3, 5), morphological operators (binary closing, to fill holes), and thresholded into binary images. If present, image slices with poor SNR were removed from the top of a stack. Such slices were present if pial vessels were broken during surgery, causing dye to leak on the surface of the brain. Fine-tuning of binary images was achieved using the Paint, Erase, Smoothing, Islands, and Logical operators effects in the Segment Editor module in 3D Slicer46. 3D Slicer is a free and open-source platform used for 3D image visualization, segmentation, and registration, among others. For manual corrections, emphasis was placed on minimizing manual drawing to reduce human error, and smoothing edges. For example, jagged borders (arrows) observed in the first round of segmentation are smooth by the final segmentation. A general workflow of the pipeline to achieve ground-truth annotations is shown in Fig. 3. Example image pre-processing code is available in the MiniVess Github repository https://github.com/ctpn/minivess.

Machine learning

To improve segmentations, a 2D U-Net23 was trained using raw images and the preprocessed, segmented images. The U-Net consisted of 5 channels, consisting of 16, 32, 64, 128, and 256 filters, a stride of 2, batch normalization, Adam optimization (1e-4 learning rate), and the Dice loss function. The training was split into 80% for training, 10% for validation, and 10% for testing. Outputs from the U-Net were refined through manual corrections in 3D Slicer, using the same ‘Effects’ listed above. Manual corrections were kept to a minimum to ensure consistency in labels within each volume, and mainly consisted of removing false positives (e.g. noise) and conserving smooth boundaries. Final segmented volumes are the result of five rounds of 2D U-Net and manual corrections in 3D Slicer (Fig. 5). Supervised learning was implemented using the PyTorch-based MONAI framework47. MONAI offers open-source, standardized model architectures, dataloaders, and various preprocessing functions that are designed for biomedical imaging. We chose to build use MONAI so that our code can be repurposed to meet other users’ needs. Of note, since the goal was to create a dataset, not a segmentation model, the model used was quite simple.

Data Records

The data is stored in the EBRAINS repository in compressed NiFTi format (*.nii.gz)38. Each raw image stack has an annotated equivalent, designated by a ‘y’ in the file name. Details for each image can be found in the metadata, encoded in the NIfTI1 header format. Each image stack represents a different field-of-view in the cerebrovasculature. Information specific to each image stack can be found in Supplementary Table 1. Maximum projection images of all image volumes are shown in Fig. 4.

Technical Validation

Image volumes were collected and curated by CP (7 years of experience). Ground truth annotations were achieved by using classic image processing tools (see Methods), manual annotations by CP, and a 2D U-Net23. Accuracy of the final annotations were qualitatively confirmed by CP, and then independently confirmed by MFR and HS (Fig. 2) using 3D Slicer. Final segmentations are the result of 5 rounds of manual annotations or corrections and outputs of the U-Net. A comparison between rounds of segmentations can be found in Fig. 5. Quantitatively, the final round of segmentations showed better agreement and less variation with the fourth round of segmentations, than the first round of segmentations (Table 1).

Fig. 2
figure 2

Example of a raw and segmented image volume pair in the MiniVess dataset. Orthogonal views and maximum projections of (a,b) raw and (c) segmented image volumes. (d) 3D visualization of the whole image volume in (X,Y) and (X,Z) views using 3D Slicer. Scale bars = 100 m.

Full size image
Fig. 3
figure 3

Blood vessel segmentation workflow. (a) Raw image volumes from 2PFM imaging went through a series of (b) preprocessing steps, followed by manual corrections conducted in 3D Slicer46. (c) These segmented image volumes were further refined by using a 2D U-Net, which outputs (d) segmented image volumes. Raw and segmented image volumes, and code for preprocessing and U-Net workflows are provided.

Full size image
Fig. 4
figure 4

MiniVess volumes. Maximum projection images of all raw and segmented (‘y’) image volumes of the MiniVess data are shown for navigation purposes. For clarity, maximum projection images consist of a maximum of 30 slices in each volume. Dark regions within the image volume that appear to have no blood vessels (e.g. diagonal in mv16, top of mv18) likely have blood vessels, but are difficult to see due to ‘shadows’ cast by larger blood vessels above, which are not included in the image volume.

Full size image
Fig. 5
figure 5

Validation of segmentations. Cropped images from the XY and XZ planes taken from the raw volume (‘raw’), and the first (‘first seg’) and final (‘final seg’) rounds of segmentations, are shown for comparison (image volume mv18). In addition to the slice-of-interest (‘slice (XY)’ and ‘slice (XZ)’), the image slices above (‘slice - 1 (XY)’) and below (‘slice + 1 (XY)’) are also shown. In the first segmentation, vessel edges are less uniform (arrows), spurious noise is evident in the segmentations (angles), and vessel segments that are outside of the slice-of-interest, but present in the slices above or below, are included in the segmentation of the current slice (compare dotted outlines). In contrast, in the final segmentation, vessel edges are more uniform, and segmentations are closer to what is visible in the current slice only, according to acquisition parameters. Each round of segmentation consisted of manual corrections and U-Net outputs. For manual corrections of U-Net outputs, emphasis was placed on minimizing manual drawing to reduce human error, and smoothing edges. For example, jagged borders (arrows) observed in the first round of segmentation are smooth by the final segmentation.

Full size image
Table 1 Quantification of the final round of segmentations with the first round, and the fourth round.
Full size table

Usage Notes

The MiniVess dataset38 contains image volumes of cerebrovasculature from wild-type mouse, transgenic mouse, and transgenic rat brains. Although small in size, the variety of background strains and species in the MiniVess dataset represents rodent strains that are commonly used in wet labs.

The dataset can be downloaded as NiFTi (.nii.gz) files which can then be easily uploaded into machine learning models, or manipulated using FIJI (Fiji Is Just ImageJ), Python, MATLAB, etc. We provide a tutorial of how to use the MiniVess dataset in a U-Net, built in the MONAI framework (https://github.com/project-monai/monai).The MONAI framework also provides several tutorials using NiFTi images, which can be further explored using the MiniVess dataset.

Limitations of the dataset

A limitation of the MiniVess dataset is the lack of diversity. All images were collected using the same microscope and lens, by the same operator. Only the species (mouse or rat), sex, and genotype (wild type or non-transgenic TgCRND848) (see EBRAINS metadata38). As such, image quality of the raw image stacks will likely differ from aged or disease animal models that have greater tissue autofluorescence, such as transgenic TgCRND8 mice that exhibit human β-amyloid 40 and β-amyloid 41 (non-transgenic animals do not exhibit pathology48).

By making the raw and annotated data available, we hope that the MiniVess dataset can be used as a validation dataset by those evaluating their supervised, semi-supervised, or unsupervised segmentation models, and assist the field to use more data-centric ways to design and evaluate their segmentation models.