Fiber enhancement and 3D orientation analysis in label-free two-photon fluorescence microscopy

Abstract

Fluorescence microscopy can be exploited for evaluating the brain’s fiber architecture with unsurpassed spatial resolution in combination with different tissue preparation and staining protocols. Differently from state-of-the-art polarimetry-based neuroimaging modalities, the quantification of fiber tract orientations from fluorescence microscopy volume images entails the application of specific image processing techniques, such as Fourier or structure tensor analysis. These, however, may lead to unreliable outcomes as they do not isolate myelinated fibers from the surrounding tissue. In this work, we describe a novel image processing pipeline that enables the computation of accurate 3D fiber orientation maps from both grey and white matter regions, exploiting the selective multiscale enhancement of tubular structures of varying diameters provided by a 3D implementation of the Frangi filter. The developed software tool can efficiently generate orientation distribution function maps at arbitrary spatial scales which may support the histological validation of modern diffusion-weighted magnetic resonance imaging tractography. Despite being tested here on two-photon scanning fluorescence microscopy images, acquired from tissue samples treated with a label-free technique enhancing the autofluorescence of myelinated fibers, the presented pipeline was developed to be employed on all types of 3D fluorescence images and fiber staining.

Introduction

The integration of different imaging modalities at micro-, meso- and macroscopic spatial scales is widely recognized as an essential requirement for advancing our current knowledge of the human brain connectional anatomy^1,2,3. In this regard, the possibility to investigate brain myeloarchitectonics with single fiber resolution over extended tissue volumes represents a compelling research area. The introduction of increased magnetic field strengths and the development of improved theoretical contrast models in diffusion-weighted magnetic resonance imaging (dMRI), fostered by the Human Brain Connectome project⁴, have recently enabled the quantitative mapping of the human brain connectivity with sub-millimetric resolution, and improved angular accuracy^5,6,7,8. The ultra-high b-values available nowadays in clinical and pre-clinical scanners provide increased signal-to-noise ratios and, thus, enhanced spatial resolution; this, in turn, represents a key factor in improving the quality of dMRI-based fiber tractography^9,10. However, despite its unique capability to provide an in vivo interrogation of the structural organization and connectivity of the human brain, state-of-the-art dMRI may still inadequately capture the tissue microstructure within voxels containing complex branching or interdigitated fiber architectures³. Therefore, the development of a gold standard method capable of providing reliable multiscale ground-truth datasets for the comprehensive validation of dMRI-based connectivity information is of paramount importance. This has motivated several research groups to apply dMRI to post-mortem histological tissue sections in combination with a range of optical modalities, namely 3D polarized light imaging (3D-PLI)^1,11, 3D polarized-sensitive optical coherence tomography (3D-PSOCT)^10,12, 2D and 3D confocal scanning fluorescence microscopy^13,14,15 and 3D light-sheet fluorescence microscopy (LSFM)¹⁶, also combining OCT and LSFM¹⁷, with the aim to identify suitable approaches for validating dMRI-based fiber tractography, and unveiling its potential limitations in brain regions with challenging fiber compositions and geometrical configurations. 3D-PLI and 3D-PSOCT exploit the inherent birefringence of brain tissue, analyzing the change in polarization of light transmitted through unstained samples in order to detect fiber structures and estimate their 3D orientation. On the other hand, fluorescence microscopy generally relies on tissue clearing methods, such as CLARITY (Clear Lipid-exchanged Acrylamide-hybridized Rigid Imaging compatible Tissue hYdrogel)^18,19, or SWITCH (System-Wide control of Interaction Time and kinetics of CHemicals)^20,21,22 to render samples transparent, and histological stains for generating contrast between fibers and other tissue components. Furthermore, despite offering a superior spatial resolution with respect to polarimetry-based imaging modalities, which enables the targeting of finer microstructural details, in fluorescence microscopy the evaluation of fiber orientations is subject to the application of dedicated image processing techniques, namely Fourier analysis^23,24 and structure tensor analysis^13,14,15. However, a relevant limitation of these techniques is the non-specificity of the estimated orientation vectors, as they are not able to distinguish between fibers and surrounding brain tissue. In this regard, Morawski and co-workers¹⁶ have applied a supervised Random Forest classifier, cascaded with a multiscale enhancement Frangi filter²⁵, for the automatic identification of fiber structures. However, their orientation analysis was limited to the image plane due to the marked anisotropic resolution of the employed light-sheet microscope and, moreover, it relied on the availability of manual segmentations produced by expert neuroanatomists. Here we present a novel image processing pipeline built around a 3D implementation of the Frangi filter that enables the unsupervised enhancement of myelinated fiber structures of varying diameters in tiled brain microscopy reconstructions of arbitrary size, and the generation of accurate 3D fiber orientation maps in both grey and white matter regions. The analysis pipeline features the generation of orientation distributions functions (ODFs) at arbitrary spatial scales²⁶, that may be used in the future to enable a direct comparison with micrometric fiber orientations obtained via 3D-PLI or 3D-PSOCT, and to validate the meso- and macro-scale brain connectivity information targeted by state-of-the-art dMRI.

Results

Figure 1

Workflow for the analysis of 3D fiber orientations from large-volume fluorescence microscopy images, showing the sample preparation, imaging and image processing stages (created with BioRender.com).

In this work, we combined the myelin autofluorescence enhancement achieved by a pioneering label-free preparation protocol named MAGIC (Myelin Autofluorescence imaging by Glycerol Induced Contrast enhancement²⁷) (Fig. 1a) with the sub-micron resolution and deep imaging capabilities offered by two-photon fluorescence microscopy (TPFM) (Fig. 1b). Adjacent TPFM image tiles were preliminary corrected for shading so as to suppress unwanted vignetting artefacts (Fig. 1c), and then aligned and stitched to generate high-resolution tiled mesoscale reconstructions of the human brain fiber organization (Fig. 1d). Single fiber orientations were finally computed from the myelinated fiber autofluorescence at $\lambda$ = 482 nm by means of a novel image processing pipeline (Foa3D) which is presented in detail in the following Methods section (Fig. 1e).

In order to validate the reliability of the 3D fiber orientation maps estimated by the developed software tool, random image slices were automatically extracted from a reference dataset acquired from different subjects, comprised of 10 TPFM stacks including grey matter (GM) tissue and 10 stacks including white matter (WM). These image slices were then methodically rotated within and out of the image plane of the microscopy system, and fed to the Frangi-based orientation analysis pipeline, quantitatively evaluating the degree of agreement between the resulting distributions of the fiber azimuth ($\phi _{xy}$) and elevation ($\theta _{zy}$) angles and the ones obtained from the original unrotated slices, in terms of their median value and their Bhattacharyya coefficient. The results of the validation procedure are summarized in Fig. 2, which separately shows the accuracy performance related to the assessment of the $\phi _{xy}$ and $\theta _{zy}$ angles in GM and WM image samples. A detailed example of the adopted validation strategy is provided in the Supplementary Information (Fig. S1).

Figure 2

Validation of the 3D Frangi-based fiber orientation maps accuracy ($N = N_{\text {TPFM}} \times N_{\text {ROT}}$, where $N_\text {TPFM}$ denotes the total number of sample image patches, and $N_\text {ROT} = 18$ is the number of test rotations). Left column: overall distributions of the inter-median errors; center column: absolute inter-median distances against the test rotation applied; right column: Bhattacharyya coefficient against test rotations. (a–c) Azimuth angle $\phi _{xy}$, grey matter images; (d–f) azimuth angle $\phi _{xy}$, white matter; (g–i) elevation angle $\theta _{zy}$, grey matter; (j–l) elevation angle $\theta _{zy}$, white matter.

In specific, the overall distributions grouping the inter-median errors pertaining to each tested rotation range (Fig. 2a,d,g,j) exhibit a near-zero mean value, which implies the absence of a systematic bias in the developed fiber orientation analysis. However, their standard deviation is notably lower in the case of the $\phi _{xy}$ orientations evaluated within the imaging plane of the TPFM system, where the optical resolution achieved is higher. This is made evident by the trend of the absolute inter-median error $|\Delta {\tilde{\phi }}_{xy}|$ along the whole range of tested rotations (Fig. 2b,e) whose median value, despite exhibiting a sensible increase for larger rotations, never exceeds 2$^{\circ }$ in the case of the azimuth angular coordinate $\phi _{xy}$, in both the test sets of GM and WM image patches. Conversely, Fig. 2h,k show a tendency towards relatively larger inter-median distances $|\Delta {\tilde{\theta }}_{zy}|$ for the elevation angle between the myelinated fiber structures detected by the pipeline and the imaging plane of the TPFM system. Despite being higher, median error values nevertheless stay below 3$^{\circ }$ in both the evaluated GM and WM samples. Moreover, as highlighted by the boxplots of the Bhattacharyya coefficient, representing the amount of overlap with the original reference angular distributions of $\theta _{zy}$ (Fig. 2i,l), the image processing pipeline appears to exhibit an asymmetrical performance with respect to the direction of the applied rotation. This may be ascribed to the preferential orientation statistics of the selected WM image tiles, despite the random axis inversion preliminarily applied to the image patches considered in the present validation. On the other hand, consistently with the trend of the inter-median distance $|\Delta {\tilde{\phi }}_{xy}|$, the estimated Bhattacharyya coefficients are remarkably uniform for the different planar rotations applied about the z-axis, with their median value never dropping below 0.99 in both the GM and WM image samples (Fig. 2c,f). Tables S1 and S2 in the Supplementary Information report the complete median values and interquartile ranges of the distributions shown in Fig. 2.

Figure 3

Runtime performance of the image processing pipeline: (a) execution times and memory usage of the Frangi-based fiber orientation analysis stage against the number of spatial scales of the Frangi filter ($\sigma$ = 1.25 $\upmu \hbox {m}$, image chunk size = 50 MB); (b) total execution time against different combinations of the spatial resolution of the fiber ODFs and the size of the basic image chunks analyzed iteratively ($\sigma$ = 1.25 $\upmu \hbox {m}$); (c) fiber ODFs estimation time, against different configurations of the designed pipeline.

Figure 4

Frangi-based orientation analysis of myelinated fibers in a mesoscale tiled TPFM reconstruction (final stitched size: 8859.9 $\upmu {\hbox {m}} \, \times$ 5204.8 $\upmu {\hbox {m}} \, \times$ 98.0 $\upmu {\hbox {m}}$, acquisition time: 14h 12min) of a brain region in between the primary and secondary visual cortex (V1/V2): (a) multichannel image (fibers: $\lambda$ = 482 nm, cell bodies: $\lambda$ = 618 nm) aligned and fused using ZetaStitcher (scale bar: 0.5 mm)^28,29, following the uneven illumination correction achieved via the CIDRE method³⁰; (b) myelinated fiber ODF map ($\lambda$ = 482 nm) returned by the image processing pipeline ($\alpha$ = 0.001, $\beta$ = 1, scales = 1.25, 2.5 $\upmu \hbox {m}$; ODF super-voxel: 80 $\upmu {\hbox {m}} \, \times$ 80 $\upmu {\hbox {m}} \, \times$ 80 $\upmu \hbox {m}$); (c) MIP of a 10 $\upmu \hbox {m}$-deep section including grey matter (green square, scale bar: 50 $\upmu \hbox {m}$); (d) corresponding grey matter ODFs (super-voxel: 10 $\upmu {\hbox {m}} \, \times$ 10 $\upmu {\hbox {m}} \, \times$ 10 $\upmu \hbox {m}$); (e) MIP of a 10 $\upmu \hbox {m}$-deep section including white matter (red square); (f) corresponding white matter ODFs (super-voxel: 10 $\upmu {\hbox {m}} \, \times$ 10 $\upmu {\hbox {m}} \, \times$ 10 $\upmu {\hbox {m}}$). Fiber ODF maps were displayed using the open-source software package for medical image processing and visualisation MRtrix3³¹.

Figure 3 summarizes the runtime performance of the Frangi-based fiber orientation analysis pipeline, applied to the TPFM reconstruction shown in Fig. 4. In detail, using Frangi filters with an increasing number of spatial scales determines an expected rise in the memory usage of the fiber orientation analysis stage (Fig. 3a). However, by parallelizing the estimation of the vesselness probability function (Eq. 9, Fig. 5), the interrogation of multiple scales does not lead to a proportional increase in the computational time of the filter. This accounts for a large percentage of the total execution time of the pipeline (Fig. 3b), with respect to the subsequent estimation of the fiber ODFs (Fig. 3c). As shown in ², the adoption of larger ODF super-voxel sizes, and the consequent decrease in their overall number, is not accompanied by a monotonic reduction in the time required to generate the fiber ODF maps of the whole TPFM volume image: this is related to the continuous setting of Alimi’s analytical approach, which computes the spherical harmonics coefficients (Eq. 16) for each native voxel of the ODF compartments, independently from their size. Similarly, processing larger chunks of the TPFM reconstruction does not appear to be generally beneficial for the computational efficiency of the developed processing pipeline, with a tendency towards longer execution times when the basic chunk size is increased, in spite of their lower number (Fig. 3b). Figure 4 finally highlights the capability of the image processing pipeline to reproduce the orientation of myelinated fiber structures at multiple spatial scales in unstained brain tissue slices imaged by TPFM. Adopting differently sized ODF super-voxels, thus producing a different level of downscaling of the high-resolution vector fields returned by the Frangi-based orientation analysis, allows to shift the focus from microscopic single fiber tracts to mesoscopic long-range fiber bundles in WM areas.

Discussion

In this study, we employed two-photon excitation microscopy with the glycerol-based MAGIC tissue preparation protocol to acquire 3D micron-resolution mesoscopic reconstructions of the human brain myeloarchitectonics, without the requirement of exogenous staining of myelinated fiber axons. We then applied the image processing tool presented in this work to derive fiber-specific orientation maps, exploiting a multiscale 3D Frangi filter for obtaining a targeted enhancement and segmentation of elongated image structures. The optimal spatial scale of the filter with respect to the cross-sectional size of myelinated fibers was thoroughly assessed, whereas the internal sensitivity parameters related to the geometrical features exploited by the enhancement mechanism were empirically tailored through a qualitative evaluation of the improvement of the fiber autofluorescence contrast at 482 nm, and the suppression of neuronal bodies with respect to the soma-specific fluorescence emission at 618 nm. A data-driven fine-tuning of these parameters on a manually generated ground truth of classified TPFM images of myelinated fiber structures may be considered in future work. Nonetheless, the results of the validation procedure outlined in the Methods section demonstrate that the current configuration of the fiber orientation analysis pipeline is able to provide an accurate evaluation of white matter fiber tract and grey matter single fiber 3D orientations, even without the application of cutting-edge deep learning techniques, and despite the widening of the point spread function (PSF) along the optical axis of the employed microscopy system. In general, the poorer axial resolution of fluorescence microscopes, LSFM systems particularly, poses a critical restraint on the feasibility of a 3D quantification of myelinated fiber orientations, and has forced other authors to limit their studies to 2D analyses orthogonal to the optical axis^16,23. In the present work, this translated in a relatively higher reliability of the $\phi _{xy}$ fiber azimuth angles, both in grey and white matter regions. The achieved performances may, however, increase as a result of a preliminary deconvolution of the TPFM volume images, not applied here because of the severe spatial undersampling of the PSF. Nevertheless, as pointed out by Morawski¹⁶, this inherent limitation can be overcome by employing multi-view imaging systems, such as dual inverted selective plane illumination microscopes (diSPIM)³², and making use of dedicated multi-view deconvolution algorithms.

The unsupervised nature of the developed fiber orientation analysis tool brings the key advantage of relaxing the need for a high-quality ground truth of volumetric segmentations of fiber architectures, manually annotated by expert operators. Nevertheless, cascading the 3D Frangi filter with state-of-the-art deep neural networks, such as the 3D U-Net classifier recently proposed in³³ may lead to a remarkable improvement of the automatic identification of myelinated fibers with respect to the binarization mechanism of Li’s thresholding algorithm, which minimizes the cross entropy between the classified foreground and the foreground mean, and the background and the background mean. Training 3D deep learning models generally requires a comprehensive manual annotation of volumetric microscopy images, possibly entailing an overwhelming human effort that was not feasible in the present work. However, in this regard, semi-supervised frameworks as the one introduced in³⁴, which make use of sparse partial annotations complemented by pseudo-labels inferred on unlabelled regions, may speed up the training process.

In the present study, the accuracy of the Frangi-based fiber orientation analysis pipeline was evaluated at the micron scale, analyzing image slices randomly sampled from a set of TPFM stacks of grey and white matter, and properly rotated within and out of the image plane of the microscopy setup. In view of this, future work may consider a multimodal evaluation of myelinated fiber orientations, comparing the 3D vector maps returned by the developed pipeline with the ones directly generated by means of polarimetry-based techniques, such as RP-CARS (Rotating-Polarization Coherent Anti-Stokes Raman Scattering^35,36) and the already mentioned 3D-PLI and 3D-PSOCT. Indeed, the unsurpassed spatial resolution of fluorescence microscopy may help corroborating 3D-PLI and 3D-PSOCT fiber orientation estimates or, on the other hand, characterize potential limitations of these polarimetry-based modalities.

Most importantly, the combination of the label-free MAGIC preparation protocol, TPFM, and 3D Frangi-based analysis of myelinated fiber orientations may represent a powerful approach for a quantitative histological validation of 3D fiber ODF maps obtained by dMRI, aiding the assessment of the brain region-specific reliability of state-of-the-art dMRI-based tractography.

Methods

Specimen collection

Tissue sections imaged and analyzed in this study were obtained from two postmortem human brains. The human brains were acquired in accordance with the ethics committee at the Medical Faculty of the University of Rostock, Germany (#A2016-0083) and the Netherlands Institute of Neuroscience, Amsterdam. The patients/participants provided written informed consent for the general use of post mortem tissue for aims of research and education. The experimental protocol was approved by the ethics committee of the medical faculty of the Heinrich Heine University Düsseldorf (#4863) and by the Netherlands Brain Bank (ethics approval NBB-1037/2018). All methods were carried out in accordance with relevant guidelines and regulations.

MAGIC preparation protocol and two-photon fluorescence microscopy

Human brain samples were preliminarily treated for fluorescence microscopy following the label-free MAGIC preparation technique, developed by Costantini et al.²⁷. This method originates from the evidence that glycerol removal from fixed and embedded brain tissue leads to a specific increase in myelin autofluorescence and thus to an enhanced signal-to-background ratio of myelinated axons. In detail, brain tissue was initially fixed with a 4% paraformaldehyde (PFA) solution for >12 weeks. Then, it was first embedded in a 10% glycerol, 2% DMSO, 4% formaldehyde solution and, second, in a 20% glycerol, 2% DMSO, 4% formaldehyde solution for >3 weeks. These steps were conducted at a tightly controlled temperature of 4$^{\circ }\hbox {C}$. Afterwards, brain samples were treated with a 2% dimethyl sulfoxide solution for cryoprotection and immersed in isopentane (${-50}^{\circ }\hbox {C}$) for >30 mins. Next, frozen human brains were sliced at a temperature of ${-30}^{\circ }\hbox {C}$ into 60 $\upmu \hbox {m}$-deep coronal sections using a cryostat microtome (Leica Microsystems, Germany). Finally, brain tissue slices were incubated at ambient temperature in a 0.01 M phosphate buffer saline solution (PBS) for three months. Prior to imaging, sections were coverslipped and mounted in PBS.

Brain slices treated with the MAGIC preparation protocol were imaged using a custom TPFM system. As excitation source, the system employs a Chameleon (Coherent, US) tunable mode-locked Ti:Sapphire laser (120 fs pulse width, 90 MHz pulse rate) operating at 800 nm. The laser source is optically coupled with a custom lateral scanning system comprising a pair of galvanometric mirrors (LSKGG4/M, Thorlabs, US), and focused onto the brain tissue slices by means of a tunable 25x objective (LD LCI Plan-Apochromat 25x/0.8 Imm Corr DIC M27, Zeiss, Germany), characterized by a 0.8 numerical aperture and a free working distance of 0.57 mm. The resulting field of view is 450 $\upmu {\hbox {m}} \, \times$ 450 $\upmu \hbox {m}$. $\approx$100 mW were delivered to the imaged brain samples, adopting a pixel dwell time of 5 ${\upmu \hbox {s}}$. 482/35 nm and 618/50 nm single-band bandpass emission filters (BrightLine^®, Semrock, US) were respectively used for detecting myelinated fibers and neuronal bodies. The fluorescence intensity signal was collected by a GaAsP photomultiplier tube (H7422, Hamamatsu Photonics, JP) and digitized with 8-bit precision. The PSF of the TPFM system was previously characterized in²⁷, by imaging 100 nm beads (FluoSpheresTM carboxylate-modified microspheres, yellow-green fluorescent, Thermo Fisher Scientific, US) embedded at a 1:1000 concentration in a PBS gel reproducing the refractive index of the immersion medium employed for the brain slice imaging. The average shape of the PSF was estimated using the Huygens PSF distiller tool (version 19.04, Scientific Volume Imaging, NL), resulting in a measured FWHM of 0.692, 0.692, 2.612 $\upmu \hbox {m}$ along the x, y, and z axes, respectively. No preliminary deconvolution was applied to the TPFM images acquired in this work. The TPFM system features a closed-loop XY stage (U-780 PILine XY Stage System, Physik Instrumente, Germany) enabling the lateral translation of the imaged specimen, and a closed-loop piezoelectric stage (ND72Z2LAQ PIFOC Objective Scanning System, Physik Instrumente, Germany) for the vertical displacement of the objective. The LabVIEW software controlling the system enables the sequential acquisition of adjacent overlapping 3D tiles, allowing for mesoscale reconstructions of entire brain slices with a maximum lateral resolution of 0.44 $\upmu \hbox {m}$ (corresponding to 1024 px $\times$ 1024 px images) and a minimum axial sampling step of 1 $\upmu \hbox {m}$. A voxel size of 0.88 $\upmu {\hbox {m}} \, \times$ 0.88 $\upmu {\hbox {m}} \, \times$ 1 $\upmu \hbox {m}$ and a lateral tile overlap of 40 $\upmu \hbox {m}$ were adopted in the present study.

Frangi filter theory

The Frangi filter produces a specific enhancement of tubular objects in 2D or 3D images, exploiting geometrical information on the local gray level structure derived from the eigenvalue decomposition of Hessian matrices estimated at the pixel level. As detailed in²⁵, the structural properties of an image I can be characterized within a neighborhood defined by the scale space parameter $\sigma$ through the following second-order Taylor expansion:

$$\begin{aligned} I({\bf {x}}+\delta {\bf {x}},\sigma )\approx I({\bf {x}},\sigma )+\delta {\bf {x}}^T \nabla _{{\bf {x}},\sigma }+\delta {\bf {x}}^T H_{{\bf {x}},\sigma } \delta {\bf {x}} \end{aligned}$$

(1)

where $\nabla _{{\bf {x}},\sigma }$ and $H_{{\bf {x}},\sigma }$ denote the gradient vector and the Hessian matrix of the partial spatial derivates computed in x which respectively describe the first- and second-order local structure of the image. In the particular case of a 3D image, the Hessian matrix is defined as:

$$\begin{aligned} H_{{\bf {x}},\sigma } = \begin{bmatrix} \dfrac{\delta ^2}{\delta x^2}I({\bf {x}},\sigma ) &{} \dfrac{\delta }{\delta x}\dfrac{\delta }{\delta y}I({\bf {x}},\sigma ) &{} \dfrac{\delta }{\delta x}\dfrac{\delta }{\delta z}I({\bf {x}},\sigma ) \\ \dfrac{\delta }{\delta y}\dfrac{\delta }{\delta x}I({\bf {x}},\sigma ) &{} \dfrac{\delta ^2}{\delta y^2}I({\bf {x}},\sigma ) &{} \dfrac{\delta }{\delta y}\dfrac{\delta }{\delta z}I({\bf {x}},\sigma ) \\ \dfrac{\delta }{\delta z}\dfrac{\delta }{\delta x}I({\bf {x}},\sigma ) &{} \dfrac{\delta }{\delta z}\dfrac{\delta }{\delta y}I({\bf {x}},\sigma ) &{} \dfrac{\delta ^2}{\delta z^2}I({\bf {x}},\sigma ) \end{bmatrix} \end{aligned}$$

(2)

where the scale representation of the image I at scale $\sigma$, $I({\bf {x}},\sigma )$, is generally obtained via a preliminary convolution with a Gaussian kernel of variance $\sigma$²:

$$\begin{aligned} I({\bf {x}},\sigma )= & {} G({\bf {x}},\sigma )*I({\bf {x}}) \end{aligned}$$

(3)

$$\begin{aligned} G({\bf {x}},\sigma )= & {} \frac{1}{{\sqrt{2\pi \sigma ^2}}^3} \textrm{e}^{-\dfrac{{\Vert {\bf {x}} \Vert }^2}{2\sigma ^2}} \end{aligned}$$

(4)

In order to compensate for the progressive attenuation of spatial image derivatives at increasing analysis scales, with the consequential underestimation of $H_{{\bf {x}},\sigma }$ matrices for larger $\sigma$ values, the Frangi filter makes use of scale-normalized spatial derivatives, that is:

$$\begin{aligned} \dfrac{\delta }{\delta x}I({\bf {x}},\sigma ) = \sigma \cdot \dfrac{\delta }{\delta x}I({\bf {x}}) \end{aligned}$$

(5)

Otherwise, finer image details would be systematically over-enhanced with respect to coarser structures.

The eigenvalue decomposition of the scaled Hessian matrix determines the principal directions of the second-order structure of the image; specifically, the Hessian matrix maps a finite image neighborhood onto a 3D ellipsoid whose semiaxes have a length corresponding to the estimated eigenvalues $\lambda _i$ and are oriented according to the related eigenvectors ${\bf {v}}_i$. This second-order ellipsoid thus provides an intuitive description of the local image behavior, exploited by Frangi et al. to design geometrical features which enable the specific segmentation of tube-like objects. Indeed, in presence of specific spatial contrast patterns, particular relations must hold between the three eigenvalues (Table 1). Namely, assuming $|\lambda _1 | \le | \lambda _2 | \le | \lambda _3 |$, a pixel belonging to a bright 3D tubular structure over a dark background would be ideally characterized by a near-zero $\lambda _1$ (the dominant eigenvalue identifying the local direction of minimum intensity variation), and by $\lambda _2$ and $\lambda _3$ having a large similar magnitude and a consistent negative sign. On this basis, the first geometrical feature proposed by Frangi et al. accounts for the similarity to blob-like structures, which attains its maximum for locally isotropic objects (Table 1, first row):

$$\begin{aligned} R_{\text {B}}(\sigma ) = \frac{|\lambda _1(\sigma )|}{\sqrt{|\lambda _2(\sigma )| |\lambda _3(\sigma )|}},\;\;\;\;\;R_{\text {B}}(\sigma )\in [0,1] \end{aligned}$$

(6)

This blobness metric alone, however, would not discriminate between plate-shaped and tubular objects; accordingly, Frangi et al. introduced an additional geometrical grey-level-invariant feature, which evaluates the aspect ratio of the image ellipsoid in the plane orthogonal to the direction of minimum contrast, i.e. the ratio between the two largest second derivatives:

$$\begin{aligned} R_{\text {A}}(\sigma ) = \frac{|\lambda _2(\sigma )|}{|\lambda _3(\sigma )|},\;\;\;\;\;R_{\text {A}}(\sigma )\in [0,1] \end{aligned}$$

(7)

Table 1 Relations between the eigenvalues of the Hessian matrix for different geometrical patterns (L/H: low/high absolute value; adapted from³⁷).

As shown in Table 1, this latter feature decays to 0 only within the neighborhood of plate-like structures, where $| \lambda _2(\sigma ) |$ becomes significantly lower than $| \lambda _3(\sigma ) |$.

Furthermore, Frangi et al. adopted a feature of image structureness, defined as the Frobenius norm of the Hessian matrix, which enables the distinction of low-contrast background regions, generally associated with small spatial derivatives and, thus, small eigenvalues:

$$\begin{aligned} S(\sigma ) = \sqrt{\sum _{d=1}^{D} \lambda _{d}^{2}(\sigma )} \end{aligned}$$

(8)

The above features are finally combined in a probability-like vesselness function which, in the case of a positive contrast polarity, is defined as:

$$\begin{aligned} V(\sigma ) = {\left\{ \begin{array}{ll} 0 &{} {\text {if }} \lambda _2(\sigma )> 0 \; or \; \lambda _3(\sigma ) > 0\\ \textrm{e}^{-\tfrac{R_\text {B}^{2}(\sigma )}{2\beta ^2}} \left( 1-\textrm{e}^{-\tfrac{R_\text {A}^{2}(\sigma )}{2\alpha ^2}}\right) \left( 1-\textrm{e}^{-\tfrac{S^2(\sigma )}{2\gamma ^2}}\right) &{} \text {otherwise} \end{array}\right. } \end{aligned}$$

(9)

where the product of the basic exponential terms implies the suppression of all image regions which do not concurrently meet all the above criteria.

$\beta$, $\alpha$ and $\gamma$ are internal parameters of the Frangi filter which respectively tune its sensitivity to variations in the $R_\text {B}$, $R_\text {A}$ and S features. Namely, an increase in $\beta$ would lead to a relatively higher sensitivity to blob-like structures, decreasing the related penalty introduced in the evaluation of the vesselness probability. On the other hand, lower $\alpha$ values would amplify the response of the filter to the presence of elongated structures. Usually, the $\alpha$ and $\beta$ sensitivity parameters are heuristically fixed for the specific application or image modality of interest. Conversely, since the structureness S is directly influenced by the dynamic range of the raw input image, the related $\gamma$ sensitivity may be automatically adapted: according to Frangi et al., setting $\gamma$ to half of the maximum Hessian norm obtained at each scale (i.e., half of the maximum image structureness, $S(\sigma )$) ensures robust results.

The Frangi filter was devised as a multiscale method: accordingly, the above vesselness likelihood function is estimated for different levels of interest of the image scale-space, after a convolution with Gaussian kernels of different variance, $\sigma ^2$. As done in the present implementation of the filter, the multiscale analysis of the vesselness probability function may be parallelized over multiple cores, thus making the computational time independent from the number of spatial scales of interest. Finally, the maximum intensity projection (MIP) of the filter response along the image scale dimension is considered:

$$\begin{aligned} V = \max _{\sigma _\text {min} \le \sigma \le \sigma _\text {max}} V(\sigma ) \end{aligned}$$

(10)

where, as discussed in the following, $V(\sigma )$ locally reaches its maximum at a scale directly related to the cross-sectional diameter of the tubular structures present in the image. The main steps of the Frangi filter’s algorithm are summarized in Fig. 5.

Figure 5

Block scheme representing the main computational steps of the Frangi filter. In the present application, the estimation of the vesselness probability function over different spatial scales was parallelized for efficiency’s sake.

Fiber orientation analysis pipeline

The image processing pipeline for the 3D analysis of myelinated fiber orientations (Foa3D) was developed in Python and is made available as an open-source software at https://github.com/lens-biophotonics/Foa3D. The following paragraphs describe its main stages.

TPFM image preprocessing

The separate TPFM image tiles composing the mesoscale reconstructions of the acquired brain slices are preliminarily aligned using ZetaStitcher, a custom software tool for high-resolution large volumetric stitching^28,29. ZetaStitcher exploits the overlap between adjacent image stacks to compensate for possible misalignments due to the microscopy setup. Firstly, a 2D cross-correlation map is evaluated at several imaging depths for each pair of neighboring stacks to estimate their relative position. Secondly, the final location of all stacks is determined through global optimization of the displacements applied to the individual pairs. ZetaStitcher is also employed to programmatically access and iteratively process basic sub-volumes fitting the system’s memory, thus enabling the analysis of submicron-resolution TPFM reconstructions of arbitrary extension.

First, however, the uneven illumination of the separate TPFM stacks is improved via the retrospective CIDRE correction method³⁰, so as to suppress the stitching artifacts which would otherwise arise when fusing the aligned TPFM tiles. Alternatively, this shading correction can also be performed using BaSiC³⁸, which similarly provides a retrospective estimate of the flat-field and dark-field images. The spatial intensity gain models related to the $\lambda$ = 618 nm and $\lambda$ = 482 nm wavelengths, shown in Fig. 6, were estimated using a reference dataset comprising 56299 and 66297 TPFM images, respectively. The effect of the CIDRE illumination correction on the fiber orientation maps generated by the Foa3D pipeline is shown in Fig. S2.

Furthermore, if required, the coronal XY-plane of the sliced TPFM sub-volumes is preliminarily blurred using a 2D Gaussian smoothing filter, adopting a variance tailored with respect to the anisotropy of the PSF of the TPFM system; i.e., $\sigma _{x}^{2} = \sigma _{\text {PSF}_z}^{2} - \sigma _{\text {PSF}_x}^{2}$, $\sigma _{y}^{2} = \sigma _{\text {PSF}_z}^{2} - \sigma _{\text {PSF}_y}^{2}$, where $\sigma _{\text {PSF}_i}^{2}$ represents the variance of the PSF model along the i-axis. Indeed, confocal laser scanning and light-sheet microscopes provide 3D images characterized by a poorer resolution along the direction of the optical axis: if not properly corrected, the anisotropic resolution of the microscope would introduce a systematic bias in the computation of the Hessian matrices of the second directional derivatives and, thus, in the resulting 3D orientation estimates¹⁶. Conversely, this transverse degradation of the optical resolution may be skipped if the TPFM tiles are preliminarily deconvolved. Following this optional smoothing stage, TPFM images are downsampled in the coronal plane, in order to obtain an isotropic 1 $\upmu {\hbox {m}} \, \times$ 1 $\upmu {\hbox {m}} \, \times$ 1 $\upmu {\hbox {m}}$ voxel size. While the TPFM tile alignment and illumination correction must be performed separately via the respective tools, the image resolution isotropization is integrated within the developed pipeline.

Figure 6

Example of the CIDRE-based tile illumination correction (MIPs, scale bar: 50 $\upmu \hbox {m}$): (a) spatial intensity gain model at $\lambda$ = 618 nm (neurons); (b) gain model at $\lambda$ = 482 nm (myelinated fibers); (c) original TPFM tile (stack size: 450 $\upmu {\hbox {m}} \, \times$ 450 $\upmu {\hbox {m}} \, \times$ 71 $\upmu {\hbox {m}}$, acquisition time: 3 min 33 s); (d) corrected TPFM tile (zero-light preserved mode³⁰).

Figure 7

(a) Cross-sectional view of the Frangi’s vesselness probability of a simulated 3D fiber (left), obtained by adopting increasing scale-to-radius ratios; (b) 1D cross-sectional vesselness profiles, normalized with respect to the original intensity; (c) normalized maximum intensity; (d) normalized FWHM. The optimal scale corresponds to a scale-to-radius ratio of 50%.

Multiscale Frangi filter

The Frangi filter was originally designed as a multiscale method for the analysis of angiography images²⁵, able to provide a concurrent enhancement of major arteries and veins, and peripheral blood vessels characterized by different cross-sectional sizes. However, for avoiding image artifacts, the analysis scales of the filter should be rigorously tuned with respect to the expected radius of the tubular structures of interest. As shown in ³⁷, the smoothing Gaussian kernel behaves similarly to a Dirac delta function if its standard deviation becomes too small compared to the diameter of the analyzed myelinated fibers; as a consequence, the scale-selection stage would yield the original fiber structure, and the subsequent analysis of the local Hessian matrices and estimation of the vesselness score would produce a spurious high-probability ridge at the fiber edge. Conversely, as the standard deviation gets relatively large, the Gaussian convolution would yield the kernel itself, with the fiber starting to act as a Dirac function. Fig. 7a shows the cross-sectional view of the Frangi’s vesselness probability of an ideal cylinder simulating a 3D myelinated axon (pixel density: 50 px/$\upmu \hbox {m}$; length/diameter aspect-ratio: 12), related to different scale-to-radius ratios. The evaluation of the 1D vesselness profile (Fig. 7b) in terms of maximum relative intensity (Fig. 7c) and FWHM (Fig. 7d) shows half of the expected fiber radius to be the optimal spatial scale which best preserves the original intensity and cross-sectional size of the analyzed tubular structure.

Regarding the three sensitivity parameters of the Frangi filter, $\gamma$ was automatically tailored for each filtered sub-volume, and filtering scale, to the maximum norm of the estimated Hessian matrices. On the other hand $\beta$ and $\alpha$, which respectively tune the sensitivity of the filter to the geometrical features $R_\text {B}$ and $R_\text {A}$ (Eqs. 6, 7), were empirically adjusted through a visual inspection of the processed images. In particular, the myelinated fiber enhancement and the background suppression were evaluated in the $\lambda$ = 482 nm channel, while the $\lambda$ = 618 nm channel was considered for assessing the rejection of the neuronal bodies. Indeed, this channel can be used as a comparative reference since the soma can be clearly distinguished due to the characteristic autofluorescence of lipofuscins, a pigment that accumulates with ageing in the lysosomal compartment of postmitotic cells, including neurons³⁹. This evaluation led to a final sensitivity configuration given by $\alpha =0.001$ and $\beta =1$ which, as made evident in Fig. 8a,b, is able to produce a marked selective enhancement of tubular fiber structures and a considerable rejection of the cell soma.

The Frangi-enhanced images generated from the $\lambda$ = 482 nm channel were binarized using Li’s minimum cross entropy thresholding method⁴⁰. 3D fiber orientations were then identified as the eigenvectors associated with the dominant eigenvalues of the Hessian matrices belonging to voxels classified as myelinated fiber structures. In order to improve the specificity of the resulting fiber orientation maps, provided by the inherent attenuation of non-tubular objects achieved by the Frangi filter, the developed pipeline optionally performs a postprocessing step which suppresses the neuronal bodies by masking the $\lambda$ = 618 nm channel via Yen’s binarization⁴¹. An example of this lipofuscin autofluorescence-based neuron segmentation is shown in Fig. 8c,d.

Figure 8

Frangi-based enhancement of myelinated fibers in the 3D TPFM tile shown in Fig. 6 (MIPs, scale bar: 50 $\upmu \hbox {m}$): (a) myelinated fibers autofluorescence ($\lambda$ = 482 nm), enhanced via the MAGIC preparation protocol²⁷; (b) Frangi’s fiber probability map ($\alpha =0.001$, $\beta =1$, scales = 1, 1.25, 1.5 $\upmu \hbox {m}$); (c) neuronal bodies ($\lambda$ = 618 nm); (d) neuron rejection mask (Yen’s thresholding method).

Analytical fiber orientation distribution functions

High-resolution fiber orientation data obtained at the native pixel size of the TPFM system can be integrated into ODFs²⁶, providing a comprehensive statistical description of 3D fiber tract orientations within larger spatial compartments or super-voxels. Due to their widespread use in the neuroimaging community, the estimation of TPFM-based ODFs is highly suitable for a multimodal quantitative comparison with spatial fiber architectures mapped by other advanced optical modalities, as 3D-PLI¹¹. Furthermore, the spatial downscaling produced by the ODF estimation allows to bridge the gulf between single fiber orientations obtained via optical microscopy and the meso- and macro-scale connectomics that is generally targeted by diffusion magnetic resonance imaging¹. In the developed image processing pipeline, the ODFs of myelinated fibers were estimated from the 3D orientation vector fields returned by the Frangi filtering stage by means of the fast analytical approach proposed by Alimi^42,43. The present method is computationally efficient and is characterized by improved angular precision and resolution with respect to deriving the ODFs by modeling local directional histograms of discretized fiber orientations². In detail, fiber orientation vectors are parametrized in terms of their coordinates $\phi$ and $\theta =90^{\circ }-\alpha$ on the unit sphere $S^2$, which respectively denote the azimuth angle representing the orientation of the fiber axis within the XY-plane, and the polar angle related to the $\alpha$ angle between the fiber and the XY-plane. Their square-integrable statistical distribution $f(\theta ,\phi ) \in L^2(S^2)$ can then be expanded as a linear combination of spherical harmonics:

$$\begin{aligned} f(\theta ,\phi ) = \sum _{l=0}^{\infty } \sum _{m=-l}^{l} c_{lm}Y_{l}^{m}(\theta ,\phi ) \end{aligned}$$

(11)

where $Y_{l}^{m}(\theta ,\phi )$ are the spherical harmonics of order l and degree m, defined by:

$$\begin{aligned} Y_{l}^{m}(\theta ,\phi ) = N_{l}^{m}P_{l}^{m}(\cos {\theta })\textrm{e}^{jm\phi } \end{aligned}$$

(12)

In the above expression, $P_{l}^{m}$ represents the associated Legendre polynomial whereas $N_{l}^{m}$ is a normalization factor given by:

$$\begin{aligned} N_{l}^{m} = {\left\{ \begin{array}{ll} \sqrt{\dfrac{2l+1}{4\pi }} &{} \text {if }m=0\\ (-1)^m \sqrt{2} \sqrt{\dfrac{2l+1}{4\pi }\dfrac{(l-|m|)!}{(l+|m|)!}} &{} \text {otherwise} \end{array}\right. } \end{aligned}$$

(13)

The coefficients $c_{lm}$ uniquely describing the fiber orientation distribution $f(\theta ,\phi )$ can be recovered from its spherical Fourier transform, defined as:

$$\begin{aligned} c_{lm} = \int _{0}^{2\pi } \int _{0}^{\pi } f(\theta ,\phi ) Y_{l}^{m}(\theta ,\phi ) \sin {\theta } d\theta d\phi \end{aligned}$$

(14)

In the adopted approach, fiber orientation vectors are regarded as two-dimensional Dirac delta functions on $S^2$; thus, the distribution $f_\text {SV}(\theta ,\phi )$ of K orientations included in a given super-voxel can be modeled as follows:

$$\begin{aligned} f_\text {SV}(\theta ,\phi ) = \dfrac{1}{K} \sum _{k=1}^{K} \delta (\cos {\theta } - \cos {\theta _k}) \delta (\phi - \phi _k) \end{aligned}$$

(15)

As shown by Alimi et al., substituting Eqs. 12 and  15 into Eq. 14, and applying the sifting property of Dirac functions, finally yields the continuous analytical expression of the spherical harmonics coefficients, reported below:

$$\begin{aligned} c_{lm} = \dfrac{1}{K} \sum _{k=1}^{K} N_{l}^{m}P_{l}^{m}(\cos {\theta })\textrm{e}^{-jm\phi } \end{aligned}$$

(16)

In this work, the series expansion of the analytical ODF was limited to the 6th order ($l_\text {max}$=6), as previously performed by Axer et al. for the estimation of 3D-PLI-based fiber ODFs¹.

Figure 9

Block diagram schematizing the image processing pipeline for the 3D analysis of myelinated fiber orientations. Adjacent microscopy image stacks are preliminary corrected for shading using the CIDRE method and then aligned to generate high-resolution mesoscale reconstructions of the human brain tissue using ZetaStitcher. Single fiber orientations are then obtained from the myelinated fiber autofluorescence at $\lambda$ = 482 nm using the novel Foa3D software tool, which iteratively analyzes basic sub-volumes of the whole image reconstruction. The $\lambda$ = 618 nm channel may be used to further reject the contribution of the soma in addition to the inherent attenuation produced by the Frangi filter (scale bar: 100 $\upmu \hbox {m}$).

Pipeline configuration and computational runtimes

The runtime performance of the designed image processing pipeline (Fig. 9) was evaluated on a high-performance computer cluster comprising a pair of twin computing nodes, each equipped with 192 GB of RAM and two Intel^® Xeon^® Silver 4214R 12-core processors operating at 2.4 GHz.

A first evaluation concerned the variation in the computational time and the memory usage of the Frangi-based fiber orientation analysis stage with respect to the number of spatial scales employed by the Frangi filter. In order to ensure an unbiased comparison, a fixed spatial scale of 1.25 $\upmu \hbox {m}$ was adopted in this regard i.e., a different number of scales of identical value was fed to the pipeline. Furthermore, the execution time of the entire processing workflow, including the generation of the Frangi-based fiber ODFs, was assessed adopting different combinations of the size of the basic image chunks analyzed iteratively, and the size of the orientation map super-voxels defined for the estimation of the fiber ODFs. In detail, chunk sizes of 50, 100, 150, 200 and 250 MB, and ODF super-voxel sides of 16, 32, 48, 64 and 80 $\upmu \hbox {m}$ were tested in this work. The present tests were conducted on a reference 8859.9 $\upmu {\hbox {m}} \, \times$ 5204.8 $\upmu {\hbox {m}} \, \times$ 98.0 $\upmu \hbox {m}$ TPFM reconstruction of a human brain region (252 3D tiles, 16.4 GB) in between the primary and secondary visual cortex (V1/V2) (Fig. 4).

Validation of fiber orientation estimates

An ad hoc validation module was developed for verifying the reliability of the 3D orientation estimates returned by the image processing pipeline designed in this work. 10 GM and 10 WM TPFM stacks were independently considered. Based on the characterization of the scale-dependent response of the Frangi filter to tubular structures of different cross-sectional size, three spatial scales, namely 1, 1.25 and 1.5 $\upmu \hbox {m}$, were adopted in the present validation; these respectively correspond to an optimal fiber diameter of 4, 5 and 6 $\upmu \hbox {m}$. In detail, an automatic image patch generator was conceived so as to randomly sample and analyze basic 75 $\upmu {\hbox {m}} \, \times$ 75 $\upmu {\hbox {m}} \, \times$ 15 $\upmu {\hbox {m}}$ (x, y, z) slices of the original TPFM stacks. The ranges of the sliced sub-volumes were properly extended in order to cope with the boundary artifacts caused by the convolution with the smoothing Gaussian kernel, implementing the scale selection step of the Frangi filter. Furthermore, each sampled image patch was randomly flipped along its three axes with the aim to decrease the dependency of the obtained results on the orientation statistics of the assessed image stacks. In detail, similarly to the approach followed in⁴⁴, the sliced patches were separately rotated about the z- and x-axis over a -45$^{\circ }$ to 45$^{\circ }$ range, with a sampling step of 5$^{\circ }$, in order to produce a controlled reorientation of myelinated fiber structures, respectively within and outside of the image plane of the TPFM system. A bicubic spline interpolation with prefiltering was applied when rotating the randomly sampled patches. The transformed images, and their original, unrotated version, were separately processed by the developed pipeline in order to derive patch-wise distributions of the in-plane azimuth angle $\phi _{xy}$ and the out-of-plane elevation angle $\theta _{zy}$. At this stage, the extended edge introduced to account for the smoothing-related boundary artifacts was properly masked prior to the evaluation of the angular distributions. These were corrected according to the applied 1D rotations, i.e. rotating the angular reference system, and then compared to the distributions generated from the original image. Specifically, the discrepancy between the distributions was characterized by means of the inter-median distances $\Delta {\tilde{\phi }}_{xy}$ and $\Delta {\tilde{\theta }}_{zy}$ and, secondly, by their Bhattacharyya coefficient⁴⁵:

$$\begin{aligned} B_\text {c}(p,q) = \sum _{x=1}^{N} \sqrt{p(x) \cdot q(x)},\;\;\;\;\;B_\text {c}(p,q) \in [0,1], \end{aligned}$$

(17)

which estimates the amount of overlap between two statistical samples, computed here adopting N=180 bins of 1$^{\circ }$ in width. A threshold of 1% was imposed on the relative number of pixels classified as belonging to myelinated fibers: TPFM patches not meeting this criterion for each tested rotation were systematically excluded from further consideration. This ensured that the above metrics were obtained only from adequately populated data samples, with an equal number of patches related to each tested rotation value.