Deep learning-enabled realistic virtual histology with ultraviolet photoacoustic remote sensing microscopy

The goal of oncologic surgeries is complete tumor resection, yet positive margins are frequently found postoperatively using gold standard H&E-stained histology methods. Frozen section analysis is sometimes performed for rapid intraoperative margin evaluation, albeit with known inaccuracies. Here, we introduce a label-free histological imaging method based on an ultraviolet photoacoustic remote sensing and scattering microscope, combined with unsupervised deep learning using a cycle-consistent generative adversarial network for realistic virtual staining. Unstained tissues are scanned at rates of up to 7 mins/cm2, at resolution equivalent to 400x digital histopathology. Quantitative validation suggests strong concordance with conventional histology in benign and malignant prostate and breast tissues. In diagnostic utility studies we demonstrate a mean sensitivity and specificity of 0.96 and 0.91 in breast specimens, and respectively 0.87 and 0.94 in prostate specimens. We also find virtual stain quality is preferred (P = 0.03) compared to frozen section analysis in a blinded survey of pathologists.


Figure 2 .
Figure 2. Example quantitative comparison of paired virtual and true H&E-stained histology images.Measured results for raw images at 390 nm optical resolution and a 250 nm pixel spacing are shown (top), in addition to low-pass Gaussian filtered results simulating an effective 2 µm optical resolution (bottom).MS-SSIM, multi-scale structural similarity index measure; PSNR, peak signal-to-noise-ratio; PCC, Pearson correlation coefficient.

Figure 3 .
Figure 3. Frequency domain comparison for representative examples of virtual and true H&E-stained histology.a) Virtual histology image.b) True H&E-stained histology image corresponding to a). c-d) Log-scale 2D spatial frequency spectrum of a) and b), respectively.e) Virtual histology image.f) True H&E-stained histology image corresponding to e). g-h) Log-scale 2D spatial frequency spectrum of e) and f), respectively.

Table 1 .
Comparison of key technical specifications for current virtual histology platforms.It is important to note that specifications represent implemented trade-offs in real reported system designs, but not necessarily fundamental limitations.† Lower-bound on effective cm 2 scan times extended from reported field-of-view imaging times: does not account for translational motion between tiles for image mosaicking, or overlap required for image stitching.* * Transmission-mode only.

Table 2 .
Comparison of reported quantitative metrics for alternative virtual staining techniques.
* Imaging of single cells or smears rather than tissues may inflate results where uniform background forms large fraction of images.† Analysis performed in L * a * b colorspace which may bias values.

Table 3 .
Quantitative nuclei metrics for brightfield H&E and virtual histology comparisons.Median values of nuclear size and shape metrics for true brightfield H&E histology as compared to our virtual histology method.a, length of semi-major axis; b, length of semi-minor axis (for ellipses with identical second central moments to segmented region); A, area.D(x, y), Euclidean distance function; c, position of segmented object centroid; p i , position of i th pixel.Overbar denotes mean taken over segmented object pixels.

Table 4 .
Deep learning-enabled breast tissue virtual histology diagnostic concordance study results.Pathologist diagnostic concordance study results for deep learning-enabled breast tissue virtual histology images and true-H&E stained counterparts, with a consensus value representing the mode of pathologist (P) interpretations for each respective image.B, benign; M, malignant.8/12

Table 5 .
Deep learning-enabled prostate tissue virtual histology diagnostic concordance study results.Pathologist diagnostic concordance study results for deep learning-enabled prostate tissue virtual histology images and true-H&E stained counterparts, with a consensus value representing the mode of pathologist (P) interpretations for each respective image.B, benign; M, malignant.

Table 6 .
Blinded pathologist survey of subjective stain quality.A summary of scores for each image and pathologist rater.VS, deep learning-enabled virtual H&E stain; HS, frozen section H&E-stained histological stain; HD, hematoxylin detail; ED, eosin detail; SQ, overall stain quality; SEM, standard error of the mean.Ratings were provided on the following scale: 1, unacceptable; 2, acceptable; 3, very good quality; 4, perfect stain.Source data are provided as a Source Data file.