Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy

Abstract

Obtaining frozen sections of bone tissue for intraoperative examination is challenging. To identify the bony edge of resection, orthopaedic oncologists therefore rely on pre-operative X-ray computed tomography or magnetic resonance imaging. However, these techniques do not allow for accurate diagnosis or for intraoperative confirmation of the tumour margins, and in bony sarcomas, they can lead to bone margins up to 10-fold wider (1,000-fold volumetrically) than necessary. Here, we show that real-time three-dimensional contour-scanning of tissue via ultraviolet photoacoustic microscopy in reflection mode can be used to intraoperatively evaluate undecalcified and decalcified thick bone specimens, without the need for tissue sectioning. We validate the technique with gold-standard haematoxylin-and-eosin histology images acquired via a traditional optical microscope, and also show that an unsupervised generative adversarial network can virtually stain the ultraviolet-photoacoustic-microscopy images, allowing pathologists to readily identify cancerous features. Label-free and slide-free histology via ultraviolet photoacoustic microscopy may allow for rapid diagnoses of bone-tissue pathologies and aid the intraoperative determination of tumour margins.

This is a preview of subscription content, access via your institution

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Fig. 1: Rapid label-free UV photoacoustic histology via deep learning.
Fig. 2: Label-free 3D contour-scanning UV-PAM of thick (>1 cm) unprocessed bone specimens.
Fig. 3: Label-free UV-PAM of decalcified bone specimens.
Fig. 4: Label-free UV-PAM for identifying tumours in decalcified bone fragments.
Fig. 5: Label-free UV-PAM of undecalcified bone specimen and H&E validation.
Fig. 6: Detailed network architecture for virtual staining.
Fig. 7: Label-free UV-PAM virtual histology of undecalcified bone via unsupervised deep learning.

Data availability

The main data supporting the findings of this study are available within the paper and its Supplementary Information. The training dataset and the fake output images for the CycleGAN network are available at https://doi.org/10.5281/zenodo.6345772. The raw data generated during the study are too large to be publicly shared, yet they are available for research purposes from the corresponding authors on reasonable request.

Code availability

The original code for CycleGAN is available at https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix. We applied this code to our dataset with the customized settings described in Methods. MATLAB was used for creating image tiles for the network and for restitching the output image tiles. The quantitative analysis of photoacoustic virtual histology was done via QuPath (https://qupath.github.io). The system control software and the data collection software are proprietary and used in licensed technologies.

References

  1. Global Cancer Observatory (WHO, accessed 19 May 2021); http://gco.iarc.fr/today/home

  2. Wyld, L., Audisio, R. A. & Poston, G. J. The evolution of cancer surgery and future perspectives. Nat. Rev. Clin. Oncol. 12, 115–124 (2015).

    Article  Google Scholar 

  3. Sullivan, R. et al. Global cancer surgery: delivering safe, affordable, and timely cancer surgery. Lancet Oncol. 16, 1193–1224 (2015).

    Article  Google Scholar 

  4. Mahe, E. et al. Intraoperative pathology consultation: error, cause and impact. Can. J. Surg. 56, E13–E18 (2013).

    Article  Google Scholar 

  5. DiNardo, L. J., Lin, J., Karageorge, L. S. & Powers, C. N. Accuracy, utility, and cost of frozen section margins in head and neck cancer surgery. Laryngoscope 110, 1773–1776 (2000).

    Article  CAS  Google Scholar 

  6. Brender, E., Burke, A. & Glass, R. M. Frozen section biopsy. JAMA 294, 3200 (2005).

    Article  CAS  Google Scholar 

  7. Campanacci, M. Bone and Soft Tissue Tumors: Clinical Features, Imaging, Pathology and Treatment (Springer, 2013).

  8. Pathology and Genetics of Tumours of Soft Tissue and Bone (IARC, WHO, 2002).

  9. Taqi, S. A., Sami, S. A., Sami, L. B. & Zaki, S. A. A review of artifacts in histopathology. J. Oral Maxillofac. Pathol. 22, 279 (2018).

    Article  Google Scholar 

  10. Gomez-Brouchet, A. et al. Assessment of resection margins in bone sarcoma treated by neoadjuvant chemotherapy: literature review and guidelines of the bone group (GROUPOS) of the French sarcoma group and bone tumor study group (GSF-GETO/RESOS). Orthop. Traumatol. Surg. Res. 105, 773–780 (2019).

    Article  Google Scholar 

  11. Gareau, D. S. et al. Confocal mosaicing microscopy in Mohs skin excisions: feasibility of rapid surgical pathology. J. Biomed. Opt. 13, 054001 (2008).

    Article  Google Scholar 

  12. Wang, M. et al. Gigapixel surface imaging of radical prostatectomy specimens for comprehensive detection of cancer-positive surgical margins using structured illumination microscopy. Sci. Rep. 6, 27419 (2016).

    Article  CAS  Google Scholar 

  13. Wang, M. et al. High-resolution rapid diagnostic imaging of whole prostate biopsies using video-rate fluorescence structured illumination microscopy. Cancer Res. 75, 4032–4041 (2015).

    Article  CAS  Google Scholar 

  14. Glaser, A. K. et al. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. Nat. Biomed. Eng. 1, 0084 (2017).

    Article  Google Scholar 

  15. Fereidouni, F. et al. Microscopy with ultraviolet surface excitation for rapid slide-free histology. Nat. Biomed. Eng. 1, 957–966 (2017).

    Article  CAS  Google Scholar 

  16. Hollon, T. C. et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Nat. Med. 26, 52–58 (2020).

    Article  CAS  Google Scholar 

  17. Orringer, D. A. et al. Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy. Nat. Biomed. Eng. 1, 0027 (2017).

    Article  Google Scholar 

  18. Assayag, O. et al. Large field, high resolution full-field optical coherence tomography: a pre-clinical study of human breast tissue and cancer assessment. Technol. Cancer Res. Treat. 13, 455–468 (2014).

    PubMed  Google Scholar 

  19. Nguyen, F. T. et al. Intraoperative evaluation of breast tumor margins with optical coherence tomography. Cancer Res. 69, 8790–8796 (2009).

    Article  CAS  Google Scholar 

  20. Fereidouni, F., Tracy, J. & Levenson, R. M. M. D. MUSE microscopy for thick tissue imaging with extended depth of field. In Proc. SPIE 10489, Optical Biopsy XVI: Toward Real-Time Spectroscopic Imaging and Diagnosis 104890H (SPIE, 2018).

  21. Gambichler, T. et al. Comparison of histometric data obtained by optical coherence tomography and routine histology. J. Biomed. Opt. 10, 044008 (2005).

    Article  Google Scholar 

  22. Wang, L. V. & Hu, S. Photoacoustic tomography: in vivo imaging from organelles to organs. Science 335, 1458–1462 (2012).

    Article  CAS  Google Scholar 

  23. Wang, L. V. & Yao, J. A practical guide to photoacoustic tomography in the life sciences. Nat. Methods 13, 627–638 (2016).

    Article  CAS  Google Scholar 

  24. Danielli, A. et al. Label-free photoacoustic nanoscopy. J. Biomed. Opt 19, 086006 (2014).

    Article  Google Scholar 

  25. Shi, J., Tang, Y. & Yao, J. Advances in super-resolution photoacoustic imaging. Quant. Imaging Med. Surg. 8, 724–732 (2018).

    Article  Google Scholar 

  26. Yao, J., Wang, L., Li, C., Zhang, C. & Wang, L. V. Photoimprint photoacoustic microscopy for three-dimensional label-free subdiffraction imaging. Phys. Rev. Lett. 112, 014302 (2014).

    Article  Google Scholar 

  27. Yao, J. et al. High-speed label-free functional photoacoustic microscopy of mouse brain in action. Nat. Methods 12, 407–410 (2015).

    Article  CAS  Google Scholar 

  28. Li, L. et al. Single-impulse panoramic photoacoustic computed tomography of small-animal whole-body dynamics at high spatiotemporal resolution. Nat. Biomed. Eng. 1, 0071 (2017).

    Article  Google Scholar 

  29. Wong, T. T. W. et al. Fast label-free multilayered histology-like imaging of human breast cancer by photoacoustic microscopy. Sci. Adv. 3, e1602168 (2017).

    Article  Google Scholar 

  30. Zhang, C., Zhang, Y. S., Yao, D.-K., Xia, Y. & Wang, L. V. Label-free photoacoustic microscopy of cytochromes. J. Biomed. Opt. 18, 020504 (2013).

    Article  Google Scholar 

  31. Xu, Z., Li, C. & Wang, L. V. Photoacoustic tomography of water in phantoms and tissue. J. Biomed. Opt. 15, 036019 (2010).

    Article  Google Scholar 

  32. Wong, T. T. W. et al. Label-free automated three-dimensional imaging of whole organs by microtomy-assisted photoacoustic microscopy. Nat. Commun. 8, 1386 (2017).

    Article  Google Scholar 

  33. Shi, J. et al. High-resolution, high-contrast mid-infrared imaging of fresh biological samples with ultraviolet-localized photoacoustic microscopy. Nat. Photonics 13, 609–615 (2019).

    Article  CAS  Google Scholar 

  34. Tschuchnig, M. E., Oostingh, G. J. & Gadermayr, M. Generative adversarial networks in digital pathology: a survey on trends and future potential. Patterns 1, 100089 (2020).

    Article  Google Scholar 

  35. Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).

    Article  CAS  Google Scholar 

  36. Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision 2223–2232 (2017).

  37. Lahiani, A. et al. Virtualization of tissue staining in digital pathology using an unsupervised deep learning approach. In European Congress on Digital Pathology 47–55 (Springer, 2019).

  38. Yao, D.-K., Chen, R., Maslov, K., Zhou, Q. & Wang, L. V. Optimal ultraviolet wavelength for in vivo photoacoustic imaging of cell nuclei. J. Biomed. Opt. 17, 056004 (2012).

    Article  Google Scholar 

  39. Li, X., Kang, L., Zhang, Y. & Wong, T. T. W. High-speed label-free ultraviolet photoacoustic microscopy for histology-like imaging of unprocessed biological tissues. Opt. Lett. 45, 5401–5404 (2020).

    Article  Google Scholar 

  40. Imai, T. et al. High-throughput ultraviolet photoacoustic microscopy with multifocal excitation. J. Biomed. Opt. 23, 036007 (2018).

    Article  Google Scholar 

  41. Li, B., Qin, H., Yang, S. & Xing, D. In vivo fast variable focus photoacoustic microscopy using an electrically tunable lens. Opt. Express 22, 20130–20137 (2014).

    Article  Google Scholar 

  42. Tang, M., Luo, F. & Liu, D. Automatic time gain compensation in ultrasound imaging system. In 2009 3rd International Conference on Bioinformatics and Biomedical Engineering 1–4 (IEEE, 2009); https://doi.org/10.1109/ICBBE.2009.5162432

  43. Xu, Z. et al. Cortex-wide multiparametric photoacoustic microscopy based on real-time contour scanning. Neurophotonics 6, 035012 (2019).

    Article  CAS  Google Scholar 

  44. Mao, X. et al. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision 2794–2802 (2017).

  45. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition 770–778 (2016).

  46. Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 1125–1134 (2017).

  47. Zhang, R. Making convolutional networks shift-invariant again. In International conference on machine learning 7324–7334 (2019).

  48. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://doi.org/10.48550/arXiv.1412.6980 (2017).

  49. Humphries, M. P., Maxwell, P. & Salto-Tellez, M. QuPath: the global impact of an open source digital pathology system. Comput. Struct. Biotechnol. J. 19, 852–859 (2021).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank M. D’Apuzzo for helpful discussions and valuable pathological feedback. This work was sponsored by the United States National Institutes of Health (NIH) grants R01 CA186567 (NIH Director’s Transformative Research Award), R35 CA220436 (Outstanding Investigator Award) and R01 EB028277A.

Author information

Authors and Affiliations

Authors

Contributions

R.C., B.C. and L.V.W. designed the experiment. R.C. and Y.Z. built the system and wrote the control programme. R.C. performed the experiment. S.D.N., B.C. and Y. Liang provided bone specimens and H&E slices. R.C., S.D. and Y. Luo performed image processing. L.V.W. and B.C. supervised the project. All authors were involved in discussions during the work and in the preparation of the manuscript.

Corresponding authors

Correspondence to Brooke Crawford or Lihong V. Wang.

Ethics declarations

Competing interests

L.V.W. has a financial interest in MicroPhotoAcoustics, CalPACT and Union Photoacoustic Technologies. (However, these companies did not financially support this work.) The other authors declare no competing interests.

Peer review

Peer review information

Nature Biomedical Engineering thanks Ji-Xin Cheng, Carolin Mogler and Ashok Veeraraghavan for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary figures and tables.

Reporting Summary

Supplementary video 1

Side-by-side comparisons of greyscale UV-PAM and H&E images.

Supplementary video 2

Side-by-side comparisons of the virtually stained UV-PAM images and the corresponding H&E images in Fig. 7a,b.

Supplementary video 3

Side-by-side comparisons of the virtually stained UV-PAM images and the corresponding H&E images in Fig. 7c,d.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cao, R., Nelson, S.D., Davis, S. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat. Biomed. Eng (2022). https://doi.org/10.1038/s41551-022-00940-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41551-022-00940-z

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing