Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Deep-learning-based accurate hepatic steatosis quantification for histological assessment of liver biopsies

This article has been updated

Abstract

Hepatic steatosis droplet quantification with histology biopsies has high clinical significance for risk stratification and management of patients with fatty liver diseases and in the decision to use donor livers for transplantation. However, pathology reviewing processes, when conducted manually, are subject to a high inter- and intra-reader variability, due to the overwhelmingly large number and significantly varying appearance of steatosis instances. This process is challenging as there is a large number of overlapped steatosis droplets with either missing or weak boundaries. In this study, we propose a deep-learning-based region-boundary integrated network for precise steatosis quantification with whole slide liver histopathology images. The proposed model consists of two sequential steps: a region extraction and a boundary prediction module for foreground regions and steatosis boundary prediction, followed by an integrated prediction map generation. Missing steatosis boundaries are next recovered from the predicted map and assembled from adjacent image patches to generate results for the whole slide histopathology image. The resulting steatosis measures both at the pixel level and steatosis object-level present strong correlation with pathologist annotations, radiology readouts and clinical data. In addition, the segregated steatosis object count is shown as a promising alternative measure to the traditional metrics at the pixel level. These results suggest a high potential of artificial intelligence-assisted technology to enhance liver disease decision support using whole slide images.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: The DELINEATE model.
Fig. 2: Comparison of segmentation results.
Fig. 3: Visualization of segmented steatosis droplets in masks of distinct colors.
Fig. 4: DELINEATE correlations with manual histology assessment results, and radiology data derived features.
Fig. 5: Steatosis component quantification in whole liver tissue images.

Data availability

All source codes and annotation data related to this paper are available at GitHub [44]. We share image data in a public repository [45].

Change history

  • 17 July 2020

    The original record of this article has been updated to correct author name ‘Eduardo Castillo-Lion’ to ‘Eduardo Castillo-Leon’.

References

  1. 1.

    Tiniakos DG, Vos MB, Brunt EM. Nonalcoholic fatty liver disease: pathology and pathogenesis. Annu Rev Pathol. 2010;5:145–71.

    CAS  Article  Google Scholar 

  2. 2.

    Brunt EM. Nonalcoholic fatty liver disease: pros and cons of histologic systems of evaluation. Int J Mol Sci. 2016;17:1–10.

    Article  Google Scholar 

  3. 3.

    Braun HA, Faasse SA, Vos MB. Advances in pediatric fatty liver disease: pathogenesis, diagnosis, and treatment. Gastroenterol Clinics N Am. 2018;47:949–68.

    Article  Google Scholar 

  4. 4.

    French MCSG, Bedossa P. Intraobserver and interobserver variations in liver biopsy interpretation in patients with chronic hepatitis C. Hepatology. 1994;20:15–20.

    Article  Google Scholar 

  5. 5.

    Al-Kofahi Y, Lassoued W, Lee W, Roysam B. Improved automatic detection and segmentation of cell nuclei in histopathology images. IEEE Trans Biomed Eng. 2009;57:841–52.

    Article  Google Scholar 

  6. 6.

    Xing F, Xie Y, Yang L. An automatic learning-based framework for robust nucleus segmentation. IEEE Trans Med Imaging. 2015;35:550–66.

    Article  Google Scholar 

  7. 7.

    Irshad H, Veillard A, Roux L, Racoceanu D. Methods for nuclei detection, segmentation, and classification in digital histopathology: a review—current status and future potential. IEEE Rev Biomed Eng. 2013;7:97–114.

    Article  Google Scholar 

  8. 8.

    Farris AB, Cohen C, Rogers TE, Smith GH. Whole slide imaging for analytical anatomic pathology and telepathology: practical applications today, promises, and perils. Arch Pathol Lab Med. 2017;141:542–50.

    CAS  Article  Google Scholar 

  9. 9.

    Arnould L, Cosyn JP, Rondelez L, Courtoy P. Automated morphometric estimation of liver steatosis. Microsc Acta Suppl. 1979;3:197–203.

  10. 10.

    Krizhevsky A, Sutskever I, Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Processing Syst. 2012;60:1097–105.

    Google Scholar 

  11. 11.

    Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations. 2015. https://arxiv.org/abs/1409.1556.

  12. 12.

    Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention: Springer, Cham; 2015. p. 234–41.

  13. 13.

    Dimitriou N, Arandjelović O, Caie PD. Deep learning for whole slide image analysis: an overview. Front Med. 2019;6:1–7.

    Article  Google Scholar 

  14. 14.

    Pan X, Li L, Yang H, Liu Z, Yang J, Zhao L, et al. Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing. 2017;229:88–99.

    Article  Google Scholar 

  15. 15.

    Li W, Manivannan S, Akbar S, Zhang J, Trucco E, McKenna SJ. Gland segmentation in colon histology images using hand-crafted features and convolutional neural networks. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI): IEEE; 2016. p. 1405–8.

  16. 16.

    Xu Y, Li Y, Wang Y, Liu M, Fan Y, Lai M, et al. Gland instance segmentation using deep multichannel neural networks. IEEE Trans Biomed Eng. 2017;64:2901–12.

    Article  Google Scholar 

  17. 17.

    Xu Y, Li Y, Liu M, Wang Y, Lai M, Eric I, et al. Gland instance segmentation by deep multichannel side supervision. In International Conference on Medical Image Computing and Computer-Assisted Intervention: Springer, Cham; 2016. p. 496–504.

  18. 18.

    Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015:3431–40.

  19. 19.

    Chen H, Qi X, Yu L, Dou Q, Qin J, Heng P-A. DCAN: Deep contour-aware networks for object instance segmentation from histology images. Med Image Anal. 2017;36:135–46.

    Article  Google Scholar 

  20. 20.

    Cui Y, Zhang G, Liu Z, Xiong Z, Hu J. A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images. Med Biol Eng Comput. 2019;57:2027–43.

    Article  Google Scholar 

  21. 21.

    Roth HR, Lu L, Farag A, Sohn A, Summers RM. Spatial aggregation of holistically-nested networks for automated pancreas segmentation. In International conference on medical image computing and computer-assisted intervention: Springer, Cham; 2016. p. 451–9.

  22. 22.

    Khoshdeli M, Parvin B. Deep learning models delineates multiple nuclear phenotypes in H&E stained histology sections. ArXiv. 2018;abs/1802.04427. https://arxiv.org/abs/1802.04427.

  23. 23.

    Kong J, Lee MJ, Bagci P, Sharma P, Martin D, Adsay NV, et al. Computer-based image analysis of liver steatosis with large-scale microscopy imagery and correlation with magnetic resonance imaging lipid analysis. In 2011 IEEE International Conference on Bioinformatics and Biomedicine: IEEE; 2011. p. 333–8.

  24. 24.

    Otsu N. A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybernet. 1979;9:62–6.

    Article  Google Scholar 

  25. 25.

    Altunbay D, Cigir C, Sokmensuer C, Gunduz-Demir C. Color graphs for automated cancer diagnosis and grading. IEEE Trans Biomed Eng. 2009;57:665–74.

    Article  Google Scholar 

  26. 26.

    Vo H, Kong J, Teng D, Liang Y, Aji A, Teodoro G, et al. MaReIA: a cloud MapReduce based high performance whole slide image analysis framework. Distrib Parallel Databases. 2019;37(2):251–72.

    Article  Google Scholar 

  27. 27.

    Roy M, Wang F, Teodoro G, Vos MB, Farris AB, Kong J. Segmentation of overlapped steatosis in whole-slide liver histopathology microscopy images. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC): IEEE; 2018. p. 810–3.

  28. 28.

    Goode A, Gilbert B, Harkes J, Jukic D, Satyanarayanan M. OpenSlide: a vendor-neutral software foundation for digital pathology. J Pathol Inform. 2013;4:27.

    Article  Google Scholar 

  29. 29.

    Reinhard E, Adhikhmin M, Gooch B, Shirley P. Color transfer between images. IEEE Comput Graph Appl. 2001;21:34–41.

    Article  Google Scholar 

  30. 30.

    Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations (ICLR). 2016. https://arxiv.org/abs/1511.07122.

  31. 31.

    Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 2010:249–56.

  32. 32.

    Kingma DP, Ba J. Adam: a method for stochastic optimization. In 3rd International Conference for Learning Representations. 2015. http://arxiv.org/abs/1412.6980.

  33. 33.

    Xie S, Tu Z. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision. 2015:1395–403.

  34. 34.

    Wen Q, Chang H, Parvin B. A Delaunay triangulation approach for segmenting clumps of nuclei. In 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro: IEEE; 2009. p. 9–12.

  35. 35.

    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16). 2016:265–83.

  36. 36.

    Keras. Available from: https://keras.io.

  37. 37.

    Powers DM. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. J Mach Learn Technol. 2011;2:37–63.

    Google Scholar 

  38. 38.

    Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell. 2017;40:834–48.

    Article  Google Scholar 

  39. 39.

    Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell. 1993;15:850–63.

    Article  Google Scholar 

  40. 40.

    Leica. Available from: https://www.leicabiosystems.com/digital-pathology/analyze/ihc/aperio-positive-pixel-count-algorithm/.

  41. 41.

    Lee MJ, Bagci P, Kong J, Vos MB, Sharma P, Kalb B, et al. Liver steatosis assessment: correlations among pathology, radiology, clinical data and automated image analysis software. Pathol Res Pract. 2013;209:371–9.

    Article  Google Scholar 

  42. 42.

    NASH CRN. Available from: https://jhuccs1.us/nash/default.asp.

  43. 43.

    Liquori GE, Calamita G, Cascella D, Mastrodonato M, Portincasa P, Ferri D. An innovative methodology for the automated morphometric and quantitative estimation of liver steatosis. Histol Histopathol. 2009;24:49–60.

    CAS  PubMed  Google Scholar 

  44. 44.

    Roy M, Wang F, Vo H, Teodoro G, Farris AB, Lion EC, et al. Data from “DELINEATE” Github. 2020. https://github.com/StonyBrookDB/DELINEATE.

  45. 45.

    Roy M. Steatosis_Dataset_Raw 2020. https://doi.org/10.6084/m9.figshare.12439886.v1.

Download references

Acknowledgements

This research is supported in part by grants from National Institute of Health 7K25CA181503, 1U01CA242936, 5R01EY028450, and National Science Foundation ACI 1443054 and IIS 1350885.

Author information

Affiliations

Authors

Contributions

MR, FW, ABF, MBV and JK designed research; MR, HV, DT, GT, ABF, ECL and JK performed research; MR, ECL and JK analyzed data; MR, FW, ABF, ECL, and JK wrote the paper.

Corresponding authors

Correspondence to Fusheng Wang or Jun Kong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Roy, M., Wang, F., Vo, H. et al. Deep-learning-based accurate hepatic steatosis quantification for histological assessment of liver biopsies. Lab Invest 100, 1367–1383 (2020). https://doi.org/10.1038/s41374-020-0463-y

Download citation

Further reading

Search

Quick links