Introduction

Machine-learning is popular for the prediction, classification and assessment of liver fibrosis/cirrhosis, and response to therapy in hepatitis B and C patients1,2,3. At present, histopathological examination of liver biopsy samples remains as the ‘gold standard’ for liver fibrosis assessment. Histopathological features such as excessive accumulation of extracellular matrix (particularly collagen) and/or parenchymal extinction4 are incorporated into descriptive or semi-quantitative scoring systems for disease staging. Examples include the Knodell histological activity index (HAI)5, Scheuer6, Ishak1, Metavir7, and Ishak-modified systems8. However, these scoring systems are inherently semi-quantitative and subject to intra- and inter-observer variability.

To overcome these limitations, we and others have previously developed systems using machine learning-based algorithms to build diagnostic tools for improved liver fibrosis quantification and scoring9,10,11. These algorithms include artificial neural networks (ANN), multinomial logistic regression (MLR), support vector machines (SVM), and random forests (RF). However, the relative accuracy of these algorithms for liver fibrosis assessment and scoring remains unknown. Furthermore, besides having a limited number of extracted features, these algorithms require manual segmentation and feature extraction from the images prior to modeling and classification12. Therefore, we sought to develop fully automated algorithms for staging liver fibrosis.

Deep learning-based algorithms provide a powerful framework for automatic feature generation and image classification using magnetic resonance imaging (MRI) and computerized tomography (CT) images on liver fibrosis13,14. Such algorithms eliminate the need for manual segmentation and feature extraction from the images. However, they demand the use of a large training dataset and biopsy sample images typically do not meet this need. Based on transfer learning being a variant of the typical deep learning-based algorithms - in that the neural network is pre-trained by a very large number of training datasets worldwide using weakly or even irrelevant image sources15,16 - we hypothesized that the pre-trained deep learning neural network in transfer learning approach can accurately stage liver fibrosis in a fully automated manner. Here, we validated our hypothesis by inheriting and adapting the most studied seven-layered AlexNet17 algorithm for computer-aided liver fibrosis scoring. We compared the accuracy of this deep learning-based algorithm with other semi-automated machine learning algorithms – ANN, MLR, SVM and RF. Validation results showed that the deep learning-based algorithm by transfer learning approach can fully automate the scoring of liver fibrosis stages, with accuracy similar to conventional ANN, non-linear MLR, linear SVM and feature-ranking based RF algorithms. It is potentially a powerful tool that can aid disease diagnosis even with limited numbers of biopsy samples or clinical information.

Materials and Methods

Tissue preparation

Male Wistar rats (average weight of 220 g) were housed 2 per cage in the Biological Resource Centre (BRC) of Biopolis, Agency for Science, Technology and Research (A*STAR) with free access to laboratory chow and water in a 12:12 h light/dark schedule. The Institutional Animal Care and Use Committee (IACUC, Biological Resource Centre, A*STAR) approved all animal-related experiments and all methods were performed in accordance with the relevant guidelines and regulations. A total of 25 rats were randomly separated into a group of 21 for thioacetamide (TAA) treatment, and a group of 4 as controls. The 21 rats in the TAA-treated group were sacrificed at time-points of 4, 6, 7, 8, 10 and 12 weeks. Another 4 rats in the control group were sacrificed at week 0 without treatment. Cardiac perfusion with 4% paraformaldehyde was performed to flush out blood cells and the liver was fixed before harvesting.

Collagen staining and pathological scoring

4 biopsy samples were taken from each rat liver. They were formalin-fixed, paraffin-embedded, cut into 5 µm sections and stained for histological assessment. A total of 100 slides from 25 paraffin blocks were stained with Sirius Red (SR) stain kit (Picro Sirius Red Stain Kit, ab150681, Abcam). Each SR-stained sample was blinded and scored by a pathologist to reduce any bias using the Metavir scoring system7. In the Metavir system, liver fibrosis was classified into five stages from F0 to F4 according to the severity of fibrosis: no fibrosis, fibrous portal expansion, few bridges or septa, numerous bridges or septa, and cirrhosis18. The number of samples that were categorized into each stage was: 20 F0, 20 F1, 20 F2, 20 F3 and 20 F4.

Image acquisition

Another 4 paired biopsy samples were taken from the previously processed 25 paraffin blocks for imaging using Second Harmonic Generation (SHG) microscopy. A total of 100 paired slides from 25 paraffin blocks were used and SHG images were acquired using the Olympus IX81 system. The laser was tuned to 810 nm to excite the samples and SHG signals were recorded at 405 nm using a 20X objective lens. A nine-by-seven multi-tile image was acquired for every slide with a final image size of 12 mm2 (4 × 3 mm).

Image pre-processing for collagen content

All raw images acquired from SHG microscopy were imported into the MATLAB Image Processing Toolbox (Mathworks) for further processing. The resulting gray-scale images were first adjusted by contrast enhancement19 and then transformed into binary images using adaptive-thresholding9,10,11. To fully reproduce the distribution of collagen fibers and eliminate any background signal, morphological closing was then performed to smooth the binary mask of the collagen and segments with less than 5 pixels were removed9,10,11.

Feature extraction and quantification for collagen content

Total collagen content and previously described collagen features, including collagen fiber morphology20,21 and collagen fiber connectivity-related texture measurements22, were investigated. A total of 21 morphological features specific to tubular-shaped objects including length, width, orientation, cross-link spaces and cross-link density, etc. and 109 textural features of both gray-level co-occurrence matrix (GLCM) and transform-based pattern including contrast, correlation, average, variation, entropy etc. from the occurrence matrix and Fourier, Gabor, and wavelet transformation were extracted from the processed SHG images.

Model construction for deep learning-based algorithms

For deep learning, a pre-trained AlexNet-CNN network was used for training and testing purposes. This network was made up of 1 input layer, 7 hidden layers (5 convolution layers and 2 fully connected layers) and 1 output layer using batch stochastic gradient descent, with specific values for momentum and weight decay17. The input and output layers of the original AlexNet-CNN network were replaced accordingly for liver fibrosis assessment as previously reported15,16, where the processed SHG images were first resized and duplicated to 224 × 224 × 3 pixels to fit into the model as input images. There were 2 max pooling layers of size 2 × 2 pixels after the first and second convolution layer where the size and number of filters are 11 × 11 × 3 pixels, 96 and 5 × 5 × 96 pixels, 256, respectively. Another 3 convolution layers where the size and number of filters are 3 × 3 × 256 pixels, 384 and 3 × 3 × 384 pixels, 384 and 3 × 3 × 384 pixels, 256, respectively were also implemented before the third max pooling layer. Layers 6 and 7 are 4096-dimension fully connected layers where the input matrix is transformed into a vector for Softmax activation function through General MATRIX Vector Multiply (GEMV) approach. The final output layer contained 5 possible outcomes corresponding to the 5 liver fibrosis stages from F0 to F4 using the Metavir scoring.

Model construction for non-deep learning-based algorithms

ANN, MLR, SVM and RF-based algorithms were applied for the training and testing datasets with pre-processed images and selected features to test the sample classification. The leave-one-out cross validation rule was applied.

For ANN, various combinations of layers and nodes a) 1 hidden layer with 10 nodes; b) 1 hidden layer with 20 nodes and c) 2 hidden layers with 20 nodes each and feed-forward neural network were established for the analysis using a back-propagation orientated gradient descent training approach to optimize the loss.

For MLR, the coefficients of the predicted probability were calculated using the logistic function, g, the logit (log odds):

$${\rm{g}}({\rm{p}}({\rm{x}}))=\,\mathrm{ln}(\frac{p(x)}{1-p(x)})={\beta }_{0}+{\beta }_{1}{x}_{1}+\ldots +{\beta }_{n}{x}_{n}$$

The coefficients \({\beta }_{0}\), \({\beta }_{1}\), …\({\beta }_{n}\) were estimated with the maximum likelihood method.

For SVM, a hyperplane was constructed for optimally and correctly classifying the images into fibrosis stages based on the feature values in the m-dimensional space. The radial basis function (RBF) kernel was used as the distance measurement between adjacent subject vectors \({x}_{1}\) and \({x}_{2}\), i.e.,

$${\rm{K}}({x}_{1},{x}_{2})=\exp (-\frac{({x}_{1}-{{x}_{2})}^{2}}{2{\sigma }^{2}})$$

\(\sigma \) controls the width of the kernel.

For RF, the training algorithm applies the general techniques of bootstrap aggregating, or bagging, to trees learners where 100 decision tree bagging with random sample replacement was used to generate the classification results and the majority vote in the case of classification trees was used for final predictions. The core algorithm for building decision trees called ID3 by J. R. Quinlan employs a top-down, greedy search through the space of possible branches with no backtracking. ID3 uses Entropy and Information Gain to construct a decision tree23.

Statistical analysis

Kruskal-Wallis (KW) tests were used to assess the performance of various classification models for their prediction capabilities in SPSS (IBM Corporation). Post-hoc analysis was performed to verify significant KW results and compare the deep learning method against other non-deep learning methods using Wilcoxon–Mann–Whitney tests and Bonferroni correction methods. Since there were 5 classification models for each comparison, 10 comparisons using Wilcoxon–Mann–Whitney test were generated. According to Bonferroni correction, the adjusted critical p-value for 0.05 significance would be 0.05 divided by the total number of comparisons, or 0.005 (0.05/10). Thus, the acquired p-values by Wilcoxon–Mann–Whitney tests were compared with the adjusted critical p-value 0.005 and the p < 0.005 criterion was used.

Results

Correlation of SHG microscopy-based quantification of collagen distribution with pathological scoring

To quantify collagen distribution in liver fibrosis of different stages, we acquired 100 liver section images of TAA-induced fibrotic rat livers using SHG microscopy. Liver samples were also stained with Sirius Red (SR) for histopathological review and scored by a pathologist (Table S1) using the METAVIR scoring system (Fig. 1). Qualitatively, SHG microscopy yielded images with higher collagen contrast as compared to the SR-stained images. To monitor collagen distribution during disease progression, we performed image processing for collagen tracing. The image-processing algorithm reinforces the collagen signal by using adaptive thresholding on the gray scale image, identifying morphological/textural changes in the binary image and removing noise by multiplying the tissue binary image. Figure 2 shows representative original and processed SHG images compared to SR-stained images at various stages of liver fibrosis.

Figure 1
figure 1

Schematic of study outline. Tissues were fixed, dehydrated, embedded in paraffin and sectioned into 2 consecutive slides. One was stained with Sirius Red (SR) for histological assessment by a pathologist using a conventional bright field microscope. The other was imaged using second harmonic generation (SHG) microscopy to generate images for subsequent use in developing a computer-aided classification model. TRN: training group, VAL: validation group, TST: testing group.

Full size image
Figure 2
figure 2

Representative images of SR-stained samples, SHG original images and SHG processed images at various stages of liver fibrosis.

Full size image

Development for various supervised classification models

Based on the pathological scoring of the METAVIR fibrosis stages, we used different supervised machine learning-based algorithms to generate classification models, namely deep learning CNN, ANN, MLR, SVM, and RF models (Fig. 3). All samples were investigated using a leave-one-out (LOO) rule for validation purposes due to the limited number of samples in the dataset. For deep learning-based algorithms, automatic feature extraction was performed during the convolution and max pooling process. For ANN, MLR, SVM and RF-based algorithms, we quantitatively measured the extent of liver fibrosis by examining the morphology and textural features of collagen fibers. 21 morphological features (Features 1–21) and 109 textural features [based on entropy (Feature 22), gray-level co-occurrence matrix (GLCM) (Features 23–34), Fourier’s transform (Features 35–40), wavelet decomposition (Features 41–100) and magnitude of convolution (Features 101–130)] were analyzed to characterize collagen quantity and distribution pattern. Table 1 lists the features that were extracted from the SHG-generated images. Morphological features (Features 1–21) were first used for model fitting. Subsequently, textural features (Features 22–130) were used for further analysis. Finally, morphological and textural features (Features 1–130) were combined. As expected, with more and some less important features being added into the model, some of the algorithms failed to correctly classify the various stages of fibrosis progression such as MLR and SVM. This is because an increase in the number of extracted features being incorporated into the model typically decrease the diagnostic power of those linear and simple classifiers without weighting on different features or feature selection step. The receiver operating characteristic (ROC) curve was used to assess the performance of the classification model with samples from the test cohort (Fig. 4). Details on the ROC curves and AUROC values can be found in Figs S1 and S2. Table 2 shows sensitivity, specificity, positive predictive values (PPV) and negative predictive values (NPV) for all classification models.

Figure 3
figure 3

The architecture and of (A) deep learning via convolutional neuron networks (CNN) using pretrained 7-layered AlexNet and non-deep learning via (B) Artificial Neural Networks (ANN), (C) Multinomial Logistic Regression (MLR), (D) Support Vector Machines (SVM) and Random Forests (RF) for computer aided liver fibrosis scoring. SHG processed images were resized and duplicated to be used as input for deep learning-based algorithm, C1-C5: convolution layer, FC6-FC7: fully connected layer. Extracted morphological or/and textural features were used as input for non-deep learning-based algorithms.

Full size image
Table 1 Extracted morphological and textural features of collagen fibers from SHG processed images.
Full size table
Figure 4
figure 4

The area under receiver operating characteristic (AUROC) values for various classification models from AlexNet-Convolutional Neuron Networks (CNN), conventional Artificial Neural Networks (ANN), non-linear Multinomial Logistic Regression (MLR), linear Support Vector Machines (SVM) and feature-ranking based Random Forests (RF) algorithms. AUROC was evaluated for CNN and ANN, MLR, SVM and RF with (A) morphological features (Features 1–21), (B) textural features (Features 21–130) and (C) both morphological and textural features (Feature 1–130). N.S.: non-significant difference, *adjusted p value is less than 0.005.

Full size image
Table 2 The performance evaluation for CNN, ANN, MLR, SVM and RF-based classification models.
Full size table

Performance of the deep learning (CNN and transfer learning)-based algorithm

For the transfer learning approach, we used a pre-trained deep learning neural network to stage the liver fibrosis samples (AUROC = 0.93 for Stage 0 vs. 1, 2, 3 and 4, AUTOC = 0.85 for Stage 0 and 1 vs. 2, 3 and 4, AUTOC = 0.95 for Stage 0, 1 and 2 vs. 3 and 4, AUTOC = 0.88 for Stage 0, 1, 2 and 3 vs. 4 (Fig. 4). We found significant differences between the transfer learning approach and the other simple learning algorithms (e.g. MLR and SVM), especially when textural features were included in the analysis. The deep learning-based algorithm enabled self-training using various combinations of filters/kernels. This learning approach was also employed in the 7-layered AlexNet-CNN algorithm during the network training process, resulting in faster and easier fine-tuning of the pre-trained network than from scratch.

Performance of the ANN-based algorithm

The ANN-based algorithm is a simplified version of the deep neural network. Next, we investigated the relationship between complexity of the neural network structure and accuracy of the classification model. The ANN-based algorithm gave rise to accurate staging of the four stages of liver fibrosis with the use of morphological features only (AUROC = 0.92 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.87 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 1.00 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.99 for Stage 0, 1, 2 and 3 vs. 4). It also accurately staged liver fibrosis using textural features (AUROC = 0.89 for Stage 0 vs. 1, 2 and 3 and 4, AUROC = 0.78 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.80 for Stage 0, 1, 2 and 3 vs. 4, except for Stage 0, 1 and 2 vs. 3 and 4 while AUROC = 0.50 for Stage 0, 1 and 2 vs. 3 and 4) or using both morphological and textural features (AUROC = 1.00 for Stage 0 vs. 1, 2 and 3 and 4, AUROC = 0.98 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 1.00 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.99 for Stage 0, 1, 2 and 3 vs. 4) (Fig. 4). Different weights were allocated to the various features depending on their contribution to model generation during the training process based on back propagation algorithms24. We also built multiple classification models using conventional ANN-based algorithms of 1 layer with 10 nodes and 2 layers with 20 nodes to compare against the standard test of 1 layer with 20 nodes. Details on the ROC curves and AUROC values can be found in Fig. S3.

Performance of the MLR-based algorithm

The MLR-based algorithm accurately scored the four stages of liver fibrosis with the use of morphological features only (AUROC = 0.94 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.89 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.95 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 1.00 for Stage 0, 1, 2 and 3 vs. 4). However, this algorithm failed to address the differences among various stages of liver fibrosis using textural features (AUROC = 0.51 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.63 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.68 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.73 for Stage 0, 1, 2 and 3 vs. 4) or both morphological and textural features (AUROC = 0.51 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.63 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.68 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.73 for Stage 0, 1, 2 and 3 vs. 4) (Fig. 4). This inaccurate staging of liver fibrosis using both morphological and textural features is because of the insufficient number of samples for model establishment and the large number of features that were being analyzed, resulting in the generation of a not-fully-ranked feature table that did not converge during model construction. This is a well-known problem for MLR-based classification models25. However, regularization or dimension reduction of feature numbers as well as use of the penalized method may overcome this problem.

Performance of the SVM-based algorithm

For the SVM-based algorithm, accuracy of the classification model significantly decreased when textural features were added into the analysis (AUROC = 0.69 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.61 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.55 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.68 for Stage 0, 1, 2 and 3 vs. 4) or when only textural features were used (AUROC = 0.51for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.53 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.53 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.68 for Stage 0, 1, 2 and 3 vs. 4), compared to the use of morphological features only (AUROC = 0.98 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.99 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.97 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.95 for Stage 0, 1, 2 and 3 vs. 4) (Fig. 4). In general, the larger the margin, the lower the generalization error of the classifier26 for the SVM-based algorithm. Intuitively, data points could be well separated by the hyper-plane that has the largest distance to the nearest training-data point of any class (so-called functional margin). New examples and features mapped into that same space may result in large margins of error and thus make it more difficult for an optimal hyper-plane to be established. This may explain the decrease in system performance when textural features were added into the analysis.

Performance of the RF-based algorithm

Similar to the ANN-based algorithm, the RF-based algorithm also resulted in accurate staging of liver fibrosis using morphological features only (AUROC = 0.98 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.92 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.99 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.98 for Stage 0, 1, 2 and 3 vs. 4). It could also accurately score all stages of liver fibrosis using either textural features only (AUROC = 0.97 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.79 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.94 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.89 for Stage 0, 1, 2 and 3 vs. 4) or both morphological and textural features (AUROC = 0.94 for Stage 0 vs. 1, 2, 3 and 4, AUROC = 0.85 for Stage 0 and 1 vs. 2, 3 and 4, AUROC = 0.98 for Stage 0, 1 and 2 vs. 3 and 4, AUROC = 0.98 for Stage 0, 1, 2 and 3 vs. 4) (Fig. 4). Within each RF, curvature or interaction tests for split-predictor selection was performed to calculate the importance of estimations against various features (predictors). For example, the median value of fiber length and ratio of short to long fibers were the two most prominent features for classification using morphological features only. The ratio of short to long fibers, median, and standard deviation of the magnitude of convolution over the image with Gabor filters at five scales were the three most important features for analysis using both morphological and textural features. In addition to generating highly accurate models, the RF-based algorithm could estimate the importance of each feature to the classification process (Fig. S4).

Discussion

Histopathological examination of stained liver biopsy samples remains as the gold standard for monitoring liver fibrosis progression21. However, inter- and intra-observer and staining variations may generate errors in disease staging27. With the development of both mode-locked lasers and highly sensitive optical sensors, non-linear optical microscopy (such as those based on multi-photon excitation fluorescence and multi-harmonic generation) has made stain-free imaging-based diagnosis a feasible approach28. With the generation of large amounts of data using these imaging modalities, several groups have explored the use of machine learning-based algorithms for scoring liver fibrosis stages10,11. However, the value of these algorithms remains unclear as they are usually case-specific and not fully automated, requiring manual segmentation and feature extraction. In this study, we demonstrate that deep learning-based algorithms can automatically quantify liver fibrosis progression in a TAA-induced fibrosis rat model and score different stages of liver fibrosis with high sensitivity and specificity. The fully automated deep learning-based algorithm exhibited the same level of accuracy in scoring liver fibrosis as other semi-automated algorithms such as ANN, MLR, SVM and RF.

Classification accuracy was found to be lower for scoring early stage liver fibrosis (Stage 0–1) compared to advanced fibrosis (Stage 2–4). The accuracy of all classification models (AUC = 0.85–1.00) using only morphological features was also found to be comparable to previously reported models (AUC = 0.80–0.95)11,22. For MLR and SVM-based algorithms, the AUC was very much lower as the system performance was affected by insignificant features. The addition of textural features did not make any difference to the accuracy of the classification model using conventional ANN and RF-based algorithms. Using our classification models, almost all fibrosis stage comparisons had high AUROC values (most were higher than 0.85), indicating that the features employed in this study were crucial indices and largely comparable to the parameters used in the clinically accepted METAVIR scoring system.

While conventional semi-automated learning algorithms are useful for liver fibrosis scoring, the deep learning-based approach is more promising as it automatically finds features and calculates the weight of each feature through their contribution, making it less tedious and more cost-effective. Inherently, large datasets and complex training neural networks are required for deep learning-based algorithms; however, this can be addressed via the use of transfer learning from weakly or even irrelevant image sources. We show that the most studied 7-layered AlexNet-based network using transfer learning was capable of dealing with a limited dataset, achieving the same level of accuracy as conventional semi-automated learning algorithms. Deep learning-based algorithms also enable the assessment of important features; this determines the minimally required set of features for the classification, hence reducing the number of parameters29. Future work will include validating these models against other scoring systems such as the Ishak staging and Beijing P-I-R classification30 for the analysis of intra-stage cirrhosis to patient samples and validate this approach by correlating with non-invasive imaging or blood test markers. We will also consider other types of deep learning neural networks besides DCNN to find whether better performance can be achieved.

Conclusion

Comparing different machine learning-based algorithms, we demonstrate that a deep learning-based algorithm using pre-trained AlexNet-CNN can automatically score liver fibrosis stages with a level of accuracy similar to conventional ANN, non-linear MLR and linear SVM, and feature ranking-based RF algorithms. A transfer learning approach using weakly or even irrelevant image sources may also help to address the requirement of large datasets for deep learning-based algorithms. This computer-aided and fully automated quantification of scoring liver fibrosis stages can be generalized to the design of high performance classification systems for other medical imaging tasks.