Deep learning enables automated scoring of liver fibrosis stages

Current liver fibrosis scoring by computer-assisted image analytics is not fully automated as it requires manual preprocessing (segmentation and feature extraction) typically based on domain knowledge in liver pathology. Deep learning-based algorithms can potentially classify these images without the need for preprocessing through learning from a large dataset of images. We investigated the performance of classification models built using a deep learning-based algorithm pre-trained using multiple sources of images to score liver fibrosis and compared them against conventional non-deep learning-based algorithms - artificial neural networks (ANN), multinomial logistic regression (MLR), support vector machines (SVM) and random forests (RF). Automated feature classification and fibrosis scoring were achieved by using a transfer learning-based deep learning network, AlexNet-Convolutional Neural Networks (CNN), with balanced area under receiver operating characteristic (AUROC) values of up to 0.85–0.95 versus ANN (AUROC of up to 0.87–1.00), MLR (AUROC of up to 0.73–1.00), SVM (AUROC of up to 0.69–0.99) and RF (AUROC of up to 0.94–0.99). Results indicate that a deep learning-based algorithm with transfer learning enables the construction of a fully automated and accurate prediction model for scoring liver fibrosis stages that is comparable to other conventional non-deep learning-based algorithms that are not fully automated.

To overcome these limitations, we and others have previously developed systems using machine learningbased algorithms to build diagnostic tools for improved liver fibrosis quantification and scoring [9][10][11] . These algorithms include artificial neural networks (ANN), multinomial logistic regression (MLR), support vector machines (SVM), and random forests (RF). However, the relative accuracy of these algorithms for liver fibrosis assessment and scoring remains unknown. Furthermore, besides having a limited number of extracted features, these algorithms require manual segmentation and feature extraction from the images prior to modeling and classification 12 . Therefore, we sought to develop fully automated algorithms for staging liver fibrosis.
Deep learning-based algorithms provide a powerful framework for automatic feature generation and image classification using magnetic resonance imaging (MRI) and computerized tomography (CT) images on liver fibrosis 13,14 . Such algorithms eliminate the need for manual segmentation and feature extraction from the images. However, they demand the use of a large training dataset and biopsy sample images typically do not meet this need. Based on transfer learning being a variant of the typical deep learning-based algorithms -in that the neural network is pre-trained by a very large number of training datasets worldwide using weakly or even irrelevant image sources 15, 16 -we hypothesized that the pre-trained deep learning neural network in transfer learning approach can accurately stage liver fibrosis in a fully automated manner. Here, we validated our hypothesis by inheriting and adapting the most studied seven-layered AlexNet 17 algorithm for computer-aided liver fibrosis scoring. We compared the accuracy of this deep learning-based algorithm with other semi-automated machine learning algorithms -ANN, MLR, SVM and RF. Validation results showed that the deep learning-based algorithm by transfer learning approach can fully automate the scoring of liver fibrosis stages, with accuracy similar to conventional ANN, non-linear MLR, linear SVM and feature-ranking based RF algorithms. It is potentially a powerful tool that can aid disease diagnosis even with limited numbers of biopsy samples or clinical information.

Materials and Methods
Tissue preparation. Male Wistar rats (average weight of 220 g) were housed 2 per cage in the Biological Resource Centre (BRC) of Biopolis, Agency for Science, Technology and Research (A*STAR) with free access to laboratory chow and water in a 12:12 h light/dark schedule. The Institutional Animal Care and Use Committee (IACUC, Biological Resource Centre, A*STAR) approved all animal-related experiments and all methods were performed in accordance with the relevant guidelines and regulations. A total of 25 rats were randomly separated into a group of 21 for thioacetamide (TAA) treatment, and a group of 4 as controls. The 21 rats in the TAA-treated group were sacrificed at time-points of 4, 6, 7, 8, 10 and 12 weeks. Another 4 rats in the control group were sacrificed at week 0 without treatment. Cardiac perfusion with 4% paraformaldehyde was performed to flush out blood cells and the liver was fixed before harvesting.
Collagen staining and pathological scoring. 4 biopsy samples were taken from each rat liver. They were formalin-fixed, paraffin-embedded, cut into 5 µm sections and stained for histological assessment. A total of 100 slides from 25 paraffin blocks were stained with Sirius Red (SR) stain kit (Picro Sirius Red Stain Kit, ab150681, Abcam). Each SR-stained sample was blinded and scored by a pathologist to reduce any bias using the Metavir scoring system 7 . In the Metavir system, liver fibrosis was classified into five stages from F0 to F4 according to the severity of fibrosis: no fibrosis, fibrous portal expansion, few bridges or septa, numerous bridges or septa, and cirrhosis 18 . The number of samples that were categorized into each stage was: 20 F0, 20 F1, 20 F2, 20 F3 and 20 F4. Image acquisition. Another 4 paired biopsy samples were taken from the previously processed 25 paraffin blocks for imaging using Second Harmonic Generation (SHG) microscopy. A total of 100 paired slides from 25 paraffin blocks were used and SHG images were acquired using the Olympus IX81 system. The laser was tuned to 810 nm to excite the samples and SHG signals were recorded at 405 nm using a 20X objective lens. A nine-by-seven multi-tile image was acquired for every slide with a final image size of 12 mm 2 (4 × 3 mm).
Image pre-processing for collagen content. All raw images acquired from SHG microscopy were imported into the MATLAB Image Processing Toolbox (Mathworks) for further processing. The resulting gray-scale images were first adjusted by contrast enhancement 19 and then transformed into binary images using adaptive-thresholding [9][10][11] . To fully reproduce the distribution of collagen fibers and eliminate any background signal, morphological closing was then performed to smooth the binary mask of the collagen and segments with less than 5 pixels were removed 9-11 . Feature extraction and quantification for collagen content. Total collagen content and previously described collagen features, including collagen fiber morphology 20,21 and collagen fiber connectivity-related texture measurements 22 , were investigated. A total of 21 morphological features specific to tubular-shaped objects including length, width, orientation, cross-link spaces and cross-link density, etc. and 109 textural features of both gray-level co-occurrence matrix (GLCM) and transform-based pattern including contrast, correlation, average, variation, entropy etc. from the occurrence matrix and Fourier, Gabor, and wavelet transformation were extracted from the processed SHG images.
Model construction for deep learning-based algorithms. For deep learning, a pre-trained AlexNet-CNN network was used for training and testing purposes. This network was made up of 1 input layer, 7 hidden layers (5 convolution layers and 2 fully connected layers) and 1 output layer using batch stochastic gradient descent, with specific values for momentum and weight decay 17 . The input and output layers of the original AlexNet-CNN network were replaced accordingly for liver fibrosis assessment as previously reported 15,16 , where the processed SHG images were first resized and duplicated to 224 × 224 × 3 pixels to fit into the model as input images. There were 2 max pooling layers of size 2 × 2 pixels after the first and second convolution layer where the size and number of filters are 11 × 11 × 3 pixels, 96 and 5 × 5 × 96 pixels, 256, respectively. Another 3 convolution layers where the size and number of filters are 3 × 3 × 256 pixels, 384 and 3 × 3 × 384 pixels, 384 and 3 × 3 × 384 pixels, 256, respectively were also implemented before the third max pooling layer. Layers 6 and 7 are 4096-dimension fully connected layers where the input matrix is transformed into a vector for Softmax activation function through General MATRIX Vector Multiply (GEMV) approach. The final output layer contained 5 possible outcomes corresponding to the 5 liver fibrosis stages from F0 to F4 using the Metavir scoring.
Model construction for non-deep learning-based algorithms. ANN, MLR, SVM and RF-based algorithms were applied for the training and testing datasets with pre-processed images and selected features to test the sample classification. The leave-one-out cross validation rule was applied. For ANN, various combinations of layers and nodes a) 1 hidden layer with 10 nodes; b) 1 hidden layer with 20 nodes and c) 2 hidden layers with 20 nodes each and feed-forward neural network were established for the analysis using a back-propagation orientated gradient descent training approach to optimize the loss.
For MLR, the coefficients of the predicted probability were calculated using the logistic function, g, the logit (log odds): The coefficients β 0 , β 1 , …β n were estimated with the maximum likelihood method. For SVM, a hyperplane was constructed for optimally and correctly classifying the images into fibrosis stages based on the feature values in the m-dimensional space. The radial basis function (RBF) kernel was used as the distance measurement between adjacent subject vectors x 1 and x 2 , i.e., σ controls the width of the kernel. For RF, the training algorithm applies the general techniques of bootstrap aggregating, or bagging, to trees learners where 100 decision tree bagging with random sample replacement was used to generate the classification results and the majority vote in the case of classification trees was used for final predictions. The core algorithm for building decision trees called ID3 by J. R. Quinlan employs a top-down, greedy search through the space of possible branches with no backtracking. ID3 uses Entropy and Information Gain to construct a decision tree 23 .

Statistical analysis.
Kruskal-Wallis (KW) tests were used to assess the performance of various classification models for their prediction capabilities in SPSS (IBM Corporation). Post-hoc analysis was performed to verify significant KW results and compare the deep learning method against other non-deep learning methods using Wilcoxon-Mann-Whitney tests and Bonferroni correction methods. Since there were 5 classification models for each comparison, 10 comparisons using Wilcoxon-Mann-Whitney test were generated. According to Bonferroni correction, the adjusted critical p-value for 0.05 significance would be 0.05 divided by the total number of comparisons, or 0.005 (0.05/10). Thus, the acquired p-values by Wilcoxon-Mann-Whitney tests were compared with the adjusted critical p-value 0.005 and the p < 0.005 criterion was used.

Results
Correlation of SHG microscopy-based quantification of collagen distribution with pathological scoring. To quantify collagen distribution in liver fibrosis of different stages, we acquired 100 liver section images of TAA-induced fibrotic rat livers using SHG microscopy. Liver samples were also stained with Sirius Red (SR) for histopathological review and scored by a pathologist (Table S1) using the METAVIR scoring system (Fig. 1). Qualitatively, SHG microscopy yielded images with higher collagen contrast as compared to the SR-stained images. To monitor collagen distribution during disease progression, we performed image processing for collagen tracing. The image-processing algorithm reinforces the collagen signal by using adaptive thresholding on the gray scale image, identifying morphological/textural changes in the binary image and removing noise by multiplying the tissue binary image. Figure 2 shows representative original and processed SHG images compared to SR-stained images at various stages of liver fibrosis. Development for various supervised classification models. Based on the pathological scoring of the METAVIR fibrosis stages, we used different supervised machine learning-based algorithms to generate classification models, namely deep learning CNN, ANN, MLR, SVM, and RF models (Fig. 3). All samples were investigated using a leave-one-out (LOO) rule for validation purposes due to the limited number of samples in the dataset. For deep learning-based algorithms, automatic feature extraction was performed during the convolution and max pooling process. For ANN, MLR, SVM and RF-based algorithms, we quantitatively measured the extent of liver fibrosis by examining the morphology and textural features of collagen fibers.  classify the various stages of fibrosis progression such as MLR and SVM. This is because an increase in the number of extracted features being incorporated into the model typically decrease the diagnostic power of those linear and simple classifiers without weighting on different features or feature selection step. The receiver operating characteristic (ROC) curve was used to assess the performance of the classification model with samples from the test cohort (Fig. 4). Details on the ROC curves and AUROC values can be found in Figs S1 and S2. Table 2 (Fig. 4). We found significant differences between the transfer learning approach and the other simple learning algorithms (e.g. MLR and SVM), especially when textural features were included in the analysis. The deep learning-based algorithm enabled self-training using various combinations of filters/kernels. This learning approach was also employed in the 7-layered AlexNet-CNN algorithm during the network training process, resulting in faster and easier fine-tuning of the pre-trained network than from scratch.
Performance of the ANN-based algorithm. The ANN-based algorithm is a simplified version of the deep neural network. Next, we investigated the relationship between complexity of the neural network structure and accuracy of the classification model. The ANN-based algorithm gave rise to accurate staging of the four   (Fig. 4).  Different weights were allocated to the various features depending on their contribution to model generation during the training process based on back propagation algorithms 24 . We also built multiple classification models using conventional ANN-based algorithms of 1 layer with 10 nodes and 2 layers with 20 nodes to compare against the standard test of 1 layer with 20 nodes. Details on the ROC curves and AUROC values can be found in Fig. S3.  (Fig. 4). This inaccurate staging of liver fibrosis using both morphological and textural features is because of the insufficient number of samples for model establishment and the large number of features that were being analyzed, resulting in the generation of a not-fully-ranked feature table that did not converge during model construction. This is a well-known problem for MLR-based classification models 25 . However, regularization or dimension reduction of feature numbers as well as use of the penalized method may overcome this problem.  (Fig. 4). In general, the larger the margin, the lower the generalization error of the classifier 26 for the SVM-based algorithm. Intuitively, data points could be well separated by the hyper-plane that has the largest distance to the nearest training-data point of any class (so-called functional margin). New examples and features mapped into that same space may result in large margins of error and thus make it more difficult for an optimal hyper-plane to be established. This may explain the decrease in system performance when textural features were added into the analysis.  (Fig. 4). Within each RF, curvature or interaction tests for split-predictor selection was performed to calculate the importance of estimations against various features (predictors). For example, the median value of fiber length and ratio of short to long fibers were the two most prominent features for classification using morphological features only. The ratio of short to long fibers, median, and standard deviation of the magnitude of convolution over the image with Gabor filters at five scales were the three most important features for analysis using both morphological and textural  features. In addition to generating highly accurate models, the RF-based algorithm could estimate the importance of each feature to the classification process (Fig. S4).

Performance of the SVM-based algorithm.
Discussion. Histopathological examination of stained liver biopsy samples remains as the gold standard for monitoring liver fibrosis progression 21 . However, inter-and intra-observer and staining variations may generate errors in disease staging 27 . With the development of both mode-locked lasers and highly sensitive optical sensors, non-linear optical microscopy (such as those based on multi-photon excitation fluorescence and multi-harmonic generation) has made stain-free imaging-based diagnosis a feasible approach 28 . With the generation of large amounts of data using these imaging modalities, several groups have explored the use of machine learning-based algorithms for scoring liver fibrosis stages 10,11 . However, the value of these algorithms remains unclear as they are usually case-specific and not fully automated, requiring manual segmentation and feature extraction. In this study, we demonstrate that deep learning-based algorithms can automatically quantify liver fibrosis progression in a TAA-induced fibrosis rat model and score different stages of liver fibrosis with high sensitivity and specificity. The fully automated deep learning-based algorithm exhibited the same level of accuracy in scoring liver fibrosis as other semi-automated algorithms such as ANN, MLR, SVM and RF. Classification accuracy was found to be lower for scoring early stage liver fibrosis (Stage 0-1) compared to advanced fibrosis (Stage 2-4). The accuracy of all classification models (AUC = 0.85-1.00) using only morphological features was also found to be comparable to previously reported models (AUC = 0.80-0.95) 11,22 . For MLR and SVM-based algorithms, the AUC was very much lower as the system performance was affected by insignificant features. The addition of textural features did not make any difference to the accuracy of the classification model using conventional ANN and RF-based algorithms. Using our classification models, almost all fibrosis stage comparisons had high AUROC values (most were higher than 0.85), indicating that the features employed in this study were crucial indices and largely comparable to the parameters used in the clinically accepted METAVIR scoring system.
While conventional semi-automated learning algorithms are useful for liver fibrosis scoring, the deep learning-based approach is more promising as it automatically finds features and calculates the weight of each feature through their contribution, making it less tedious and more cost-effective. Inherently, large datasets and complex training neural networks are required for deep learning-based algorithms; however, this can be addressed via the use of transfer learning from weakly or even irrelevant image sources. We show that the most studied 7-layered AlexNet-based network using transfer learning was capable of dealing with a limited dataset, achieving the same level of accuracy as conventional semi-automated learning algorithms. Deep learning-based algorithms also enable the assessment of important features; this determines the minimally required set of features for the classification, hence reducing the number of parameters 29 . Future work will include validating these models against other scoring systems such as the Ishak staging and Beijing P-I-R classification 30 for the analysis of intra-stage cirrhosis to patient samples and validate this approach by correlating with non-invasive imaging or blood test markers. We will also consider other types of deep learning neural networks besides DCNN to find whether better performance can be achieved.

Conclusion
Comparing different machine learning-based algorithms, we demonstrate that a deep learning-based algorithm using pre-trained AlexNet-CNN can automatically score liver fibrosis stages with a level of accuracy similar to conventional ANN, non-linear MLR and linear SVM, and feature ranking-based RF algorithms. A transfer learning approach using weakly or even irrelevant image sources may also help to address the requirement of large datasets for deep learning-based algorithms. This computer-aided and fully automated quantification of scoring liver fibrosis stages can be generalized to the design of high performance classification systems for other medical imaging tasks.