Multi-center study on predicting breast cancer lymph node status from core needle biopsy specimens using multi-modal and multi-instance deep learning

The objective of our study is to develop a deep learning model based on clinicopathological data and digital pathological image of core needle biopsy specimens for predicting breast cancer lymph node metastasis. We collected 3701 patients from the Fourth Hospital of Hebei Medical University and 190 patients from four medical centers in Hebei Province. Integrating clinicopathological data and image features build multi-modal and multi-instance (MMMI) deep learning model to obtain the final prediction. For predicting with or without lymph node metastasis, the AUC was 0.770, 0.709, 0.809 based on the clinicopathological features, WSI and MMMI, respectively. For predicting four classification of lymph node status (no metastasis, isolated tumor cells (ITCs), micrometastasis, and macrometastasis), the prediction based on clinicopathological features, WSI and MMMI were compared. The AUC for no metastasis was 0.770, 0.709, 0.809, respectively; ITCs were 0.619, 0.531, 0.634, respectively; micrometastasis were 0.636, 0.617, 0.691, respectively; and macrometastasis were 0.748, 0.691, 0.758, respectively. The MMMI model achieved the highest prediction accuracy. For prediction of different molecular types of breast cancer, MMMI demonstrated a better prediction accuracy for any type of lymph node status, especially in the molecular type of triple negative breast cancer (TNBC). In the external validation sets, MMMI also showed better prediction accuracy in the four classification, with AUC of 0.725, 0.757, 0.525, and 0.708, respectively. Finally, we developed a breast cancer lymph node metastasis prediction model based on a MMMI model. Through all cases tests, the results showed that the overall prediction ability was high.


Introduction
Breast cancer is the most prevalent malignant cancer among women worldwide [1] .Observing the occurrence of axillary lymph node (ALN) metastasis in breast cancer patients is not only important for prognosis, but also for clinical diagnosis and treatment decisions [2][3] .Sentinel lymph node (SLN) is the rst drainage site to experience the lymphatic spread of breast cancer.SLN biopsy (SLNB) is the standard method of ALN staging, which can guide clinicians in deciding axillary lymph node dissection (ALND), surgery, and follow-up treatment [4][5] .Preoperative prediction of lymph node status is critical for individualized treatment and for avoiding unnecessary surgery.Based on the idea of noninvasive prediction, several studies have attempted to utilize clinical predictors for establishing models to evaluate the possibility of SLN metastasis, and certain important prediction models have been developed [6][7] .
Deep learning has achieved progress and application in the medical eld [8][9][10][11] , yielding remarkable results in diagnosis and prognosis by automatically learning the latent features from medical data (i.e., histopathological images and clinical characteristics) [12][13][14][15][16] .For example, Cao et al. developed a deep learning model to predict the microsatellite instability status [17] .Meanwhile, clinical characteristics, which re ect the clinical status of the patients, are easier to obtain and have also been used by deep learning.Liang et al. applied a deep learning model to clinical characteristics at admission to predict the risk of COVID-19 patients developing critical illness [18] .Additionally, extensively utilizing various modal information has become an increasingly developed technology in the medical arti cial intelligence (AI) eld, and it has been widely demonstrated to be signi cantly useful [19][20][21] .
Multivariate logistic regression, tree-based methods, and shallow neural network-based methods have been used in previous studies to analyze clinical indicators [22][23][24] .Recent studies have shown that using tabular learning can effectively extract latent features and the interaction between previous methods on several tasks [25] .For histopathological image analysis, a weakly supervised method based on multiinstance learning can be used to learn latent features for subsequent analysis [26][27] .
Our research is based on the multi-modal prediction of clinicopathological indicators and pathological images, and utilizes all aspects of information that can be obtained before surgery.Therefore, we combined clinical pathological indicators with digital pathological images to establish a prediction model of breast cancer lymph node metastasis.This model performs a more comprehensive analysis for breast cancer, resulting in the improvement of the accuracy of clinical applications.

Patients characteristics
In this study, the clinicopathological data and corresponding digital pathological images of 3701 female breast cancer patients were enrolled, with a mean age of 53 years.Patients were divided into training set (2222 cases), validation set (736 cases) and test set (743 cases).Among 3701 patients, according to postoperative pathological results and con rmed by immunohistochemical results, 1953 patients had no lymph node metastasis, 118 patients were isolated tumor cells (ITCs), 564 were micrometastasis, and 1066 were macrometastasis.There was no signi cant difference in clinicopathological features among the three cohorts (P > 0.05) (Table 1).For histopathological images, we used multi-instance learning method to tile them into patches, and then learn the embedding for the bag-level feature.As for the clinicopathological parameters, we applied tabular learning model to learn the interaction between the features and abstract the nal representation of the tabular data by nonlinear combination of the features.After effective pre-processing, data cleaning and imputation, we developed a novel modal fusion module that aims at borrowing information from clinicopathological parameters to focus on discriminative patches in multi-instance learning of the histopathological images, and promoting the ow of complementary information between modalities through intermediate fusion (Fig. 1).

Predictive performance of lymph node metastasis model
The test set was used to test the prediction of lymph node status (no metastasis and metastasis) by deep learning model.The area under curve (AUC) of deep learning model for clinicopathological features was 0.770, WSI was 0.709, and MMMI was 0.809.MMMI developed by combining clinicopathological features with WSI showed a more accurate prediction effect for lymph node status prediction (Fig. 2).
In order to predict lymph node status more accurately and provide a more detailed basis for clinical decision, we classi ed lymph node status in more detail (no metastasis, ITCs, micrometastasis and macrometastasis).For metastasis-free, predicted by deep learning model of tabular, the AUC was 0.770 (95%CI: 0.737-0.804),accuracy was 0.723, sensitivity was 0.791, and speci city was 0.649.Predicted by deep learning of WSI, the AUC was 0.709 (95%CI: 0.672-0.746),accuracy was 0.669, sensitivity was 0.593, and speci city was 0.757.Predicted by MMMI, the AUC was 0.809 (95%CI: 0.779-0.840),accuracy was 0.751, sensitivity was 0.768, and speci city was 0.734.In contrast, MMMI demonstrated better prediction performance.The same results were found in ITCs, micrometastasis and macrometastasis (Table 2).Finally, no matter which kind of lymph node status was predicted, the prediction of MMMI was obviously better than that of single model based on clinicopathological features or digital pathological images.The ROC curves were shown in Fig. 3A-D.respectively.The AUC in TNBC were 0.895 (95% CI: 0.781-1), 0.968(95% CI: 0.905-1) and 0.75 (95% CI: 0.583-0.917),respectively.Due to the limitation of ITCs samples, the AUC results of ITCs were not obtained in the TNBC group.However, by comparing all results, we found that MMMI demonstrated a better prediction effect no matter which kind of lymph node status, especially in the molecular subtype of TNBC, and the ROC curve was shown in Fig. 4.

Feature importance analysis
We explored the feature importance by using MMMI.The analysis results showed that the characteristics of mitosis, glandular ducts and vascular invasion played an important role in predicting lymph node metastasis (Fig. 5).

Discussion
ALN metastasis of breast cancer not only determines the method of operation, but is also an important prognostic factor.Accurate prediction of lymph node metastasis in breast cancer patients can assist clinicians to develop axillary lymph node dissection, reduce postoperative complications, and improve prognosis.In AJCC, lymph node metastasis can be divided into ITCs (≤ 200 scattered tumor cells or tumor clusters ≤ 0.2mm), micrometastasis (tumor > 0.2 mm and ≤ 2 mm), and macrometastasis (tumor > 2 mm) [38] , according to the number of cancer cells in metastatic lymph nodes and the size of the tumor focus.Previous studies predicted ALN status from clinicopathological data, such as tumor grade, tumor size, lymphatic vascular invasion, etc.However, these studies only predicted the presence or absence of lymph node metastasis and could not distinguish between ITCs, micrometastases, or macrometastases.Moreover, the characteristics of the tumor micro-environment in pathological images cannot be described in words or quanti ed into clinicopathological indicators, and clinical application accuracy and external validation are insu cient.
Based on the idea of noninvasive prediction, several studies have attempted to use clinical predictors for establishing models to evaluate the possibility of SLN metastasis.Some previous studies have developed models for predicting ALN status.For instance, the most important prediction model is the Memorial Sloan-Kettering Cancer Center (MSKCC) [6] , which developed a nomogram to predict SLN metastasis.The ROC curve was 0.75, indicating an adequate level of prediction and discrimination.Liu, et al [39] adopted the smote-bagged-tree algorithm to establish a model for predicting SLN metastasis in early breast cancer patients.The ROC curve was 0.801, and the overall prediction ability was extremely high, indicating that the prediction model was accurate and stable.
Deep learning has gained increasing attention in the eld of medical imaging.Currently, deep convolutional neural networks (DCNNs) are one of the well-known types of deep learning algorithms.DCNNs are widely used in medical image processing and pattern recognition because of their simple structure and strong applicability, especially in imaging and pathology.In a previous study, researchers successfully developed a prediction model for lymph node metastasis in breast cancer patients using a deep learning neural network.The AUC of the CNN model with the best performance was 0.89.Additionally, the ROC performance of this model was better than that of the three experienced radiologists.These results demonstrated the feasibility of using CNNs to predict whether early primary breast cancer will metastasize and determine the feasibility of using deep learning methods to predict clinically negative ALN metastasis from ultrasound images in patients with primary breast cancer [40] .A deep learning radiography (DLR) method based on clinical parameters of breast conventional ultrasound (CUS) and shear wave elastography (SWE) images has been developed and veri ed [41] , which can be used to predict the ALN status of clinical T1 or T2 breast cancer patients before surgery.The differential diagnosis effect of this method on axillary negative (N0) and axillary metastasis (N+(≥ 1)) is better than that of the single method.Furthermore, the model indicated high discrimination between patients with low risk of axillary metastasis (N+(1-2)) and high risk (N+(≥ 3)).
In clinical practice, an increasing number of patients wish to understand the SLNs state before undergoing surgery.The prediction results obtained using these models are more reliable than simple clinical estimates.In our study, four classi cations of lymph node metastasis can be accurately predicted using preoperative multi-modal data, combined with clinicopathological indexes and pathological image features.For patients with different metastases, providing targeted surgical methods can avoid overtreatment and improve the quality of patients' lives.However, some information in HE stained slices, such as the tumor micro-environment, cannot be quanti ed in tables.Deep learning can be used to extract more information about the tumor micro-environment from pathological images.These two methods re ect the patients' information at different levels, and when combined, they provide a more comprehensive representation of the patient's condition and disease progression.There are currently several studies on lymph node metastasis in imaging that have obtained certain results.However, the detection of imaging focuses on macroscopic features, and it is easy to miss the detection of early small metastases such as micrometastases or ITCs.In this study, we predict whether lymph nodes have metastasis, and more speci cally, its status (no metastasis, ITCs, micrometastases, and macrometastases) and compared the prediction e ciency of the models.The results showed that the MMMI model had better prediction ability than the single model.We also veri ed the predictive ability of each molecular subtype and the results showed that MMMI could predict the lymph node status of each group, especially TNBC.Although a small number of cases with ITCs and micrometastases, MMMI has a satisfactory prediction ability.We will further expand the amount of data for increasing convince.
To test the applicability of the MMMI, we selected 190 cases for multi-center veri cation.For predicting the presence of lymph node metastasis, the AUC value was 0.6258.In addition, we tested the performance of the model for predicting no metastasis, ITCs, micrometastases, and macrometastases.

Patients
We collected the clinicopathological data and pathological images of preoperative core needle biopsy of 4038 female invasive breast cancer patients in the Fourth Hospital of Hebei Medical University from January 2015 to December 2018.Additionally, the clinicopathological data and whole slide imagin (WSIs) of 190 female invasive breast cancer patients from four medical centers in Hebei Province were collected for external validation of the proposed method.The inclusion criteria were as follows: 1) three experienced pathologists con rmed that all breast biopsy specimens were invasive breast cancer; 2) no neoadjuvant treatment (NAT) pre-operation was performed; 3) histopathology and immunohistochemistry were used to postoperatively con rm lymph node metastasis; and 4) complete clinical pathological data was obtained.The exclusion criteria were as follows: 1) microinvasive carcinoma (invasive lesions < 1 mm); 2) special types of invasive carcinoma; 3) poor/blurred scanned pathological image quality; 4) preoperative treatment (NAT, chemotherapy, radiotherapy and chemotherapy, ablation, etc.); and 5) incomplete clinical pathological data.Finally, 3701 patients were selected for this study.

Pathological evaluation
Histological grading was based on the World Health Organization classi cation of breast tumors (5th Edition) [28] and the Nottingham grading system.All cases were classi ed as grade I, grade II, or grade III.
TILs evaluation criteria: area occupied by mononuclear in ammatory cells over total stromal area [29][30] .More than 1% of positive tumor cell nuclei are considered hormone receptor-positive for ER and PR.IHC (Immunohistochemistry) score of 3 + or FISH (Fluorescence in situ hybridization) ampli cation was de ned HER2 positivity.All cases divided into three subtypes: luminal (hormone receptor-positive, Except for the low AUC value of micrometastases due to the number of cases, the other groups showed highly predictive performance.The performance of the model declined in external validation because of the differences in interpretation between different centers and the in uence of HE staining.The model can be improved by adding external data, unifying interpretation and marking, and optimizing the WSI.
This study has some limitations.There was no predictive veri cation of ITCs in TNBC due to the few cases of ITCs and uneven distribution, and an excessive AUC value was observed in other molecular classi cations.Although MMMI can predict lymph node metastasis more accurately than single clinicopathological factors or WSI features, it has a certain decline in the four classi cations of lymph node metastasis.In the future, we plan to optimize MMMI by increasing the sample size, adding other central sample data or gene test results to obtain more accurate and detailed prediction results of lymph node status.
including luminal A and luminal B), HER2 over-expression (hormone receptor negative, HER2 positive), and triple negative breast carcinoma (both hormone receptor and HER2 negative, TNBC).
3. Structure and standardization of the data.
Clinicopathological parameters were extracted from this report using a text pattern-matching algorithm.
For the categorical variables, the LabelEncoder function in the scikit-learn package was used to encode the target categorical variables into numerical variables.Thus, our algorithm generated structured data for each patient.Multivariate imputation via chained equations was applied to impute missing data [31] .
Color normalization was performed on all scales of histopathological images using an enhanced cycleconsistent generative adversarial network [32] .
The dataset was strati ed at the patient level and randomly divided into training (60%), validation (20%), and test (20%) sets.Given the gigantic size (typically 130,000 × 50,000 pixels) of a WSI, the WSIs were tiled into 512 × 512 patches in the form of a grid for subsequent processing.In this study, three magni cation scales (5×, 10×, and 20×) were explored, under which tiling was performed [33] .The threshold of overlap varied among different magni cations.Data augmentation was applied to the patches during the training process to improve the generalization.

Development, validation and interpretation of the model.
MIL-based representation of WSI.
Each WSI was tiled into patches, and the prediction of lymph node metastasisv (LNM) relies on the entire Region of Interest(ROI) of WSIs instead of individual patches [34] .E cientNet [35] pre-trained on the ImageNet dataset [36] was applied to extract patch-level features, and attention layers on the instance-level and feature-level were applied as the WSI modality network backbone.
Tabular learning-based representation of the clinicopathological parameters.
We adopted an attentive interpretable tabular learning network, TabNet [25] , to generate a representation of the clinicopathological parameters.The network employed sequential attention on features for inference in each decision step and learned the salient features from the structured clinicopathological parameters.
Integrating the representation of WSI and clinicopathological parameters.
Deep learning, as a form of representation learning, transforms raw data into a suitable representation for pattern recognition in speci c tasks [37] .We developed a new multi-modal multi-instance (MMMI) fusion module comprising multi-modal joint instance aggregate learning and global-aware instance aggregation.The representation of WSIs and clinicopathological parameters were input to the module and embedded as the global multi-modal feature, which was used to guide the learning process of each modality in turn.
Model training and testing.
Because WSIs in the MIL method have a variable patch number, the model was designed to accept different instance numbers as input.Label smoothing was used to prevent the model from learning the label-related bias.A weighted sampling method was applied to the distributed training to achieve an imbalanced distribution of samples across the four categories.The nal loss was computed as follows: where denotes the predicted likelihood from the model for sample , is the number of samples, is the number of candidate labels, and is a weight factor.In practice, is not dependent on data; thus, we set .
Feature importance.
Both MIL and tabular methods are based on the attention mechanism.We investigated the feature importance based on the learned weights of the instances in the MIL and the features of the clinicopathological parameters after the joint learning process.

Statistical analysis
The area under the receiver operating characteristic (ROC) curve was calculated using the pROC in R (version 3.6.1),and the Delong test was applied to compare ROC curves.Cutpointr was used to estimate the optimal cutoff points of the ROC curves.The Wilcoxon rank-sum test was used to compare the signatures.Pearson correlation coe cients were used for the correlation analysis.

Declarations
Funding This work was supported by the grant from the Beijing Jingjian Foundation for the Advancement of Pathology (No. 2019-0007).

Declaration of competing interest
There were no con ict of interest relevant to this article.
Author contributions   .The E cientNet35 pre-trained on ImageNet dataset36 was applied to extract features from each patch.All the patches inside one WSI can be combined as WSI-level representation.
Since the prediction on WSI using patch-level features can be formulated to a multi-instance learning problem, the patch-level feature vectors can be considered as instances, while WSI-level representation is a bag containing all the instances.Each scale can be processed in the same pipeline.b, Tabular modality pre-processing.Medical records and clinicoparameters were obtained from the hospital system and slide reading by experts.Both of them are in semi-structured natural language descriptions.A set of matching rules based on regular expression were applied to extract the structured information.Then we encoded the category variables and inputted the encoded table into a feature extractor (TabNet [25]) to generate tabular data representation.c, Supervised learning and gold standard labels.Then the fused crossmodality representation was processed by a classi cation network to produce the probability of nonmetastases, isolated tumor cells, micro-metastases, and macro-metastasis.This part was trained end-to-    Feature importance analysis Through comprehensive analysis of clinicopathological features, the weight of each factor in the prediction of lymph node metastasis was calculated.The results showed that pathological mitosis had the highest weight in breast cancer lymph node metastasis, and the other factors with higher weight were gland formation, ER, stoma changes, vascular invasion, Ki67, and TILs.
The ROC curve for external validation of lymph node metastatic status MMMI was used to predict lymph node metastasis using external data.A: No metastasis B: ITCs C: micrometastasis D: macrometastasis.In the external validation set, MMMI also achieved good results in predicting lymph node status, with AUC of 0.725, 0.757,0.525and 0.708, respectively.Except for the low AUC value of micrometastasis due to the number of cases, the other groups showed higher prediction performance.

Figures
Figures

Figure 1 Model
Figure 1 end and was supervised by the gold standard labels generated by expert diagnosis on IHC stained slide of lymph node after surgery.

Table 1
Patient and tumor characteristics of training set, validation set and test set Note: UOQ, upper outer quadrant;UIQ, upper inner quadrant; LOQ, lower outer quadrant; LIQ, lower inner quadrant.

1 K
Yueping Liu and Jianhua Yao conceived and designed the study.Chunhui Li, Yanan Wang, Xin Xu, Min Zhao, Meng Zhao, Meng Yue, Huiyan Deng, Huichai Yang collected the experiment data and literature.Ying Ding, Fan Yang, Mengxue Han and Yueping Liu wrote the manuscript, made the gures, edited, and made signi cant revisions to the manuscript.All authors read and approved the nal manuscript.Yan Ding, Fan Yang and Mengxue Han contributed equally to this article.Yueping Liu is corresponding author.