Comparison of the diagnostic efficacy of mathematical models in distinguishing ultrasound imaging of breast nodules

This study compared the diagnostic efficiency of benign and malignant breast nodules using ultrasonographic characteristics coupled with several machine-learning models, including logistic regression (Logistics), partial least squares discriminant analysis (PLS-DA), linear support vector machine (Linear SVM), linear discriminant analysis (LDA), K-nearest neighbor (KNN), artificial neural network (ANN) and random forest (RF). The clinical information and ultrasonographic characteristics of 926 female patients undergoing breast nodule surgery were collected and their relationships were analyzed using Pearson's correlation. The stepwise regression method was used for variable selection and the Monte Carlo cross-validation method was used to randomly divide these nodule cases into training and prediction sets. Our results showed that six independent variables could be used for building models, including age, background echotexture, shape, calcification, resistance index, and axillary lymph node. In the prediction set, Linear SVM had the highest diagnosis rate of benign nodules (0.881), and Logistics, ANN and LDA had the highest diagnosis rate of malignant nodules (0.910~0.912). The area under the ROC curve (AUC) of Linear SVM was the highest (0.890), followed by ANN (0.883), LDA (0.880), Logistics (0.878), RF (0.874), PLS-DA (0.866), and KNN (0.855), all of which were better than that of individual variances. On the whole, the diagnostic efficacy of Linear SVM was better than other methods.


Instruments and methods
The birth number, menarche and breast appearance (redness, swelling, and dimples) of patients were recorded.A thorough breast ultrasound was performed using ESAOTE MyLab Twice color Doppler ultrasonic diagnosis instrument with linear array high-frequency probe LA523 and the frequency of 3 ~ 12 MHz probe frequency.The initial preset conditions were as follows: imaging gain at 65%, dynamic range at 10, enhancement at 4, density at 1, depth at 44 mm, persistence at 6, dynamic compression at 3, and transducer resolution set to low (RES-L).In practice, the B-mode image is adjusted to incorporate the target lesion according to patient actual situation to achieve optimal resolution.In the color Doppler ultrasound mode, the sampling frame size was optimized to fully encompass the mass.The color gain was optimized, enabling the detection of low-velocity vascular flow within target lesions with minimal background noise.Ultrasound features included background echotexture, nodular size, shape, margin, internal echo, echo intensity, calcification, alder grade, resistance index and axillary lymph node.These ultrasound features for each nodule were graded according to the BI-RADS lexicon 9 .Background echotexture can be classified as either homogeneous or heterogeneous.Homogeneous background echotexture is defined by the predominant presence of parenchyma displaying a uniform hyperechoic appearance with minimal isoechoic or hypoechoic characteristics and less than 25% fibro glandular tissue 22 .Other background echotextures are defined as heterogeneous type.
Pathological diagnosis of nodules was taken as the dependent variable (Y), and the above ultrasound features and patient's clinical information were taken as independent variables (X).The assignment of these variables is shown in Table 1.

Statistical analysis
The relationship between ultrasonographic characteristics and clinical information (age, birth number, menarche, breast appearance and pathological type) was analyzed using Pearson's correlation, with P<0.05 indicating a significant correlation.Principal component analysis (PCA) was adopted to analyze the difference between nodule features in benign and malignant groups.Before establishing the models, the relevant variables were selected via the stepwise regression method, where the variable with a P-value < 0.05 was considered to have a significant relationship with the dependent variable 23 .In data analysis, 90% and 10% of 926 nodule cases were randomly divided into training and prediction sets, respectively, using the Monte Carlo cross-validation method 24,25 .In order to mitigate errors arising from a single calculation, Monte Carlo simulation was performed 100 times.The average diagnostic rate of each model for benign and malignant breast nodules in the training and prediction sets over 100 computations was separately computed.The diagnostic efficacy among models was compared by using the area under the receiver operating characteristic curves (ROC) in the prediction set.All the programs of these models were performed using MATLAB software.t-test was used to analyze the difference in diagnostic effectiveness among different models, with P<0.05 indicating a significant difference.

Ethics statement
This study was approved by the Institutional Review Committee of the First Affiliated Hospital of Nanjing Medical University (Approval number: 2022-SR-048) and patient informed consent was waived.This study retrospectively analyzed the ultrasonographic image of the patient's previous examination, which posed no potential risk or harm to the patient.The study was conducted with strict adherence to the Declaration of Helsinki.No patient privacy data was included in the data collection, and the data remained strictly confidential throughout the collection process.

Principal component analysis and variable selection
PCA enables the projection of samples from a high-dimensional space to a lower-dimensional space, revealing the spatial distribution characteristics among different samples in the data 26 .PCA demonstrated a certain degree of overlap in the distribution regions between benign and malignant nodule samples (Fig. 3).This overlap introduces an error rate in diagnosing benign and malignant breast nodules, which is consistent with the diagnosis results of individual ultrasonographic characteristics.Therefore, we could apply several mathematical models to analyze ultrasonographic characteristics and clinical information for diagnosing benign and malignant nodules.The stepwise regression method showed that 6 variances had a significant relationship with the dependent variable, including age (Coeff = 3.553, P < 0.01), background echotexture (Coeff = 0.887, P < 0.01), shape (Coeff = 1.835,P < 0.01), calcification (Coeff = 2.157, P < 0.01), resistance index (Coeff = 2.786, P < 0.01), and axillary lymph node (Coeff = 2.320, P < 0.01) (Fig. 4).

Model analysis
The model for each mathematical model was built based on the training set using the six variables selected by the stepwise regression method.After 100 random runs of Monte Carlo simulation, the diagnosis results of seven methods in the training and the prediction sets are shown in Table 2.In the training set, the diagnostic rates of all tested methods ranged from 0.849 to 0.999 for benign nodules and from 0.915 to 0.971 for malignant nodules, indicating that the diagnosis rate of all models was satisfactory.In the prediction set, the diagnostic rates of Logistics, PLS-DA, Linear SVM, RF, ANN, KNN and LDA, for benign nodules were 0.845, 0.833, 0.881, 0.850, 0.858, 0.846, and 0.851, respectively, and Linear SVM has the higher values than other methods (P < 0.05).
The diagnostic rates of Logistics, ANN and LDA for malignant nodules (ranging from 0.910 to 0.912) were the highest and the diagnostic rate of KNN was the lowest (0.865).Among these methods, only the diagnostic rate of RF and KNN in the prediction sets (ranging from 0.846 to 0.898) were much lower than that in train sets (> 0.947).The AUC value of Linear SVM was the highest (0.890), followed by ANN (0.883), LDA (0.880),

Discussion
Ultrasound is the primary method used to differentiate between benign and malignant breast nodules 27 .Our result indicated that patient's age, birth number, shape, resistance index, calcification, axillary lymph node, internal echo, margin, alder grade and background echotexture were significantly correlated with benign and malignant breast nodules.It has been reported that the breast nodules with irregular morphology, indistinct  www.nature.com/scientificreports/borders, hypoechoic pattern and suspicion of calcifications suggest malignancy 28 .Besides, there existed a significant correlation between nodule size and its pathology.In general, the larger the nodules are, the higher their degree of malignancy.However, it is important to note that nodules with smaller size are often overlooked by patients and sonographers.Neovascularization plays a crucial role in the onset, progression, invasion, and metastasis of breast cancer 29 .The higher the alder grade and resistance index of nodules were, the higher the possibility of malignancy, which should be paid attention to.In the BI-RADS classification, the probability of malignancy ranged from 2 to 95% for nodules in the 4 categories 30 .However, when a nodule is classified as grade 4, breast surgeons may recommend conducting additional examinations for patients, such as mammography, magnetic resonance imaging, or even core needle biopsy.These often increase the economic cost and psychological burden of patients.In addition, BI-RADS classification depends on the experience of physicians 31 .Our results showed that the AUC value of the test models ranged from 0.855 to 0.890, which was better than that of the BI-RADS classification based on junior and senior physicians (0.718-0.790 and 0.766-0.870,respectively) reported in the literature 32,33 .Our finding suggested that all seven models could effectively predict benign and malignant nodules, which could help doctors judge the malignant probability of nodules and reduce unnecessary examinations for patients 34 .
The stepwise regression method showed that age, background echotexture, shape, calcification, resistance index, and axillary lymph node had significant relationships with breast nodule pathology (P < 0.05), suggesting that these variables had a substantial contribution to the model.Using these relevant variables instead of all variables has the potential to enhance model performance, simplify the model, and avoid overfitting 35 .Among these models, Linear SVM had the highest diagnosis rate of benign breast nodules, and Logistics, ANN and LDA had the highest diagnosis rate of malignant breast nodules.Linear SVM could be recommended for diagnosing benign nodules, while Logistics, ANN and LDA could be recommended for diagnosing malignant ones.The diagnosis rate of the KNN and RF in the prediction set was significantly lower than that in the training set, which is likely attributed to model overfitting 36 .Compared with the training set, the results of the prediction set better reflect the performance of the two models.However, the overfitting of RF and KNN in the training set could limit their application in practical work.
Many reports indicate that artificial intelligence technology can classify images directly into benign and malignant categories by extracting feature variables such as color, contour, and texture [37][38][39] .However, these image feature variables extracted by artificial intelligence technology often lack clinical significance, posing significant limitations during the practical application of the model.In contrast, the clinical information and ultrasonographic characteristics of breast nodules were used for building our models and these variables hold clinical significance.This approach not only uncovers the relationship between breast nodule features and pathology, but also enhances the generalizability of these models.
In our study, these models were constructed using algorithm programs provided by MATLAB software, which could reduce the complexity of data analysis.Indeed, a certain level of programming knowledge is still required when using MATLAB software, particularly for modifying or rectifying inappropriate commands.Additionally, it is important to note that a single computation can result in biased estimates and the Monte Carlo cross-validation method could be employed to obtain robust statistical analysis results.
While mathematical models cannot fully substitute doctors, they can effectively aid in diagnosis.However, there are some limitations to the present study.For example, the present work relied on a limited sample of nodules for assessment.The assessment of controversial nodules by doctors involves subjectivity, and inaccurate judgment of nodule characteristics can further impact the diagnosis 40 .In future work, we will further incorporate more variables, including pathological types and ultrasound elasticity, to improve model diagnosis performance.In addition, we further used ultrasound images coupled with mathematical models to predict the presence of lymph node metastasis in malignant nodules.

Figure 2 .
Figure 2. Area under the ROC curve (AUC) of individual variances using Logistics.

Figure 3 .
Figure 3. PCA analysis of ultrasonographic characteristics in benign and malignant breast nodules.

Figure 4 .
Figure 4. Variance selection using the stepwise regression method.

Table 1 .
Information on the variable assignment.

Table 2 .
Diagnosis result of benign and malignant thyroid nodules using different mathematical models.* 95% confidence intervals in parentheses.