Development and validation of a machine learning-based prognostic risk stratification model for acute ischemic stroke

Acute ischemic stroke (AIS) is a most prevalent cause of serious long-term disability worldwide. Accurate prediction of stroke prognosis is highly valuable for effective intervention and treatment. As such, the present retrospective study aims to provide a reliable machine learning-based model for prognosis prediction in AIS patients. Data from AIS patients were collected retrospectively from the Second Affiliated Hospital of Xuzhou Medical University between August 2017 and July 2019. Independent prognostic factors were identified by univariate and multivariate logistic analysis and used to develop machine learning (ML) models. The ML model performance was assessed by area under the receiver operating characteristic curve (AUC) and radar plot. Shapley Additive explanations (SHAP) values were used to interpret the importance of all features included in the predictive model. A total of 677 AIS patients were included in the present study. Poor prognosis was observed in 209 patients (30.9%). Six variables, including neuron specific enolase (NSE), homocysteine (HCY), S-100β, dysphagia, C-reactive protein (CRP), and anticoagulation were included to establish ML models. Six different ML algorithms were tested, and Random Forest model was selected as the final predictive model with the greatest AUC of 0.908. Moreover, according to SHAP results, NSE impacted the predictive model the most, followed by HCY, S-100β, dysphagia, CRP and anticoagulation. Based on the RF model, an online tool was constructed to predict the prognosis of AIS patients and assist clinicians in optimizing patient treatment. The present study revealed that NSE, HCY, CRP, S-100β, anticoagulation, and dysphagia were important factors for poor prognosis in AIS patients. ML algorithms were used to develop predictive models for predicting the prognosis of AIS patients, with the RF model presenting the optimal performance.

number of features, outperforming traditional statistical methods 4,5 .Consequently, ML can be used to improve the prediction accuracy of prognosis.
ML algorithms using big data have improved and optimized the predictive performance of the prognosis prediction, as reported by prior literature [6][7][8] .Therefore, the goal in this retrospective study was to identify factors associated with prognosis for patients with stroke and develop an ML-based predictive model.Moreover, the proposed model was integrated in an online tool to provide clinicians and patients with visual and practical predictive assessment.

Materials and methods
Data source and collection.Retrospective data were obtained from electronic health record of the Second Affiliated Hospital of Xuzhou Medical University.Patients diagnosed with AIS from August 2017 to July 2019 were enrolled in this study cohort.Inclusion criteria were: The diagnosis of AIS met the requirements of the World Health Organization with symptom onset less than 24 h 9 .Exclusion criteria: (1) Incomplete clinical data.
(2) Patients with severe abnormal organ function.(3) Inadequate auxiliary examinations.(4) Follow-up less than one year.The diagnosis process of AIS was completed and confirmed independently by two physicians.In situations where there was diagnostic disagreement, the final diagnosis was reviewed with a senior physician to reach a consensus.This study was approved by the Ethics Committee of the Second Affiliated Hospital of Xuzhou Medical University, and all studies were conducted in accordance with relevant guidelines/regulations and the Declaration of Helsinki, and informed consent was obtained from all participants.
Data processing and variable selection.In the data cleaning process, the information of subjects with missing value was deleted, and normality tests were performed for the continuous numerical variables to assess the presence of outliers.Subsequently, the information of subjects with outliers was deleted.Ultimately, the current study included a total of 677 subjects in the database.The data were collected by trained and qualified members of the research group using a uniformly designed questionnaire, which included living area, occupation, education level, family economic status, eating habits (alcohol intake, drinking sugary drinks, smoking, high-fat diet, etc.), lifestyle and habits, disease medication, menstrual history (female) and psychosocial factors.First, the purpose and significance of the survey were explained to the patients.On the basis of patients fully understanding, informed consent forms were signed, and the patients were asked to read the guidance on the questionnaire carefully and fill it in.Patients with reading difficulties, such as illiteracy or poor eyesight, were aided by members of the research group and the contents of the questionnaire were read to them to help them fill it in.The type and severity of stroke were evaluated by clinical symptoms, head CT, MR, and angiography results.Blood lipid, blood glucose, homocysteine, and other data were obtained by laboratory tests.Blood pressure, weight, height, time of onset, and baseline National Institutes of Health Stroke Scale (NIHSS) score were also recorded.Drug use, stroke recurrence and clinical prognosis were recorded during follow-up.Dysphagia was assess by water swallowing test according to the criteria 10 .Clinical prognosis was assessed according to the modified Rankin Scale (mRS) according to the results of 1-year follow-up.The mRS ≥ 3 was classified as poor prognosis and mRS < 3 was classified as good prognosis.All detection indexes were completed by the Second Hospital of Xuzhou Medical University.The flowchart of data collection is shown in Fig. 1.
Binary categorical features were encoded with 0 and 1; for example, the gender of patients was encoded as 0 or 1 (0 = male, 1 = female).A total of 30 demographic and clinical variables were included as baseline variables Model development and performance evaluation.This dataset was randomly split into a training cohort (70%) and a validation cohort (30%).The training cohort was used to construct and perform crossvalidated ML models to avoid overfitting, and the validation database was used to validate the predictive power of models.
Based on the results of multivariate logistic analysis, the identified statistically significant variables were used to construct ML models.Six ML algorithms were employed to develop predictive models based on the training cohort, the performance of all candidate models was assessed using tenfold cross-validation within the training cohort, the final model was identified according to the highest mean AUC and the performance was further validated on the validation cohort.During tenfold cross-validation, the training cohort was divided into ten sets, nine of them were used for model testing and one for model assessment.Meanwhile, radar plots were drawn and the accuracy, sensitivity and specificity were calculated to evaluate performance.Due to the "black-box" nature of ML algorithms, the Shapley Additive explanations (SHAP) metric was used to interpret the models and assist doctors understand the findings of the models.The contribution of each variable to the model prediction was evaluated by SHAP values 11,12 .
Statistical analysis.Continuous and categorical variables were expressed as mean ± SD and frequency, respectively.P-Values < 0.05 were considered statistically significant with 95% confidence intervals (CIs) applied for all analyses.Statistical analyses of the demographic and clinical characteristics of all included patients were performed in R (version 4.0.5, HTTPS: //www.r-project.org/).Python (version 3.8) was used to develop ML predictive models and a web risk calculator.
Ethics approval and consent to participate.This study was approved by the Ethics Committee of the Second Affiliated Hospital of Xuzhou Medical University (ethics number: [2020] 081603), and all studies were conducted in accordance with relevant guidelines/regulations, informed consent was obtained from all participants, and all studies were conducted in accordance with the Declaration of Helsinki.

Results
Characteristics of study population.A total of 713 AIS patients were included in this study, and 36 patients with missing value were deleted.Finally, 677 patients diagnosed as AIS were enrolled in the present study, and 209 patients had poor prognosis (30.9%).The differences between AIS patients with good and poor prognosis are described in Table 1.No differences in stroke occurrence between age and gender groups were observed, and there were no significant differences in the location of stroke-associated arterial blood vessels.

Correlation of variables with clinical outcome.
A total of 474 patients were assigned to the training cohort (70% of the total population).In the training cohort, univariable regression was used to identify fifteen significant risk factors, including Systolic Blood Pressure (SBP), Diastolic Blood Pressure (DBP), Homocysteine (HCY), Myoglobin (MB), C-reactive protein (CRP), Neuron-Specific Enolase (NSE), S100β, treatment history (thrombolysis, thrombectomy, antiplatelet, anticoagulation, statin, PPI), and complicates (dysphagia and strokeassociated pneumonia) (Table 2).Subsequently, the previously identified variables were used to determine independent risk factors using multivariate regression.The results of the multivariable analysis in the training cohort are presented in Table 2. Independent risk factors associated with prognosis included NSE, HCY, CRP, S-100β, dysphagia, and anticoagulation.

Development and validation of predictive models.
Based on the six significant risk factors identified through multivariable Cox regression analysis on the training cohort, the following six machine learning algorithms were used to develop predictive models: Naive Bayesian classification (NBC), extreme Gradient Boosting (XGB), Random Forest (RF), Decision Tree (DT), Gradient Boosting Machine (GBM), and Logistic Regression (LR).For internal validation, tenfold cross-validation was employed to compare the performance of all models.ROC curves were plotted, and the AUCs of all established models are illustrated in Fig. 2.Among them, the RF model exhibited the best prediction performance with an AUC of 0.931.The AUCs of the XGBboost, GBM, DT, LR, and NBC models were 0.922, 0.923, 0.923, 0.829, and 0.871, respectively.Additionally, the selected models were comprehensively evaluated by calculating their accuracy, sensitivity, and specificity, as depicted in Fig. 3. Consequently, the RF model was hereby identified as the most effective predictive model for the prognosis of AIS, displaying high accuracy (0.789), sensitivity (0.755), and specificity (0.877).The F1 score is a commonly used metric for evaluating the overall performance of classification models, especially in situations with imbalanced class distributions.The F1 score ranges between zero and one, with values closer to one indicating better model performance.In the case of the final RF model, the F1 score was 0.699.Additionally, an external cohort was used to test the model, and the result also suggested that the RF model is the optimal model to predict the prognosis, with the highest ACU of 0.908.The AUCs for each model were: 0.884 for the XGB model, 0.883 for the GBM model, 0.879 for the DT model, 0.882 for the LR model, and 0.860 for the NBC model (Fig. 4).

Model interpretation.
This study analyzed the independent validation set in the RF model through the SHAP package, as shown in Fig. 5.In the final RF model, feature importance rankings of six predictors are Table 1.Baseline characteristics of AIS patients between good and poor prognosis.SBP systolic blood pressure, DBP diastolic blood pressure, SD stroke distribution, SOH side of hemisphere, SOS site of stroke lesions, NOS number of stroke lesions, LDL low-density lipoprotein, FBG fasting blood glucose, HCY homocysteine, UA uric acid, MB myoglobin, CRP C-reactive protein, FIB fibrinogen, BNP brain natriuretic peptide, NSE neuron-specific enolase, PPI proton pump inhibitor therapy, SAP stroke-associated pneumonia.The smaller the feature value, the corresponding SHAP value is less than zero7, indicating a negative impact.On the other hand, the larger the feature value, the corresponding SHAP value is greater than zero, indicating a positive impact.The region with the widest distribution is NSE, indicating that it has the greatest impact.

Online application for prognosis prediction.
Based on the RF model, an easy-to-use online calculator was built to predict the prognosis of AIS patients, which can be obtained at https:// mlmed icine-prog-progstroke-ysnamt.strea mlita pp.com/.By entering the patient's clinical data, doctors and patients can obtain the estimated probability of poor prognosis immediately.Readers can also use these detailed parameter settings to reproduce the proposed model in Python.The model_parameter_settings.txt can be downloaded from the following publicly available GitHub repositories.(https:// github.com/ Wu-Shi-Nan/ Acute-ische mic-stroke/ blob/ main/ model_ param eter_ setti ngs.txt).

Discussion
Stroke is the second most common cause of disability and the leading cause of death among adults 13 , imposing a substantial burden on families and society in terms of disease and medical impact.Furthermore, this burden has consistently increased over the past three decades 14,15 .Consequently, there is an urgent need to research early and accurate prediction of adverse outcomes in patients with AIS, to facilitate precise clinical management and surveillance.
According to the present study, dysphagia was identified as a significant cause of poor prognosis in patients with stroke.Dysphagia is a common complication after a stroke 16 .Martino et al. reported that dysphagia occurs in 37% to 78% of patients with stroke and increases the risk for pneumonia 3-11 times in patients with confirmed aspiration 17 .Additionally, previous literature has proved that dysphagia is the most important cause of post-stroke pneumonia [18][19][20] .Stroke inhibits immunological responses through the activation of the autonomic nervous system and stress axis, contributing to the development of stroke-associated pneumonia, further exacerbating the condition 21 .
The present findings showed that CRP has a negative effect on the prognosis in stroke patients.CRP is an inflammatory response indicator, which can be significantly elevated during the acute phase of an inflammatory response.Meanwhile, stroke and infection are strongly intertwined.Many studies have reported that impaired immune response occurs in stroke patients, resulting in increased susceptibility to infections 22 .It has been previously reported that the presence of infection during acute and subacute phases of stroke is a common predictor of a poorer prognosis 23 .Higher CRP values as associated with worse the infection 24 .Increased c-reactive protein is related to a poorer prognosis regarding the course of stroke 23 .
Meanwhile, AIS patients are prone to develop inflammatory response, and the immune-inflammatory response is an essential process post-stroke 25 .Xie et al. showed that the interaction between the activation of coagulation and the inflammatory response leads to the consumption of coagulation factors.Additionally, patients with a higher international normalized ratio had more serious strokes, and the increased international normalized ratio was an independent predictor of one-year all-cause mortality 26 .Anticoagulation increases the international normalized ratio and bleeding risk, thus increasing the incidence of poor prognosis.Bautista et al. suggested that warfarin use at baseline was associated with mortality 27 , which is consistent with the findings of this study.
S100β and NSE were positively associated with poor prognosis in the present findings.S-100β, a member of a family of Ca + binding proteins, is a small acidic calcium-binding protein and abundantly expressed in the nervous system, mostly in astrocytes and several neuronal populations 28 .Clinically, it is considered to be a serum marker of cerebral damage and hypoxia for assessing neurological prognosis 29 .NSE, a dimeric isoenzyme of the glycolytic enzyme enolase, is involved in the glycolytic pathway.It is abundantly present in neurons and cells of neuroendocrine origin and used as a serum marker for neuronal loss, and its highest activity is found in brain tissue cells 30,31 .S100β and NSE are markers of glial cells 32 .Due to stroke, ischemia and hypoxia occur in the brain, leading to rupture of the membrane structure of the neurons, and the process of acute stroke causes a large release of S100β and NSE into the blood.The degree of neurological impairment can be assessed to some extent by S100β and NSE 23,33 .Previous studies have reported that increased levels of NSE and S100B are positively correlated with poor outcomes of stroke patients [34][35][36] , which is consistent with our findings.NSE and S100β were also included in the presented predictive model to evaluate the prognosis of patients with stroke.Meanwhile, NSE ranked first among all predictors based on SHAP analysis.In addition, HCY was related to a poor outcome in stroke patients, and the relationship between hyperhomocysteinemia and poor outcome in stroke patients has been previously reported 37 .When stroke patients presented with inflammatory response, cellular injury and necrosis would occur.Furthermore, as a response to this reaction, adenosine triphosphate would be released into the extracellular space, and hydrolyzed into adenosine.Adenosine has anti-inflammatory and tissue-protective properties 38 .However, high HCY could reduce activities and protein content of electron transport chain components, hence lowering mitochondrial energy metabolism and decreasing adenosine triphosphate production 39 .Thereby, for patients with high HCY, the anti-inflammatory and tissue protection in AIS patients were worse, and the prognosis was poorer accordingly.www.nature.com/scientificreports/An RF model was developed in this study to predict the risk of poor prognosis in AIS patients, which showed excellent performance compared with other ML models.The factors NSE, HCY, CRP, S-100β, anticoagulation, and dysphagia were identified as independent factors for poor prognosis.This study has laid a foundation for timely and accurate prediction of poor prognosis risk in order to further organize stroke care.Finally, poor prognosis rates for AIS patients can be easily obtained by entering corresponding clinical features in the developed online calculator.As shown in Fig. 6, the probability is calculated online quickly (Probability of poor prognosis = 9.4%, low-risk group).The wide application of smart devices nowadays makes the developed tool greatly convenient to use.
Previous studies have attempted to use ML algorithms to evaluate the prognosis of stroke 5,[40][41][42][43] .Peng et al. leveraged a cohort of 423 patients for predicting 30-day mortality in spontaneous intracerebral hemorrhage patients 42 .Bacchi et al. designed models using LR, RF, DT, and artificial neural networks to predict in-hospital mortality 43 .Chen et al. used five ML techniques, including CatBoost, XGB, Boosting Decision Tree, RF, and AdaBoost, to investigate the 90-day poor prognosis in patients with transient ischemic attack or minor stroke.The factors associated with poor prognosis were identified, and the CatBoost model (AUC = 0.839) had the best predictive performance 40 .However, their study had a relatively short predictive window time.Heo et al. explored the applicability of ML algorithms to predict long-term prognosis in AIS patients.They reported that the deep neural network with an AUC of 0.888 can improve the prediction of long-term outcomes, but the predictive variables were not identified 5 .
Compared to other studies, the proposed predictive model has the following advantages.First, a comprehensive range of ML algorithms were tested, including NBC, XGB, RF, DT, GBM, and LR.This comprehensive approach allows exploration of different modeling techniques and selecting the most suitable algorithm for the specific prediction task.Second, several significant risk factors were identified through the analysis, such as NSE, HCY, CRP, S-100β, anticoagulation, and dysphagia, which are independently associated with unfavorable outcomes in AIS patients.The variables mentioned above are mostly readily available patient information, which makes this model potentially suitable for assisting clinical applications.Proper use of the model can provide valuable insights into the factors influencing prognosis and aids in clinical decision-making.Third, this study employed a relatively long prediction window, which extends the observation period and captures a broader range of patient outcomes.This reduces the impact of short-term fluctuations and random variability, enhancing the reliability and stability of the predictions.Fourth, the SHAP values were used to interpret the model, providing insights into the contribution and importance of each feature in the prediction process.This enhances the model transparency and interpretability, allowing for better understanding and trust in the results.Fifth, a web-based platform that visualizes the predictive model and provides a dynamic calculator for easy and practical usage was developed.This user-friendly interface enhances the accessibility and usability of the proposed model, making it more convenient for clinical applications.Overall, the predictive model stands out by utilizing clinical samples combined with various ML algorithms, identifying significant risk factors, employing a long prediction window, providing model interpretation, and offering a user-friendly platform.These advantages contribute to its potential clinical utility and effectiveness in predicting AIS outcomes.
There are several limitations present in this study.First, this is a retrospective study, which introduces the potential for selection bias and limits the establishment of causal relationships.Second, the sample size was not large enough, which may affect the generalization ability of the findings, and the data collection was limited to a single medical center, which may restrict the external validity of the results.Despite the good predictive power of our ML model, external validation in a completely different patient cohort is unavailable.Third, it is recommended to compare the predictive model with another prognostic system, such as the Acute Stroke Registry and Analysis of Lausanne score, to provide a more comprehensive evaluation and enhance the persuasiveness of the findings 44 .Finally, it should be noted that the NIHSS score, which stands for National Institutes of Health Stroke Scale, is a widely predictor of outcomes in AIS predictive models 45 .The NIHSS score upon admission was found to be a significant factor in the development of unfavorable outcomes.It is important to acknowledge that the absence of the NIHSS score data in the final predictive model might have limited its performance.A prospective multiple-center cohort could be designed to enhance the credibility of the results in the future.

Figure 1 .
Figure 1.The flowchart of the patient selection.
(1.31-3.79)0.002 1.42 (0.55-3.66) 0.464 PPI Continued shown in Fig. 5A.Based on the SHAP summary plots for poor prognosis in AIS patients, the related features ranked from highest to lowest importance were NSE, HCY, S-100β, dysphagia, CRP and anticoagulation.Figure 5B showed the distribution of the contribution of each feature to the model output.The influence of feature values on the results is represented by colors, with each dot representing a case in each row, red dots representing larger feature values, and blue representing lower ones.

Figure 2 .
Figure 2. The results of tenfold cross-validation in the training cohort.

Figure 3 .
Figure 3. Performance of different models.

Figure 4 .
Figure 4.The results of the validation in the validation cohort.

Figure 5 .
Figure 5. SHAP for the RF model.(A) importance ranking of features.(B) the distribution of the impacts of each feature on the model output.

Figure 6 .
Figure 6.The web calculator for predicting prognosis of AIS patients.

Table 2 .
Univariate and multivariate logistic regression analysis of risk factors for poor prognosis in AIS patients.SBP systolic blood pressure, DBP diastolic blood pressure, SD stroke distribution, SOH side of hemisphere, SOS site of stroke lesions, NOS number of stroke lesions, LDL low-density lipoprotein, FBG fasting blood glucose, HCY homocysteine, UA uric acid, MB myoglobin, CRP C-reactive protein, FIB fibrinogen, BNP brain natriuretic peptide, NSE neuron-specific enolase, PPI proton pump inhibitor therapy, SAP strokeassociated pneumonia.