A novel simple risk model to predict the prognosis of patients with paraquat poisoning

To identify risk factors and develop a simple model to predict early prognosis of acute paraquat (PQ) poisoning patients, we performed a retrospective cohort study of acute PQ poisoning patients (n = 1199). Patients (n = 913) with PQ poisoning from 2011 to 2018 were randomly divided into training (n = 609) and test (n = 304) samples. Another two independent cohorts were used as validation samples for a different time (n = 207) and site (n = 79). Risk factors were identified using a logistic model with Markov Chain Monte Carlo (MCMC) simulation and further evaluated using a latent class analysis. The prediction score was developed based on the training sample and was evaluated using the testing and validation samples. Eight factors, including age, ingestion volume, creatine kinase-MB [CK-MB], platelet [PLT], white blood cell [WBC], neutrophil counts [N], gamma-glutamyl transferase [GGT], and serum creatinine [Cr] were identified as independent risk indicators of in-hospital death events. The risk model had C statistics of 0.895 (95% CI 0.855–0.928), 0.891 (95% CI 0.848–0.932), and 0.829 (95% CI 0.455–1.000), and predictive ranges of 4.6–98.2%, 2.3–94.9%, and 0–12.5% for the test, validation_time, and validation_site samples, respectively. In the training sample, the risk model classified 18.4%, 59.9%, and 21.7% of patients into the high-, average-, and low-risk groups, with corresponding probabilities of 0.985, 0.365, and 0.03 for in-hospital death events. We developed and evaluated a simple risk model to predict the prognosis of patients with acute PQ poisoning. This risk scoring system could be helpful for identifying high-risk patients and reducing mortality due to PQ poisoning.

The non-selective contact herbicide paraquat (PQ) is predominantly used in developing agricultural countries 1,2 . PQ is lethal when ingested orally, and there are still no effective and specific antidotes. Patients with acute PQ poisoning usually die within several days to weeks after exposure due to hypoxemia or multiple organ failure. PQ poisoning results in increased medical resource use, causing a considerable economic impact [3][4][5][6] . Therefore, timely clinical outcome evaluation and risk assessment for critically ill PQ poisoning patients and essential for appropriate medical resource allocation, which has become an important public health and social security agency issue concerning doctors and patients.
To our knowledge, several systems have been reported to predict the prognosis of patients with PQ poisoning, including the Acute Physiology and Chronic Health Evaluation II (APACHE II) score 7 , Sequential Organ Failure Assessment (SOFA) score 8 , the Severity Index of PQ Poisoning (SIPP) 9 , Poisoning Severity Score (PSS) 10 , and several equations and nomograms based on large cohort studies [11][12][13] . Most of these models are suitable for critically ill patients rather than patients with minimal exposure or early-stage patients with mild symptoms. Moreover, these scoring systems can fail to predict mortality and conduct risk assessment for PQ poisoning patients instantly because of their complicated calculations or the unavailability of laboratory tests. Thus, establishing an effective, simple, and universal predictive model based on common laboratory tests would be invaluable for risk stratification and therapeutic regimen adjustment for patients with acute PQ poisoning of all stages. www.nature.com/scientificreports/ Here, based on data from 1199 patients with PQ poisoning from two large academic hospitals in China, we developed and evaluated a simple risk model by identifying significant clinical risk factors to predict in-hospital death. Data for our study were collected from the medical records of acute PQ poisoning patients from different time periods and geographic regions. This study describes a well-designed and easy to administer tool for predicting in-hospital death of PQ poisoning patients using a combination of simple and clinically relevant variables. ). Detailed information on these variables in the 1199 samples is shown in "Supplementary file S1", Table S1. To facilitate the risk score calculation, we examined the nonlinear relationship of each continuous factor with the outcome and categorized it using a cut-off point while taking into account both in-hospital death rates and sample sizes ("Supplementary file and testing (33.3% [304 patients]). All analyses were conducted using SAS ® statistical software version 9.4 (https ://www.sas.com). S1", Fig S1). Parameters with missing data rate > 15% were excluded. Factors with missing values were imputed using ten imputations. The final imputed value was the average of the ten imputations. Missing values ranged from 1.1% (age) to 13.3% (CK-MB). The primary outcome was in-hospital death, which defined as death during hospitalization.

Methods
Statistical analysis. Risk factor selection and evaluation. Using the training sample with all candidate risk factors, we fit a logistic model with Markov Chain Monte Carlo (MCMC) simulation, and we calculated a posterior probability for each factor 14 . The posterior probability could assess the association strength between a factor and the outcome. Factors with a posterior probability > 0.95 (or < 0.05 for factors with estimates < 0.0) were considered significant for predicting the outcome and were included in the final risk factor list 15 . Then, we developed the final risk model to predict the outcome by fitting a logistic regression to the training sample data using the selected risk factors identified by the MCMC method.
We used the following indicators to evaluate the risk model performance: the Harrell C statistic to assess the overall predictive accuracy 16,17 , the McFadden R square to assess explained variation 18 , and the Hosmer-Lemeshow goodness-of-fit test to assess calibration 19 . Discrimination was assessed between the observed outcomes in strata defined by the predictive probabilities' deciles. As described previously, we divided patients in the training sample into five mutually exclusive risk classes based on the deciles, ranking them from lowest (class 1) to highest risk (class 5) for evaluation 20 .
To further evaluate the selected risk factors, we used the training data to conduct a latent class analysis by an unsupervised machine learning algorithm that does not require a specified outcome 14 . Conceptually, unsupervised learning algorithms can assign patients to risk classes if the selected risk factors are substantially associated with the outcome. Thus, we used latent class analysis to classify patients into five mutually exclusive classes and ranked them from lowest (class 1) to highest risk (class 5) based on the observed outcome. We chose five classes to align with the decile-specific classes based on the risk model described previously. We calculated a Spearman correlation coefficient between the risk classes based on the risk model and the risk classes determined by the latent class analysis. A high coefficient was indicative of good agreement between the classified results, thereby providing information on the robustness of the selected risk factors.
We further validated the risk model by comparing its performance with the training sample, test sample and 2 independent validation samples. Risk score. As described previously 15 , to facilitate the use of the selected risk factors and the risk model, we developed a simple risk score for each patient based on the regression coefficients estimated from the risk model with the training sample. Points for each risk factor were calculated by dividing the risk factor's coefficient by the sum of all coefficients in the model, multiplying by 100, and rounding to the nearest integer. We stratified patients into 3 risk groups based on the distribution of the risk score: low (< 25th percentile), average (25th-75th percentile), and high (> 75th percentile).
All analyses were conducted using SAS ® software version 9.4 (SAS Institute Inc.). Latent class analysis was conducted using the PROC LCA method (version 1.3.2 beta). Nonlinear relationships were assessed using the PROC GAM method. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline. All 22 items of the TRIPOD statement were addressed.

Results
Study sample. A total of 1199 (609 training, 304 test, 207 validation by time, and 79 validation by site) participants were included in this study. The mean (SD) age was 35.1 (16.4) years, and 599 patients (50.0%) were female. There were some differences in the basic clinical characteristics of patients in training, test, and two validation samples. The differences between the training, test, and validation samples demonstrate the predictability and extrapolation of the model (Table 1).
Risk factor selection and testing. The MCMC simulation identified eight candidate factors with a posterior probability of at least 0.95 (Table 2), including age, ingestion volume, CK-MB, PLT, WBC, N, GGT, and Cr (Fig. 2). The risk model based on the eight risk factors and the training sample demonstrated good discrimination, calibration, and fit. The overall C statistic was 0.926 (95% CI 0.891-0.924) for the risk model (Fig. 3). The mean observed in-hospital death rate ranged from 3.3% in the lowest predicted quintile to 99.2% in the highest predicted quintile, with a range of 95.9% and variation of 0.4999 (Fig. 4). Moreover, the P-values of the Hosmer-Lemeshow goodness-of-fit test were 0.4262 in the training sample and 0.9078 in the test sample, indicating a good fit with the test cohort (Fig. 5).
Additionally, model performance with the test sample was comparable to that with the training sample. The overall C Statistic was 0.895 (95% CI 0.855-0.928) in the test sample (Fig. 3). The observed in-hospital death rate ranged from 4.6% in the lowest predicted quintile to 98.2% in the highest predicted quintile, and the explained variation was 0.4182 for the test sample (Fig. 4).
Furthermore, in the latent class analysis, 609 patients in the training sample were assigned into five classes based on the combination of the eight risk factors (Fig. 6). For this analysis, the area under the receiver operating curve (ROC) curve was 0.877 (95% CI 0.832-0.906), and the mean observed in-hospital death rate ranged www.nature.com/scientificreports/ from 0.0% in the lowest-to 99.2% in the highest rating group. The Spearman's correlation coefficient between the predicted quintile based on the logistic model and the latent class analysis was 0.754 (95% CI 0.715-0.874).
WBC ≥ 20, CK-MB ≥ 50, N ≥ 80, ingestion volume ≥ 100, and Cr ≥ 150 were the top five factors with odds ratios > 5.0 (Fig. 2). The training sample had a mean (SD) risk score of 26.6 (21.6). The mean (SD) score of the test sample was 24.8 (21.1). Further, in the training sample, 18.4%, 59.9%, and 21.7% of patients were stratified into the high, average, and low-risk groups, with corresponding probabilities of 0.985, 0.365, and 0.03 for in-hospital death, respectively (Fig. 7). The stratification of the test sample was not markedly different from that of the training sample (Fig. 7, Table 3).

Risk model validation.
The in-hospital death events rates were 42.0% (95% CI 35.2-49.1%) and 3.8% (95% CI 0.8-10.7%) for the time and site validation samples, respectively ( Table 1). The observed in-hospital death rates ranged from 2.3% and 0.0% in the lowest predicted quintiles to 94.9% and 12.5% in the highest predictive quintiles, and the explained variations were 0.3798 and 0.2421 for the time and site validation samples, respectively. The overall C statistics were 0.891 (95% CI 0.848-0.932) and 0.827 (95% CI 0.455-1.000) (Fig. 3), and the P-values of the Hosmer and Lemeshow's Goodness of Fit Test were 0.9671 and 0.9999 for the two independents samples (Fig. 5).  www.nature.com/scientificreports/ The mean (SD) risk scores were 23.5 (19.7) and 16.3 (14.1) for the time and site validation samples, respectively. In the validation_time sample, 27.1%, 59.4%, and 13.5% of patients were classified into the low-, average-, and high-risk groups, respectively, with corresponding probabilities of 0.98, 0.38, and 0.03 for in-hospital death events ( Fig. 7 and Table 3). In the validation_site group, 36.7%, 60.8%, and 2.5% of patients were classified into the low-, average-, and high-risk groups, with corresponding probabilities of in-hospital death events of 0.03, 0.33, and 0.97. The probabilities for in-hospital death events of the validation samples were identical to those of the training sample ( Fig. 7 and Table 3).

Discussion
In this study, we evaluated 609 patients with acute PQ poisoning, and we identified eight prognostic factors for survival after PQ poisoning. Furthermore, a random forest risk model and a scoring system based on these risk factors were developed to predict the in-hospital death probability for patients diagnosed with PQ poisoning. Importantly, our model was tested both in internal and external validation samples. The risk factors were selected based on data from medical records and ease of collection and availability after discharge and during long-term follow-up. Furthermore, the statistical algorithms used in this study are robust. The risk prediction model and its corresponding risk scoring system may help clinicians to identify the patients at highest risk of in-hospital death after PQ poisoning.
PQ exposure is a major cause of fatal poisoning in most Asian nations. PQ can cause severe multiple organ damage, such as lungs and gastrointestinal tract, which are the main causes of death after PQ poisoning [21][22][23] . Although therapeutic strategies for the management of acute PQ poisoning have been extensively investigated, PQ poisoning remains poor 24,25 . Generally, the severity and prognosis of acute poisoning are determined by the ingested dose. Previous studies have reported that plasma PQ concentrations can be used to predict clinical outcomes for patients with PQ poisoning 26,27 . In view of plasma PQ concentration is quantitatively measured using radioimmunoassay or liquid chromatography-mass spectrometry methods, which cannot be done in most hospitals, we used PQ ingestion volume rather than PQ plasma concentration at present medical situation. Moreover, Patients with PQ poisoning were conscious on admission, including critical PQ poisoning patients. Clinicians can easily obtain PQ ingestion volume data from the patients or their relatives. Identifying potential risk factors to predict the clinical outcome of PQ poisoning patients at an early stage may improve early diagnosis and facilitate rapid medical intervention for patients who have the highest risk of in-hospital death. Therefore, there is an urgent need to develop a reliable and universal risk prediction model based on available laboratory data to evaluate the early prognosis of PQ poisoning patients. Furthermore, validating and classifying the risk of www.nature.com/scientificreports/ in-hospital death based on a multi-clinical factor index model is an important innovation in emergency medical research, which could aid in risk stratification and therapeutic regimen adjustment for patients with acute PQ poisoning.
The MCMC algorithm was used to evaluate the association strength between the risk factors and the outcome. Compared with prior studies, the clinical indicators included in this model were routine and inexpensive, and every hospital could easily acquire the data. Our model had better predictive accuracy based on two external validation samples from a different time and site, which were used to validate our scoring system. The SIPP has been recognized as a valuable prognostic model for PQ poisoning patients in previous studies 9, 28 . The unavailability of SIPP indicators in many hospitals makes it difficult to apply the scoring system routinely, limiting the accurate evaluation of PQ poisoning severity. The APACHE II scoring system is also in clinical settings to evaluate the prognosis of PQ poisoning patients 29,30 . However, this scoring system might underestimate mortality in critically ill patients. Our prediction model was developed using the baseline clinical indicators of PQ poisoning patients. It is easy to use and allows the exclusion of terminally-ill or minimally affected patients from needless aggressive therapy.
Hidden factors used for risk model development may affect the efficiency of the model in different contexts. Effective risk prediction factors must be supported by clinical research, and the data must be easily collected and  Previous studies demonstrated that WBC and neutrophil counts were essential indexes for the prognosis of PQ poisoning patients 31,32 . Creatinine, one of the risk factors associated with a poor prognosis, could be induced by direct oxidative injury in renal tubules 33 . Elevated serum Cr level is closely associated with acute kidney injury (AKI). Further, PQ poisoning patients with AKI had a higher mortality risk than those with normal kidney function 34 . Through risk stratification, we found that in the training sample, the proportion of patients at high risk or moderate risk of in-hospital death after PQ poisoning was 98% and 36%, respectively. The eight risk factors we identified could help clinicians make medical interventions that may improve the prognosis and reduce unnecessary treatment of PQ poisoning patients. Improving the clinical outcomes of PQ poisoning patients and reducing in-hospital death rates would decrease the economic burden on the healthcare system.

Conclusion
The early clinical outcomes for PQ poisoning patients have been difficult to evaluate. The novel risk model we developed is easy to use and can exclude the terminally-ill or minimally affected PQ poisoning patients from needless aggressive therapy better than conventional scoring systems such as APACHE II, SOFA, and SIPP. This risk score system is suitable for clinical use for recognizing high-risk PQ poisoning patients and for predicting the probability of in-hospital death.