Introduction

Medical patients admitted to Emergency Department (ED) are highly heterogeneous in terms of disease spectrum and severity. Mortality is the most important outcome of ED care, and the rate of mortality can be used as a means for better prioritization of care and resource allocation. The rate of mortality among ED patients is high1. In ED, the death of a patient is commonly preceded by a cumulative deterioration of vital signs and clinical abnormalities2,3. Therefore, several prognostic models, including the Rapid Emergency Medicine Score4, Rapid Acute Physiology Score5 and Worthing Physiological Scoring system6, have been developed to make use of the clinical signs and abnormalities for predicting the risk of death in ED patients.

In a previous study7, we demonstrated that the above algorithms had good prognostic performance in the prediction of 30-day mortality in a tertiary hospital in Vietnam. However, the degree of discrimination, as measured by the area under the receiver operating characteristic curve (AUC), was between 0.70 and 0.807, suggesting that there is room for further improvement of the existing prognostic models. In developing countries, EDs are normally overcrowded, and staff constantly struggle with an overwhelming number of patients from unplanned admissions. Moreover, the condition of care and disease severity among patients in developing countries are different from those in industrialized countries, and the difference calls for new prognostic models that can be applicable to ED patients in developing countries.

The present study sought to develop and validate a clinical predictive model for predicting 30-day mortality risk in ED patients by using routinely collected clinical, physiological and vital signs. We demonstrated that there exists a series of models that have comparable predictive accuracy, and that these models can be used to identify high-risk patients in ED. The models reported here can empower medical care providers to individualise ED care and optimise ED utilisation.

Results

The study was carried out in two cohorts: development and validation. The two cohorts were recruited from two independent hospitals. The development cohort was used to derive predictive models, and the validation cohort was used for testing the predictive models.

Between 13 March 2013 and 1 June 2013, we had enrolled 2175 medical patients in ED for development cohort. However, after excluding patients who did not meet the inclusion criteria and patients who withdrew from the study, 1765 patients remained for the model development. During the follow-up period, the 30-day incidence of mortality in the development cohort was 9.8% (n = 173). Baseline characteristics of patients in the development cohort stratified by mortality status are shown in Table 1. The average age of participants was 65.8 years (range: 16–105 years). There was no statistically significant difference in age between survivors and deceased.

Table 1 Clinical characteristics of 1765 patients in the development cohort and 1728 patients in the validation cohort classified by mortality status.
Full size table

Bivariate analysis (Table 2) showed that the following risk factors were associated with an increased risk of mortality: male gender, advancing age, increased pulse, increased body temperature, lower systolic blood pressure and diastolic blood pressure, increased respiratory rate, reduced peripheral oxygen saturation, increased duration of illness, and lower Glasgow Coma Score (GCS). Moreover, cardiopulmonary resuscitation, mechanical ventilation, and admission to intensive care unit (ICU) were each associated with increased risk of mortality. Among comorbidities, cancer, chronic renal failure, cirrhosis, chronic respiratory failure, heart failure, and diabetes mellitus were also associated with a greater risk of mortality in ED patients.

Table 2 Risk factors for 30-day mortality from the development cohort: bivariate analysis.
Full size table

Model development

The Bayesian Model Average (BMA) algorithm identified 3 most parsimonious models for predicting the risk of 30-day mortality. The three models included the following factors: gender, respiratory rate, peripheral oxygen saturation, duration of illness, GCS, ICU admission, serum urea, glycaemia, serum albumin, alanine aminotransferase (ALT), and high-sensitivity C-reactive protein (hsCRP).

Model I included 10 factors and had a posterior probability of 5.5%; model II included 8 factors with 4% posterior probability; and model III included 11 factors with 3.8% posterior probability. Several risk factors were present in different models. The common factors in all three models were: gender, respiratory rate, peripheral oxygen saturation, duration of illness, GCS, ICU admission, serum urea, and serum albumin. The hazard ratio (HR) and 95% CI associated with each factor and each model are shown in Table 3. Among the risk factors, ICU admission was associated with the greatest risk or mortality, with the average HR ranging from 5.6 (Model I) to 6.6 (Model II).

Table 3 Association between risk factors and 30-day mortality risk: results of multivariable analyses.
Full size table

The AUC for the three models (Fig. 1) were comparable. Model I yielded the AUC value of 0.873 (95% CI, 0.848–0.90), not significantly different from that of Model II (AUC = 0.871; 95% CI, 0.844–0.898) or Model III (AUC = 0.873; 95% CI, 0.845–0.90). From these results, it appeared that Model II was the most optimal model, because it has the least number of risk factors but yielded good discriminatory power.

Figure 1
figure 1

Area under the receiver operating characteristic curve (left panel) and calibration plot of three parsimonious models (middle and right panels).

Full size image

Model validation

Between 19 October 2013 and 31 March 2014, 2060 patients had been recruited for the validation cohort. After excluding patients who did not meet the inclusion/exclusion criteria, 1728 patients were available for analysis (Fig. 2). The incidence of 30-day mortality was 7.8% (n = 135), and this incidence rate was slightly lower than that in the development cohort (P = 0.04, Fisher’s exact test). When the models derived from the development cohort were applied to predict the risk of mortality in the validation cohort, we found that there was a good agreement between observed and predicted risk of death. The maximum calibration error in predicting probability of mortality was about 4% for Model I and Model III, and 3.4% for Model II. In general, the AUC values obtained from the validation cohort was slightly lower than those obtained from the development cohort, but all AUC values were close to 0.8, a level deemed to be good discrimination. For example, the AUC for Model I, II and III was 0.788, 0.783, and 0.790, respectively (Fig. 1). Based on parameter estimates of Model II, a nomogram was built for individualising the risk of 30-day mortality (Fig. 3).

Figure 2
figure 2

CONSORT diagram: Description of recruitment of study participants for the development and validation cohort.

Full size image
Figure 3: Nomogram for predicting the 30-day mortality risk based on Model II.
figure 3

Instruction for usage: Mark the gender of an individual on the “Gender” axis and draw a vertical line to the “Points” axis to determine how many points toward the probability of mortality the individual receives for his/her gender value. Repeat the process for each additional predictor. Sum the points of the predictors. Locate the final sum on the “Total” axis. Draw a vertical line down to the 30-day risk line to find the individual’s probability of sustaining mortality within 30 days (ICU: Intensive care unit).

Full size image

Discussion

Emergency Departments worldwide are typically overcrowded and understaffed, and these problems have recently become a topic of discussion in emergency medicine. In developing countries, the problem is even more severe as EDs are often under-resourced. Under these circumstances, a risk stratification-based prioritization of care is a sensible approach. However, predicting the risk of mortality in EDs is a challenging task, because patients are not only highly heterogeneous but also medically complicated. In this study, by using a Bayesian approach to combine common clinical and biochemistry tests, we have developed and validated a series of predictive models for individualising the risk of 30-day mortality in ED patients. These models have proven good discrimination and calibration in external validation.

It is commonly thought that clinician’s assessment or clinical intuition can accurately predict the risk of mortality in EDs. However, when dealing with multiple risk factors, clinician’s assessment can be problematic, because they are unable to weigh information in a rational and objective manner. The lack of objectivity can result in inconsistent risk assessment between clinicians8. In some cases, considerable discrepancies between clinical intuition and actual mortality9,10. Therefore, in the presence of multiple risk factors, some of which may be potentially important, statistical thinking offers clinicians some useful guidance for dealing with uncertainty of risk assessment.

One of the most important aspects of statistical model building is to find relevant risk factors that have good predictive values. Typically, researchers develop a hypothesis about the relationship between risk factors and outcome, and then apply an algorithm, mostly stepwise regression, to identify a set of possible relevant risk factors. However, this classical approach suffers from two shortcomings that the stepwise method tends to identify redundant variables11, and that it produces only one “final” model. In reality, there are more than one competing model that can explain the relationship. A better approach is Bayesian Model Averaging which was introduced to scientific research about 20 years ago12, but has not been widely used in medical research. The Bayesian Model Averaging method has been shown to have superior and more robust performance than the stepwise method in the identification of relevant risk factors11,13. In this study, by using the Bayesian Model Averaging approach we have identify three sets of risk factors that can help clinicians estimate the risk of mortality in EDs patients.

Some risk factors for mortality identified in the present study have also been identified in previous studies6,14,15,16,17. It is well known that in ED men have a greater risk of mortality than women18, and this was also observed in our study. In clinical setting, about 45% of patients who died had antecedent abnormal vital signs, including respiratory distress, from 8 hours to 48 hours19. Moreover, among patients with normal vital signs at admission, 30% will be deteriorated within 24 hours20.

Tachypnea was a predictor of mortality in this study after adjusting for other predictors, and this finding is consistent with previous studies4,5,6,16. Peripheral oxygen saturation, another index for monitoring respiration, is also a known risk factor for mortality4,6,16,17. Low oxygen saturation (<90%) increases the odds of mortality within 24 hours by five-fold17, and in our study, decrease in peripheral oxygen saturation had a less pronounced effect, with a hazard ratio of 1.17 after adjusting for other covariates in the model.

In the present study, we found that decreased GCS, increased serum urea, increased glycaemia, decreased serum albumin, increased ALT, and increased hsCRP were independently associated with increased mortality risk. These associations have also been reported by various studies15,21,22,23,24,25. Interestingly, we found that a decision of ICU admission was strongly associated with mortality. Indeed, the risk of mortality among patients with this indication was increased by 6.6-fold compared with those without the indication. It could be argued that decision of ICU admission reflects a “clinical impression” which is known to be a good predictor of mortality status18,26. This finding suggests that clinical intuition can be an important factor, in addition to other measured factors, in the assessment of mortality in ED patients.

A number of multivariable models for predicting mortality in ED patients have been developed, and in validation cohorts, these models have good discrimination. For instance, the AUC for the Early Warning Score (7 factors), Simple Clinical Score (14 factors), Rapid Emergency Medicine Score (7 factors), and Worthing Physiological Scoring system (5 factors) for predicting mortality was 0.7327, 0.82628, 0.7127, and 0.7977, respectively. Among the models, only the Rapid Emergency Medicine Score (REMS) and Worthing Physiological System (WPS) were externally validated in developing countries. However, none of the available prognostic models appears to be suitable for individualised prognosis in ED patients. Our models have comparable AUC to those existing models. In previous studies, combining physician’s prediction with an objective model did not substantially improve the discrimination of mortality26; however, in our study, the integration of physician’s decision of ICU admission with objective models markedly increased the discrimination.

Our results should be interpreted within context of strengths and potential limitations. The study was designed as a prospective investigation with large sample size which allows us to define modest associations between risk factors and mortality. We have rigorously validated our models in a totally independent cohort which avoided the problem of over-fitting commonly found in previous studies. Moreover, we used sophisticated statistical approaches to derive at a series of predictive models, not just a single model that was reported in previous studies. In our view, there exists more than one model that can help predict mortality status, and the reliance on a single model is an underestimate of the uncertainty in the data. However, since patients’ information were collected at one time point, which might not reflect a full spectrum of a patient’s dynamics. The decision of ICU admission is, of course, not objective, and could potentially be a bias.

In summary, we have developed and validated new prognostic models to predict the risk of 30-day mortality in medical ED patients. The models will allow accurate risk assessment to identify high risk patients, and help optimize resources and patient management in ED.

Methods

Setting and patients

The study was conducted at the Can Tho National Hospital, Vietnam (for development cohort) and the Can Tho General Hospital, Vietnam (for validation cohort). The Can Tho National Hospital is a tertiary teaching hospital that serves 17 million residents in the Mekong Delta region; its ED on average admits 75 non-surgical patients per day. The Can Tho General Hospital has 400 beds, providing care to the residents of Can Tho City; on average, the ED admits 50 non-surgical patients per day.

We enrolled medical patients (non-trauma and non-surgical) from the two emergency departments between 13 March 2013 and 31 March 2014. The inclusion criteria were: all patients aged 16 years and older and who could give informed consent. Patients were excluded from the study if they had one of the following conditions: acute coronary syndrome, burns, cardiac arrest before admitting to the hospital or which occurred in the ED with failure of cardiopulmonary resuscitation, snakebite, insect bite or sting, poisoning (drugs, alcohol, intoxication, paraquat, insecticides, rodenticides, corrosive substances). We also excluded patients with burns, cardiac arrest with failure of cardiopulmonary resuscitation. Women in labor and dead-on-arrival patients were also excluded from the study. The study protocol and procedure were approved by the Can Tho National Hospital ethics committee. All patients gave written informed consent. Patients could withdraw from the study at any time without giving reasons. All methods were performed in accordance with the relevant guidelines and regulations.

Study procedure

All patients who met inclusion and exclusion criteria were invited to participate in the study. Upon giving the written informed consent, data collection was conducted by a trained research worker by using a structured questionnaire. The questionnaire collected data concerning demographic characteristics, medical history, physiological data, haematological and biochemistry tests (see Supplementary Table S1). After 30 days of admission, the research worker would contact patients or relatives or guardians to obtain information on survival status. Patients who had stayed in hospital for more than 30 days were considered “censored”.

Outcome measure

The primary outcome of the study was mortality occurred within 30 days of hospital admission. Mortality was defined as (1) death in hospital from any cause; (2) family-initiated discharge and death either on the way home or within 24 hours after discharge; (3) doctor-initiated discharge and death at home. It should be noted that in Vietnamese culture, when a patient was in the end stage of disease, the patient or family often requests for discharge from hospital because they prefer to pass away at home.

Risk factors

Risk factors considered in this study included vital signs, peripheral oxygen saturation, duration of illness, GCS (see Supplementary Table S2), cardiopulmonary resuscitation, mechanical ventilation, ICU admission, functional status, comorbidity, haematological and biochemical test (see Supplementary Table S1). Blood pressure and pulse rate were measured electronically (OMRON HEALTHCARE Co, Vietnam) and rechecked manually where blood pressure was either too low or too high. Peripheral oxygen saturation was measured by an electronic device (NONIN Co, USA). Decision of ICU admission was made by a senior attending physician after considering laboratory tests and primary diagnosis.

Data management and analysis

Data were entered into a designed database twice. The first data entry was made within a day after admission. The second data entry was undertaken at the termination of the study. Data from the two entries were used to check for potential inconsistencies, and any inconsistency was adjudicated with the original patient record.

Data analysis was performed according to a statistical analysis plan prior to the collection of data. First, we used descriptive statistics (i.e., mean, standard deviation, median with interquartile range, proportion) for each clinical and laboratory variable with stratification by survival status. Difference between groups in categorical variables was tested by the Fisher’s exact test. The normal distribution of continuous variables was tested by the Shapiro-Wilk normality test. The association between continuous variables and 30-day mortality was tested by the Student’s t test for normally distributed variables or Mann-Whitney test for non-normally distributed variables.

Second, we used the Cox’s proportional hazards model29 to assess the association between potential risk factors and 30-day mortality (i.e. hazard ratio) from the development cohort. Since there were several potential risk factors and the number of “potential models” for predicting 30-day mortality can be very large, the Bayesian Model Averaging (BMA)30 was used to search for the most parsimonious models from the development cohort. This approach has been shown to have superior performance compared to “traditional” approaches such as stepwise regression11,13. In the BMA approach, the regression analysis was performed for 2M (where M is the number of risk factors) competing models. The BMA averaged point estimates for regression coefficient over the all possible models. BMA produces a posterior probability of each possible model and posterior probability for regression coefficient associated with each genetic variant. The posterior probability is a function of a prior probability and the likelihood of a model. In this study, given the large number of genetic variant and there is little information available for eliciting prior distributions, we used the “uninformative” prior distributions, that a priori, make all models and parameters equally likely are appealing. Thus, in BMA, we consider a set of competing models, not just a single model, to account for the variation in the data. The analysis was done with the R statistical environment31 and the BMA package32.

The discrimination of the most parsimonious models was estimated by the AUC33 for the development and validation cohort. The difference between two AUCs was tested by the Delong method34. Agreement between observed mortality and predicted mortality was evaluated using the bootstrap technique. In this technique, 1000 sub-samples, each with 500 subjects, of the entire sample were repeatedly sampled (with replacement) and analyzed35. Based on the parameters selected from the most parsimonious models, a nomogram using the “rms” library35,36 was constructed for predicting 30-day mortality. In a further validation, the predicted probability of 30-day mortality was compared with the observed probability on the development cohort and validation cohort. All statistical analyses were performed with the R Statistical Environment version 3.1.0 on Windows platform37.

Additional Information

How to cite this article: Ha, D. T. et al. Development and validation of a prognostic model for predicting 30-day mortality risk in medical patients in emergency department (ED). Sci. Rep. 7, 46474; doi: 10.1038/srep46474 (2017).

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.