External validation of a triage tool for predicting cardiac arrest in the emergency department

Early recognition and prevention comprise the first ring of the Chain of Survival for in-hospital cardiac arrest (IHCA). We previously developed and internally validated an emergency department (ED) triage tool, Emergency Department In-hospital Cardiac Arrest Score (EDICAS), for predicting ED-based IHCA. We aimed to externally validate this novel tool in another ED population. This retrospective cohort study used electronic clinical warehouse data from a tertiary medical center with approximately 130,000 ED visits per year. We retrieved data from 268,208 ED visits over a 2-year period. We selected one ED visit per person and excluded out-of-hospital cardiac arrest or children. Patient demographics and computerized triage information were retrieved, and the EDICAS was calculated to predict the ED-based IHCA. A total of 145,557 adult ED patients were included. Of them, 240 (0.16%) developed IHCA. The EDICAS showed excellent discrimination with an area under the receiver operating characteristic (AUROC) of 0.88. The AUROC of the EDICAS outperformed those of other early warning scores (0.80 for Modified Early Warning Score [MEWS] and 0.83 for Rapid Emergency Medicine Score [REMS]) in the same ED population. An EDICAS of 6 or above (i.e., high-risk patients) corresponded to a sensitivity of 33%, a specificity of 97%, and a positive likelihood ratio of 12.2. In conclusion, we externally validated a tool for predicting imminent IHCA in the ED and demonstrated its superior performance over other early warning scores. The real-world impact of the EDICAS warning system with appropriate interventions would require a future prospective study.

www.nature.com/scientificreports/ However, the predictive performance of EDICAS in external validation remains unclear. Therefore, we aimed to externally validate this novel tool in another ED population of a tertiary medical center.

Methods
Study design and setting. We conducted a retrospective cohort study using data from the Medical Database of the Far Eastern Memorial Hospital (FEMH), a tertiary academic medical center with approximately 1,200 beds and 130,000 ED visits per year. This database serves as a central clinical data repository for all electronic medical records of the healthcare system, including inpatient, outpatient, and ED records. The electronic database houses various information of the patient, including demographics, diagnosis, treatment, imaging, laboratory, prescription, nursing, billing, and administrative data. In this investigation, we retrieved data from 268,208 ED visits in the FEMH from January 1 2016 to December 31 2017. For those with multiple visits, we selected the last visit per patient to maximize the statistical power for the cardiac arrest analysis. Cardiac arrest may result in death during an ED visit, and that visit became the last visit for the patient. Besides, patients younger than 18 years of age or those who presented with out-of-hospital cardiac arrest (OHCA) were excluded. The subject selection process is shown in Fig. 1. ED-based IHCA was defined as the cardiac arrest that occurred in the ED. This study is a retrospective study of medical records and all experimental protocols have been approved by Institutional Review Board (IRB) of FEMH (IRB number: 108026-E) and was conducted in compliance with the Declaration of Helsinki in 1964. The informed consent was waived by the IRB of FEMH.
Data collection. The patient's demographics and time-stamped clinical information at triage were retrieved, including chief complaint on presentation, mode of arrival, transfer status, vital signs (temperature, heart rate, systolic and diastolic blood pressure, respiratory rate, oxygen saturation), and levels of consciousness coded as Glasgow Coma Scale (GCS). The data extractors were hospital information technology engineers who were blinded to the study hypothesis. After the investigators' meetings, the data underwent electronic cleaning, and invalid data were designated as missing values. Pain scores were evaluated on a numeric rating scale (NRS) of 0 to 10, where 0 indicated no pain and 10 indicated the worst pain imaginable. We further categorized the NRS scores into no (0), mild (1-3), moderate (4)(5)(6), and severe (7-10) pain 19 . We also classified levels of consciousness as severe coma (GCS ≤ 8), moderate coma (9)(10)(11)(12), and minor coma to normal status (GCS ≥ 13) 20 . Emergency department shifts were classified as day (07:00-14:59), evening (15:00-22:59), and night (23:00-06:59). We retrieved the five-level computerized Taiwan triage and acuity scale (TTAS), which contains information on 179 structured chief complaints. The chief complaints included OHCA, which was used to identify and  The EDICAS, an 8-item composite score with a possible range of 0 to 13, included age, arrival mode, and categorized vital signs with simple cutoff points (supplemental eTable 1). We fit the eight variables comprising the EDICAS in a multivariable logistic regression model to examine the independent association between each variable and the ED-based IHCA. We also calculated each patient's EDICAS and computed its sensitivity and specificity with varying cutoff points. The discriminatory ability of the EDICAS was evaluated by using the area under the receiver operating curve (AUROC). For comparison purposes, we also computed the AUROCs by using other EWSs, including the Modified Early Warning Score (MEWS) 12 and Rapid Emergency Medicine Score (REMS) 23 . Finally, the test characteristics of the EDICAS were computed using varying cutoff points. We conducted two sensitivity analyses to test the robustness of our findings. First, we excluded patients with DNR status (n = 293) and repeated the AUROC analysis. Second, we tested the EDICAS performance separately in non-traumatic and trauma patients.
All odds ratios (ORs) and beta-coefficients are presented with 95% confidence intervals (CIs). All analyses were performed using Stata 16.0 software (StataCorp, College Station, TX). A 3-D figure for visualizing IHCA data was plotted using Python matplotlib v3.2.1. All P values are two-sided, and those less than 0.05 were considered to be statistically significant.

Results
Of the 268,208 ED visits over the 2-year study period, 184,184 unique patient visits were included in our study. After excluding those aged less than 18 years or with out-of-hospital cardiac arrest, 145,557 patient visits served as the study population, including 240 cases of IHCA and 145,317 cases of non-IHCA. The patient selection process is illustrated in Fig. 1. In our study population, the mean age of these patients was 47 years, and 52% were women. The overall incidence of ED-based IHCA was 0.16% (95% CI 0.14 to 0.19%). As shown in Table 1, patients with IHCA were much older, predominantly male, and were likely to present to the ED in the winter in comparison with non-IHCA patients. In terms of weekend or time of ED presentation, there were no significant differences between these two groups. Compared with non-IHCA patients, IHCA patients were more likely to arrive by ambulance, to be transferred from other facilities, and to present with dyspnea and chest pain. IHCA patients were less likely to present with injuries. IHCA patients were more likely to be categorized to higher triage levels, with altered mental status, but were less likely to express pain of any levels. Regarding vital signs at triage, IHCA patients were more likely to present with lower body temperature, lower heart and higher respiratory rates, and lower systolic blood pressure and oxygen saturation status. The median length of ED stay was about 4 h in the IHCA group, and was about 2 h in the non-IHCA group. Compared with non-IHCA patients, the admission and ED mortality rates were significantly higher among patients with IHCA.
The EDICAS, including age, arrival mode, and categorized vital signs with simple cutoffs, showed excellent discrimination with an AUROC of 0.88 when applied to the ED population (Fig. 3) (Fig. 4) showed a good agreement between the observed and predicted probability of IHCA within approximately 1% risk. A fair agreement was noted at a higher-risk range (> 1%) of IHCA.
The test characteristics of the EDICAS are shown in Table 3. An EDICAS of 6 or above (i.e., high-risk patients) corresponded to a sensitivity of 33%, a specificity of 97%, a positive likelihood ratio of 12.2, and overall accuracy of 97%.
In the sensitivity analyses, when we excluded the DNR patients, the AUROC did not materially change (0.88). Similarly, when we tested the EDICAS performance in medical and trauma patients, and its AUROC did not materially change (0.87 in medical and 0.88 in traumatic patients).

Discussion
In this ED-based study of 145,557 patients, we externally validated a triage tool, the EDICAS, for predicting imminent IHCA in the ED. Overall, this simple 8-item triage tool outperformed other previously established EWSs and remained to possess excellent discriminatory power in another ED population with a different casemix. The logical next step would be to embed the EDICAS in the EMR system with appropriate interventions to evaluate its real-world impact in the ED. Recent epidemiological evidence has suggested that the characteristics of ED-based IHCAs are quite different from those of ward-based IHCA 1,5 . Previous studies revealed that the median time to cardiac arrest was approximately two days in ward patients 2,23 . However, patients in the ED could develop IHCAs within hours after ED arrival. Such a short interval from ED arrival to cardiac arrest in these patients was commonly associated with the time-sensitive nature of certain emergent situations, such as acute coronary syndrome, acute respiratory distress, and so on. Indeed, chest pain and dyspnea were the most common presenting complaints in our IHCA population, a finding that is consistent with the original study. Hence, how to promptly recognize and appropriately manage these high-risk patients within hours is an important issue in the hectic ED environment. We externally validated the EDICAS in a different ED population, and its excellent performance was sustained, suggesting its clinical utility as a screening measure at ED triage for imminent IHCAs in the ED 11,12,24 . www.nature.com/scientificreports/   www.nature.com/scientificreports/ In the current study, our estimate of IHCA incidence was about 1.6 per 1,000 ED visits, which was similar to that in the original study (1.9 per 1,000 visits). In addition to the similar prevalence of IHCA, the test characteristics and the model performance were also remarkably similar in the current study. For example, an EDICAS of 6 or above was used to define the high-risk group in the original study, corresponding to a specificity of 98% and a positive likelihood ratio of 12.7. Using the same cutoff, the high-risk EDICAS corresponded to a specificity of 97% and a positive likelihood ratio of 12.2 in the current study. The AUROC of the EDICAS was 0.86 in the interval validation of the original study, and this figure even raised a bit to 0.88 in the current study. Taken together, these robust results suggested that the EDICAS could be used to raise the possibility of ED-based IHCA and to prompt the second level of physician review. Moreover, the EDICAS outperformed the general EWSs (MEWS and REMS) 25,26 , supporting its role as an ED-specific predictive tool for ED-based IHCA. Alternatively, the EDICAS can be used particularly in patients already triaged to levels 1 or 2, further raising its positive predictive value of IHCA.
There were several potential limitations in the study. First, as this external validation was done in another tertiary medical center, a prospective multicenter study would be ideal to further validate the EDICAS in different hospital settings. For example, differences in triage methods, ED length of stay, and emergency medical services systems exist between hospitals. Second, the real-world clinical impact was not addressed in the current study. The EDICAS may be integrated with the EMR system as clinical decision support with sending alerts to ED staff. It would be of interest to see whether there would be an increase in lead time to identify and intervene high-risk patients, thereby potentially preventing ED-based IHCA and associated deaths. Finally, as our study was retrospective in nature, the results were susceptible to selection and information bias.
In summary, we externally validated EDICAS, a novel 8-item ED triage tool predicting IHCA with the excellent discriminatory ability and test characteristics. We also demonstrated the superior performance of EDICAS over other pre-existing EWSs. The logical next step would be to embed the EDICAS in the EMR system with appropriate interventions to evaluate its real-world impact in the ED.