Abstract
The early identification of vulnerable patients has the potential to improve outcomes but poses a substantial challenge in clinical practice. This study evaluated the ability of an artificial intelligence (AI)-enabled electrocardiogram (ECG) to identify hospitalized patients with a high risk of mortality in a multisite randomized controlled trial involving 39 physicians and 15,965 patients. The AI-ECG alert intervention included an AI report and warning messages delivered to the physicians, flagging patients predicted to be at high risk of mortality. The trial met its primary outcome, finding that implementation of the AI-ECG alert was associated with a significant reduction in all-cause mortality within 90 days: 3.6% patients in the intervention group died within 90 days, compared to 4.3% in the control group (4.3%) (hazard ratio (HR) = 0.83, 95% confidence interval (CI) = 0.70–0.99). A prespecified analysis showed that reduction in all-cause mortality associated with the AI-ECG alert was observed primarily in patients with high-risk ECGs (HR = 0.69, 95% CI = 0.53–0.90). In analyses of secondary outcomes, patients in the intervention group with high-risk ECGs received increased levels of intensive care compared to the control group; for the high-risk ECG group of patients, implementation of the AI-ECG alert was associated with a significant reduction in the risk of cardiac death (0.2% in the intervention arm versus 2.4% in the control arm, HR = 0.07, 95% CI = 0.01–0.56). While the precise means by which implementation of the AI-ECG alert led to decreased mortality are to be fully elucidated, these results indicate that such implementation assists in the detection of high-risk patients, prompting timely clinical care and reducing mortality. ClinicalTrials.gov registration: NCT05118035.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
Patient data cannot be made publicly available due to privacy concerns. De-identified tabular data can be obtained from the corresponding author on approval from the ethics committee of the Tri-Service General Hospital. Approval from this committee can be requested from the Tri-Service General Hospital’s Clinical Trial Management System (https://tsgh.cims.tw/wiPtms/index.html), with an expected review period of approximately 2–3 months. After approval, researchers will be granted VPN access to perform analyses, ensuring data security and confidentiality (summary data can be exported), with measures in place to prevent any breach of personal information.
Code availability
The model weights of the AI algorithm used in this study cannot be made publicly available due to the proprietary nature of the algorithm. However, the computer code for training is available from GitHub: https://github.com/Imshepherd/ECGSurvNet. This code can be used to train a survival deep learning model using public ECG databases with survival information, such as SaMi-Trop (https://doi.org/10.5281/zenodo.4905617)36 and CODE-15% (https://doi.org/10.5281/zenodo.4916205)37, both available from Zenodo.
References
Adhikari, N. K., Fowler, R. A., Bhagwanjee, S. & Rubenfeld, G. D. Critical care and the global burden of critical illness in adults. Lancet 376, 1339–1346 (2010).
Pronovost, P. J. et al. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA 288, 2151–2162 (2002).
Chalfin, D. B., Trzeciak, S., Likourezos, A., Baumann, B. M. & Dellinger, R. P. Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Crit. Care Med. 35, 1477–1483 (2007).
Hodgetts, T. J. et al. Incidence, location and reasons for avoidable in-hospital cardiac arrest in a district general hospital. Resuscitation 54, 115–123 (2002).
Jones, D. A., DeVita, M. A. & Bellomo, R. Rapid-response teams. N. Engl. J. Med. 365, 139–146 (2011).
Goldstein, B. A., Navar, A. M., Pencina, M. J. & Ioannidis, J. P. A. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J. Am. Med. Inform. Assoc. 24, 198–208 (2017).
Smith, G. B., Prytherch, D. R., Schmidt, P. E., Featherstone, P. I. & Higgins, B. A review, and performance evaluation, of single-parameter ‘track and trigger’ systems. Resuscitation 79, 11–21 (2008).
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E. & Featherstone, P. I. The ability of the national early warning score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 84, 465–470 (2013).
Subbe, C. P., Kruger, M., Rutherford, P. & Gemmel, L. Validation of a modified Early Warning Score in medical admissions. QJM 94, 521–526 (2001).
Mann, K. D. et al. Predicting patient deterioration: a review of tools in the digital hospital setting. J. Med. Internet Res. 23, e28209 (2021).
Romero-Brufau, S. et al. Widely used track and trigger scores: are they ready for automation in practice? Resuscitation 85, 549–552 (2014).
Ludikhuize, J. et al. Standardized measurement of the Modified Early Warning Score results in enhanced implementation of a rapid response system: a quasi-experimental study. Resuscitation 85, 676–682 (2014).
Chan, P. S., Jain, R., Nallmothu, B. K., Berg, R. A. & Sasson, C. Rapid response teams: a systematic review and meta-analysis. Arch. Intern. Med. 170, 18–26 (2010).
Tsai, D.-J. et al. Mortality risk prediction of the electrocardiogram as an informative indicator of cardiovascular diseases. Digit. Health 9, 20552076231187247 (2023).
Raghunath, S. et al. Prediction of mortality from 12-lead electrocardiogram voltage data using a deep neural network. Nat. Med. 26, 886–891 (2020).
Plana, D. et al. Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw. Open 5, e2233946 (2022).
Attia, Z. I. et al. Screening for cardiac contractile dysfunction using an artificial intelligence-enabled electrocardiogram. Nat. Med. 25, 70–74 (2019).
Attia, Z. I. et al. An artificial intelligence-enabled ECG algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction. Lancet 394, 861–867 (2019).
Downey, C. L., Tahir, W., Randell, R., Brown, J. M. & Jayne, D. G. Strengths and limitations of early warning scores: a systematic review and narrative synthesis. Int. J. Nurs. Stud. 76, 106–119 (2017).
van der Sijs, H., Aarts, J., Vulto, A. & Berg, M. Overriding of drug safety alerts in computerized physician order entry. J. Am. Med. Inform. Assoc. 13, 138–147 (2006).
Bedoya, A. D. et al. Minimal impact of implemented Early Warning Score and best practice alert for patient deterioration. Crit. Care Med. 47, 49–55 (2019).
Embi, P. J. & Leonard, A. C. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J. Am. Med. Inform. Assoc. 19, e145–e148 (2012).
Lyons, P. G., Edelson, D. P. & Churpek, M. M. Rapid response systems. Resuscitation 128, 191–197 (2018).
Iung, B. & Vahanian, A. Epidemiology of valvular heart disease in the adult. Nat. Rev. Cardiol. 8, 162–172 (2011).
de Lemos, J. A., McGuire, D. K. & Drazner, M. H. B-type natriuretic peptide in cardiovascular disease. Lancet 362, 316–322 (2003).
Kollef, M. H. et al. A randomized trial of real-time automated clinical deterioration alerts sent to a rapid response team. J. Hosp. Med. 9, 424–429 (2014).
Perkins, G. D., Temple, R. M. & George, R. Time to intervene: lessons from the NCEPOD report. Resuscitation 83, 1305–1306 (2012).
Yao, X. et al. Artificial intelligence-enabled electrocardiograms for identification of patients with low ejection fraction: a pragmatic, randomized clinical trial. Nat. Med. 27, 815–819 (2021).
Austrian, J. et al. Applying A/B testing to clinical decision support: rapid randomized controlled trials. J. Med. Internet Res. 23, e16651 (2021).
Horwitz, L. I., Kuznetsova, M. & Jones, S. A. Creating a learning health system through rapid-cycle, randomized testing. N. Engl. J. Med. 381, 1175–1179 (2019).
Simon, G. E., Platt, R. & Hernandez, A. F. Evidence from pragmatic trials during routine care—slouching toward a learning health system. N. Engl. J. Med. 382, 1488–1491 (2020).
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J. & Denniston, A. K. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat. Med. 26, 1364–1374 (2020).
Crawford, M. H. et al. ACC/AHA Guidelines for Ambulatory Electrocardiography. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to Revise the Guidelines for Ambulatory Electrocardiography). Developed in collaboration with the North American Society for Pacing and Electrophysiology. J. Am. Coll. Cardiol. 34, 912–948 (1999).
Attia, Z. I., Harmon, D. M., Behr, E. R. & Friedman, P. A. Application of artificial intelligence to the electrocardiogram. Eur. Heart J. 42, 4717–4730 (2021).
Huang, P.-F., Kung, P.-T., Chou, W.-Y. & Tsai, W.-C. Characteristics and related factors of emergency department visits, readmission, and hospital transfers of inpatients under a DRG-based payment system: a nationwide cohort study. PLoS ONE 15, e0243373 (2020).
Ribeiro, A. L. P. et al. Sami-Trop: 12-lead ECG traces with age and mortality annotations. Zenodo https://doi.org/10.5281/zenodo.4905617 (2021).
Ribeiro, A. H. et al. CODE-15%: a large scale annotated dataset of 12-lead ECGs. Zenodo https://doi.org/10.5281/zenodo.4916205 (2021).
Acknowledgements
This study was supported by funding from the National Science and Technology Council, Taiwan (NSTC110-2314-B-016-010-MY3 to C.L.; NSTC112-2321-B-016-003 to S.-H.L.), the Cheng Hsin General Hospital, Taiwan (CHNDMC-113-01 to C.-S.L.; CHNDMC-113-11205 to C.L.) and the Medical Affairs Bureau, Taiwan (MND-MAB-C07-113022, MND-MAB-C13-112051, MND-MAB-C08-111032 and MND-MAB-C08-11032 to C.-S.L.; MND-MAB-110-113, MND-MAB-D-111045, MND-MAB-C13-112050 and MND-MAB-C07-113021 to C.L.). We thank Y.-C. Wong for English editing and final proofreading of the manuscript.
Author information
Authors and Affiliations
Contributions
C.-S.L. and C.L. conceived and designed the study. D.-J.T. and Y.-S.L. acquired the data. C.-S.L. and C.L. analyzed the data. C.-S.L., W.-T.L. and C.L. interpreted the data. W.-T.L., C.-H.C., W.-S.L., C.-C.C. and Chiao-Chin Lee reviewed the medical records. C.-C.W. and Y.-Y.C. provided expert opinion on medical ethics when designing the study. D.-J.T., Chia-Cheng Lee and C.L. provided a deep learning model to stratify a high risk of mortality. W.-H.F., Chia-Cheng Lee, C.-H.W., C.-S.T. and S.-H.L. integrated the deep learning model with the hospital information system. C.-S.L. and C.L. drafted the initial manuscript. W.-T.L., W.-H.F. and S.-H.L. revised the manuscript for important intellectual content. C.L. takes final responsibility for the manuscript and provided final approval for the version to be published.
Corresponding author
Ethics declarations
Competing interests
The National Defense Medical Center has granted Quanta Computer, originally a personal computer and cloud server manufacturer, a license for its AI-ECG algorithm as part of Quanta’s shift toward investing in smart hospital information systems. The National Defense Medical Center will not receive any financial gains from deploying the AI-ECG technology in patient care across Taiwan’s military hospitals. Financial benefits from using the AI-ECG outside these military hospitals may accrue to C.-S.L., W.-T.L., Y.-S.L., C.-H.C., Chiao-Chin Lee, W.-H.F., W.-S.L., C.-C.C., Chia-Cheng Lee, C.-H.W., C.-S.T., S.-H.L. and C.L. The remaining authors declare no competing interests.
Peer review
Peer review information
Nature Medicine thanks Zachi Attia, Rahul Deo, Fu Siong Ng, Jill Waalen and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Michael Basson, in collaboration with the Nature Medicine team.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 The association between the AI-ECG predicted all-cause mortality risk score and traditional risk features.
(a) Scatter plot and Spearman’s rho coefficient were used to examine the relationship. In this analysis, the ECG-risk score was transformed into percentiles (PR), with a PR of 95 indicating that the patient’s ECG-risk score is higher than 95% of individuals, which was the threshold to send AI-ECG alert in this trial. These PR values were determined based on previous research based on all population. Therefore, there were approximately 10% patients in emergency department (ED) and inpatient department (IPD) with an ECG-risk score higher than 95 because their conditions were collectively worse than the overall population. Age, MEWS (Modified Early Warning Score), and heart rate (HR) were chosen as they exhibited the highest correlation. The scatter plot color-codes the data points, with red indicating the highest density, followed by yellow, green, light blue, and dark blue. To better present MEWS on the scatter plot, we assigned a random number (without changing the rank) to all values during plotting. (b) A segmented scatter plot. As the mortality risk is extremely low for patients with a PR < 75 reported by a previous study, we performed a stratified analysis for this group (even if PR75 is not the cutoff point to send alarm message in this study, it still distinguishes between patients with low risk and median-to-high risk among these features). It can be observed that the correlation between age and ECG-risk score mainly occurs among relatively low-risk patients, while MEWS and HR are associated with relatively high-risk patients.
Extended Data Fig. 2 The comparison between AI-ECG and patient data.
(a) The components of AI-ECG identified high-risk group. We trained three xgboost models using patient characteristics, ECG features, and combination of them to predict AI-ECG results. The bars are the related importance of the components to predict AI-ECG. (b) The prediction abilities of all patient data on the AI-ECG results (AUC). The error bars are the 95% confidence intervals (CI) of each AUC. c) The prediction abilities of all patient data on all-cause mortality within 90 days (C-index). The error bars are the 95% confidence intervals (CI) of each AUC. The sky-blue and light-red bars represent the results of prediction using individual patient characteristics and ECG features, respectively. The blue-green bars represent predictions integrating features from xgboost and logistic regression. We used the continuous value of AI-ECG in this analysis and presented it in the gray bar. All analyses were based on the entire population data in this trial (n = 15,965).
Extended Data Fig. 3 Performance of AI-ECG on risk groups stratified by causes of death in the control group.
Cox proportional hazard models were used for the statistical test; this was two-sided, with no adjustment for multiple comparison. The hazard ratios (HRs) were adjusted by age and sex. Red line and green line represent high risk [indicating an AI-ECG prediction greater than the operational cutoff (PR95)] and low risk [indicating an AI-ECG prediction less than the operational cutoff (PR95)] groups, respectively. The table shows the at-risk population and cumulative risk for the given time intervals in each risk group. In the analysis of causes of death, the patients died due to other cause were considered as censored data. The exact p values were 2.5 × 10−87 (all-cause mortality), 1.1 × 10−10 (all cardiac mortality), 4.4 × 10−78 (all non-cardiac mortality), 2.1 × 10−6 (myocardial infarction death), 2.3 × 10−25 (cancer death), 1.8 × 10−43 (sepsis death), and 5.3 × 10−13 (other non-cardiac death), respectively.
Extended Data Fig. 4 AI-ECG risk groups on subsequent heart rate of ≥ 110 bpm, new-onset atrial fibrillation, and new-onset heart failure in the control group.
Cox proportional hazard models were used for the statistical test; this was two-sided, with no adjustment for multiple comparison. The hazard ratios (HRs) were adjusted by age and sex. Red line and green line represent high risk [indicating an AI-ECG prediction greater than the operational cutoff (PR95)] and low risk [indicating an AI-ECG prediction less than the operational cutoff (PR95)] groups, respectively. The table shows the at-risk population and cumulative risk for the given time intervals in each risk group. For subsequent heart rate of ≥110 bpm, we only included patients with heart rate of <100 bpm on index ECG. For new-onset atrial fibrillation, we only included patients with sinus rhythm on index ECG and without history of atrial fibrillation. For new-onset heart failure, we only included patients without history of heart failure. The exact p values were 6.2 × 10−20 (subsequent heart rate ≥110), 1.4 × 10−31 (new-onset atrial fibrillation), and 3.9 × 10−39 (new-onset heart failure), respectively.
Extended Data Fig. 5 The pre-specified secondary analysis for effectiveness of AI-ECG for detailed causes of death.
The black square represents the point estimate of the HR, while the error bars indicate the 95% confidence intervals (CI). Cox proportional hazard mixed effect models were used for the statistical test; this was two-sided, with no adjustment for multiple comparison.
Supplementary information
Supplementary Information
Supplementary Tables 1–5, Figs. 1 and 2 and Note 1.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lin, CS., Liu, WT., Tsai, DJ. et al. AI-enabled electrocardiography alert intervention and all-cause mortality: a pragmatic randomized clinical trial. Nat Med (2024). https://doi.org/10.1038/s41591-024-02961-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41591-024-02961-4
