Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis

Nature Medicine volume 28, pages 1455–1460 (2022)Cite this article

Subjects

Abstract

Early recognition and treatment of sepsis are linked to improved patient outcomes. Machine learning-based early warning systems may reduce the time to recognition, but few systems have undergone clinical evaluation. In this prospective, multi-site cohort study, we examined the association between patient outcomes and provider interaction with a deployed sepsis alert system called the Targeted Real-time Early Warning System (TREWS). During the study, 590,736 patients were monitored by TREWS across five hospitals. We focused our analysis on 6,877 patients with sepsis who were identified by the alert before initiation of antibiotic therapy. Adjusting for patient presentation and severity, patients in this group whose alert was confirmed by a provider within 3 h of the alert had a reduced in-hospital mortality rate (3.3%, confidence interval (CI) 1.7, 5.1%, adjusted absolute reduction, and 18.7%, CI 9.4, 27.0%, adjusted relative reduction), organ failure and length of stay compared with patients whose alert was not confirmed by a provider within 3 h. Improvements in mortality rate (4.5%, CI 0.8, 8.3%, adjusted absolute reduction) and organ failure were larger among those patients who were additionally flagged as high risk. Our findings indicate that early warning systems have the potential to identify sepsis patients early and improve patient outcomes and that sepsis patients who would benefit the most from early treatment can be identified and prioritized at the time of the alert

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Waterfall diagram for the primary analysis cohort.
Fig. 2: Distribution of the time from which the TREWS alert was given to the time confirmation was entered for the target population.

Similar content being viewed by others

Data availability

The data are not publicly available because they are from EHRs approved for limited use by Johns Hopkins University investigators. Making the data publicly available without additional consent, ethical or legal approval might compromise patients’ privacy and the original ethical approval. To perform additional analyses using this data, researchers should contact A.W.W. or S.S. to apply for an IRB-approved research collaboration and obtain an appropriate data use agreement.

Code availability

The TREWS early warning system described in the present study is available from Bayesian Health, New York. The underlying source code is proprietary intellectual property and is not available. Code for the primary statistical analyses and development of the high-risk cohort can be found at https://github.com/royadams/adams_et_al_2022_code.

References

  1. Rhee, C. et al. Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals. JAMA Netw. Open 2, e187571–e187571 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Riedemann, N. C., Guo, R. F. & Ward, P. A. The enigma of sepsis. J. Clin. Invest. 112, 460–467 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Marshall, J. C. Why have clinical trials in sepsis failed? Trends Mol. Med. 20, 195–203 (2014).

    Article  PubMed  Google Scholar 

  4. Rhodes, A. et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Crit. Care Med. 43, 304–377 (2017).

  5. Kumar, A. et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit. Care Med. 34, 1589–1596 (2006).

    Article  PubMed  Google Scholar 

  6. Ferrer, R. et al. Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program. Crit. Care Med. 42, 1749–1755 (2014).

    Article  CAS  PubMed  Google Scholar 

  7. Liu, V. X. et al. The timing of early antibiotics and hospital mortality in sepsis. Am. J. Respir. Crit. Care Med. 196, 856–863 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Peltan, I. D. et al. ED door-to-antibiotic time and long-term mortality in sepsis. Chest 155, 938–946 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Chamberlain, D. J., Willis, E. M. & Bersten, A. B. The severe sepsis bundles as processes of care: a meta-analysis. Aust. Crit. Care 24, 229–243 (2011).

    Article  PubMed  Google Scholar 

  10. Damiani, E. et al. Effect of performance improvement programs on compliance with sepsis bundles and mortality: a systematic review and meta-analysis of observational studies. PLoS ONE 10, e0125827 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Giannini, H. M. et al. A machine learning algorithm to predict severe sepsis and septic shock: development, implementation, and impact on clinical practice. Crit. Care Med. 47, 1485–1492 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  12. Desautels, T. et al. Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med. Inform. 4, 1–15 (2016).

    Article  Google Scholar 

  13. Shashikumar, S. P., Josef, C. S., Sharma, A. & Nemati, S. DeepAISE—an interpretable and recurrent neural survival model for early prediction of sepsis. Artif. Intell. Med. 113, 102036 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Horng, S. et al. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS ONE 12, e0174708 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  15. Bedoya, A. D. et al. Machine learning for early detection of sepsis: an internal and temporal validation study. JAMIA Open 3, 252–260 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J. & Das, R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir. Res. 4, e000234 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  17. McCoy, A. & Das, R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual. 6, e000158 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  18. Escobar, G. J. et al. Automated identification of adults at risk for in-hospital clinical deterioration. N. Engl. J. Med. 383, 1951–1960 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  19. Topiwala, R., Patel, K., Twigg, J., Rhule, J. & Meisenberg, B. Retrospective observational study of the clinical performance characteristics of a machine learning approach to early sepsis identification. Crit. Care Explor. 1, e0046 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ginestra, J. C. et al. Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock. Crit. Care Med. 47, 1477 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122–299ra122 (2015).

    Article  PubMed  Google Scholar 

  22. Henry, K. E. et al. Factors driving provider adoption of the TREWS machine-learning-based early warning system and its effects on sepsis treatment timing. Nat. Med. https://doi.org/10.1038/s41591-022-01895-z (2022).

  23. Henry, K. E., Hager, D. N., Osborn, T. M., Wu, A. W. & Saria, S. Comparison of automated sepsis identification methods and electronic health record-based sepsis phenotyping: improving case identification accuracy by accounting for confounding comorbid conditions. Crit. Care Explor. 1, e0053 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  24. Rhee, C. et al. Infectious diseases society of america position paper: recommended revisions to the national severe sepsis and septic shock early management bundle (SEP-1) sepsis quality measure. Clin. Infect. Dis. 72, 541–552 (2021).

    Article  PubMed  Google Scholar 

  25. Seymour, C. W. et al. Time to treatment and mortality during mandated emergency care for sepsis. N. Engl. J. Med. 376, 2235–2244 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Vanderweele, T. J., Luedtke, A. R., Van Der Laan, M. J. & Kessler, R. C. Selecting optimal subgroups for treatment using many covariates. Epidemiology 30, 334–341 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Manaktala, S. & Claypool, S. R. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J. Am. Med. inform. Assoc. 24, 88–95 (2017).

    Article  PubMed  Google Scholar 

  28. Burdick, H. et al. Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals. BMJ Health Care Inform. 27, e100109 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  29. Guy, J. S., Jackson, E. & Perlin, J. B. Accelerating the clinical workflow using the sepsis prediction and optimization of therapy (SPOT) tool for real-time clinical monitoring. NEJM Catal. Innov. Care Deliv. https://doi.org/10.1056/CAT.19.1036 (2020).

  30. Rosenbaum, P. R. & Briskman. Design of Observational Studies Vol. 10 (Springer, 2010).

  31. Hernán, M. A. & Robins, J. M. Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183, 758–764 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern. Med. 48109, 1–6 (2021).

    Google Scholar 

  33. Henry, K. E. et al. Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system. NPJ Digit. Med. https://doi.org/10.1038/s41746-022-00597-7 (2022).

  34. Saria, S. & Henry, K. E. Too many definitions of sepsis: Can machine learning leverage the electronic health record to increase accuracy and bring consensus? Crit. Care Med. 48, 137–141 (2020). https://doi.org/10.1097/CCM.0000000000004144

  35. Rhee, C. et al. Prevalence of antibiotic-resistant pathogens in culture-proven sepsis and outcomes associated with inadequate and broad-spectrum empiric antibiotic use. JAMA Netw. Open 3, e202899 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  36. Jordan, M. I. & Jacobs, R. A. Hierarchical mixtures of experts and the EM algorithm. Proceedings of International Conference on Neural Networks 2, 1339–1344 (1993).

    Google Scholar 

  37. Seymour, C. W. et al. Assessment of clinical criteria for sepsis for the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 315, 762–774 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Rhee, C., Dantes, R. B., Epstein, L. & Klompas, M. Using objective clinical data to track progress on preventing and treating sepsis: CDC’s new adult sepsis event surveillance strategy. BMJ Qual. Saf. 28, 305–309 (2019).

    Article  PubMed  Google Scholar 

  39. Vincent, J. L. et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Intens. Care Med. 22, 707–710 (1996).

    Article  CAS  Google Scholar 

  40. Knaus, W. A., Draper, E. A., Wagner, D. P. & Zimmerman, J. E. APACHE II: a severity of disease classification system. Crit. Care Med. 13, 818–829 (1985).

    Article  CAS  PubMed  Google Scholar 

  41. Norton, E. C., Miller, M. M. & Kleinman, L. C. Computing adjusted risk ratios and risk differences in Stata. Stata J. 13, 492–509 (2013).

    Article  Google Scholar 

  42. Peng, L. Quantile regression for survival data. Annu. Rev. Stat. Its Appl. 8, 413–437 (2021).

    Article  Google Scholar 

  43. Seabold, S. & Perktold, J. statsmodels: econometric and statistical modeling with python. In van der Walt, S. & Millman, J. (Eds.) Proc. 9th Python in Science Conference 92–96 (2010).

  44. Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).

    Google Scholar 

  45. Horvitz, D. G. & Thompson, D. J. A generalization of sampling without replacement from a finite universe. J. Am. Stat. Assoc. 47, 663–685 (1952).

    Article  Google Scholar 

  46. Robins, J. M. Marginal structural models versus structural nested models as tools for causal inference. In Halloran, M. E. & Berry, D. (Eds.) Statistical Models in Epidemiology, the Environment, and Clinical Trials 95–133 (Springer, 2000).

  47. Hernán, M. A. & Robins, J. M. Causal Inference: What If (Chapman & Hall/CRC, 2020).

  48. Lee, B. K., Lessler, J. & Stuart, E. A. Weight trimming and propensity score weighting. PLoS ONE 6, e18174 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. World Health Organization. ICD-10 : international statistical classification of diseases and related health problems : tenth revision (World Health Organization, 2004).

Download references

Acknowledgements

We thank Y. Ahmad, M. Yeo and Y. Karklin whose work significantly contributed to early iterations of the development of the deployed system. Further, we thank R. Demski, K. D’Souza, A. Kachalia, A. Chen and clinical and quality stakeholders who contributed to tool deployment, education and championing the work. We gratefully acknowledge the following sources of funding: the Gordon and Betty Moore Foundation (award no. 3926), the National Science Foundation (NSF) Future of Work at the Human-technology Frontier (award no. 1840088) and the Alfred P. Sloan Foundation research fellowship (2018). This information or content and conclusions are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by, the NSF of the US Government.

Author information

Authors and Affiliations

  1. Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, MD, USA

    Roy Adams & Suchi Saria

  2. Department of Psychiatry and Behavioral Science, Johns Hopkins School of Medicine, Baltimore, MD, USA

    Roy Adams, Katharine E. Henry & Andong Zhan

  3. Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

    Katharine E. Henry, Andong Zhan & Suchi Saria

  4. Howard County General Hospital, Columbia, MD, USA

    Anirudh Sridharan & Robert C. Linton

  5. Health Informatics, University of California, San Francisco, CA, USA

    Hossein Soleimani

  6. Armstrong Institute for Patient Safety and Quality, Johns Hopkins School of Medicine, Baltimore, MD, USA

    Nishi Rawat & Albert W. Wu

  7. Department of Quality Improvement, Johns Hopkins Hospital, Baltimore, MD, USA

    Lauren Johnson, Maureen Henley, Katrina Houston & Anushree R. Ahluwalia

  8. Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA

    David N. Hager, Sara E. Cosgrove, Edward S. Chen, Albert W. Wu & Suchi Saria

  9. Suburban Hospital, Bethesda, MD, USA

    Andrew Markowski

  10. Department of Emergency Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA

    Eili Y. Klein & Mustapha O. Saheed

  11. Department of Medicine, Johns Hopkins Hospital, Baltimore, MD, USA

    Sheila Miranda

  12. Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Albert W. Wu & Suchi Saria

  13. Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Albert W. Wu

  14. Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Albert W. Wu

  15. Bayesian Health, New York, NY, USA

    Suchi Saria

Authors
  1. Roy Adams
    View author publications

    Search author on:PubMed Google Scholar

  2. Katharine E. Henry
    View author publications

    Search author on:PubMed Google Scholar

  3. Anirudh Sridharan
    View author publications

    Search author on:PubMed Google Scholar

  4. Hossein Soleimani
    View author publications

    Search author on:PubMed Google Scholar

  5. Andong Zhan
    View author publications

    Search author on:PubMed Google Scholar

  6. Nishi Rawat
    View author publications

    Search author on:PubMed Google Scholar

  7. Lauren Johnson
    View author publications

    Search author on:PubMed Google Scholar

  8. David N. Hager
    View author publications

    Search author on:PubMed Google Scholar

  9. Sara E. Cosgrove
    View author publications

    Search author on:PubMed Google Scholar

  10. Andrew Markowski
    View author publications

    Search author on:PubMed Google Scholar

  11. Eili Y. Klein
    View author publications

    Search author on:PubMed Google Scholar

  12. Edward S. Chen
    View author publications

    Search author on:PubMed Google Scholar

  13. Mustapha O. Saheed
    View author publications

    Search author on:PubMed Google Scholar

  14. Maureen Henley
    View author publications

    Search author on:PubMed Google Scholar

  15. Sheila Miranda
    View author publications

    Search author on:PubMed Google Scholar

  16. Katrina Houston
    View author publications

    Search author on:PubMed Google Scholar

  17. Robert C. Linton
    View author publications

    Search author on:PubMed Google Scholar

  18. Anushree R. Ahluwalia
    View author publications

    Search author on:PubMed Google Scholar

  19. Albert W. Wu
    View author publications

    Search author on:PubMed Google Scholar

  20. Suchi Saria
    View author publications

    Search author on:PubMed Google Scholar

Contributions

K.E.H., R.A., A.W.W. and S.S. contributed to the initial study design and preliminary analysis plan. S.S. led the development and deployment efforts for the TREWS software. K.E.H., H.S., N.R., A.Z., A.S., R.C.L., L.J., M.H., S.M., D.N.H., A.R.A., A.W.W. and S.S. contributed to the system deployment. K.E.H., R.A., E.Y.K., S.E.C., E.S.C., D.N.H., A.W.W. and S.S. contributed to the review and analysis of the results. All authors contributed to the final preparation of the manuscript.

Corresponding authors

Correspondence to Albert W. Wu or Suchi Saria.

Ethics declarations

Competing interests

Under a license agreement between Bayesian Health and the Johns Hopkins University, K.E.H., S.S. and Johns Hopkins University are entitled to revenue distributions. In addition, the university owns equity in Bayesian Health. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict-of-interest policies. S.S. also has grants from the Gordon and Betty Moore Foundation, the NSF, the National Institutes of Health, Defense Advanced Research Projects Agency, the Food and Drug Administration and the American Heart Association; she is a founder of and holds equity in Bayesian Health; she is the scientific advisory board member for PatientPing; and she has received honoraria for talks from a number of biotechnology, research and health-tech companies. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict-of-interest policies. D.N.H. discloses salary support and funding to his institution from the Marcus Foundation for the conduct of the vitamin C, thiamine and steroids in sepsis trial. S.E.C. received consulting fees from Basilea for work on an infection adjudication committee for a Staphylococcus aureus bacteremia trial. The remaining authors declare no disclosures of conflicts of interest.

Peer review

Peer review information

Nature Medicine thanks Derek Angus, Melanie Wright and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling editor: Michael Basson in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Table 1 Sample statistics by hospital
Full size table
Extended Data Table 2 Extended sample statistics
Full size table
Extended Data Table 3 Sensitivity analyses
Full size table
Extended Data Table 4 Associations between antibiotics timing and patient outcomes
Full size table
Extended Data Table 5 List of included antibiotics
Full size table

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Adams, R., Henry, K.E., Sridharan, A. et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med 28, 1455–1460 (2022). https://doi.org/10.1038/s41591-022-01894-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-022-01894-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing