Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing

Abstract

Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments at early time points, which is critical for improving sepsis outcomes. In view of the increasing use of such systems, better understanding of how they are adopted and used by healthcare providers is needed. Here, we analyzed provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System), which was deployed at five hospitals over a 2-year period. Among 9,805 retrospectively identified sepsis cases, the early detection tool achieved high sensitivity (82% of sepsis cases were identified) and a high rate of adoption: 89% of all alerts by the system were evaluated by a physician or advanced practice provider and 38% of evaluated alerts were confirmed by a provider. Adjusting for patient presentation and severity, patients with sepsis whose alert was confirmed by a provider within 3 h had a 1.85-h (95% CI 1.66–2.00) reduction in median time to first antibiotic order compared to patients with sepsis whose alert was either dismissed, confirmed more than 3 h after the alert or never addressed in the system. Finally, we found that emergency department providers and providers who had previous interactions with an alert were more likely to interact with alerts, as well as to confirm alerts on retrospectively identified patients with sepsis. Beyond efforts to improve the performance of early warning systems, efforts to improve adoption are essential to their clinical impact and should focus on understanding providers’ knowledge of, experience with and attitudes toward such systems.

Your institute does not have access to this article

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Fig. 1: Included study population by study question.

Data availability

The data are not publicly available because they are from electronic health records approved for limited use by Johns Hopkins University investigators. Making the data publicly available without additional consent, ethical or legal approval might compromise patients’ privacy and the original ethical approval. To perform additional analyses using these data, researchers should contact A.W.W. or S.S. to apply for an institutional review board-approved research collaboration and obtain an appropriate data-use agreement.

Code availability

The TREWS early warning system described in this study is available from Bayesian Health. The underlying source code is proprietary intellectual property and is not available. Code for the primary statistical analyses can be found at https://github.com/royadams/henry_et_al_2022_code.

References

  1. Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A. & Escobar, G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff. 33, 1123–1131 (2014).

    Article  Google Scholar 

  2. Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44–56 (2019).

    CAS  PubMed  Article  Google Scholar 

  3. Liang, H. et al. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nat. Med. 24, 443–448 (2019).

    Google Scholar 

  4. Bauer, M. et al. Automation to optimise physician treatment of individual patients: examples in psychiatry. Lancet Psychiat. 6, 338–349 (2019).

    Article  Google Scholar 

  5. Beam, A. L. & Kohane, I. S. Big data and machine learning in health care. JAMA 319, 1317–1318 (2019).

    Article  Google Scholar 

  6. Castaneda, C. et al. Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine. J. Clin. Bioinforma. https://doi.org/10.1186/s13336-015-0019-3 (2015).

  7. Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122 (2015).

    PubMed  Article  Google Scholar 

  8. Churpek, M. M., Adhikari, R. & Edelson, D. P. The value of vital sign trends for detecting clinical deterioration on the wards. Resuscitation 102, 1–5 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  9. Amland, R. C. & Sutariya, B. B. Quick sequential (sepsis-related) organ failure assessment (qSOFA) and St John sepsis surveillance agent to detect patients at risk of sepsis: an observational cohort study. Am. J. Med. Qual. 33, 50–57 (2018).

    PubMed  Article  Google Scholar 

  10. Tomašev, N. et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature 572, 116–119 (2019).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  11. Sittig, D. F. et al. Grand challenges in clinical decision support. J. Biomed. Inform. 41, 387–392 (2008).

    PubMed  Article  Google Scholar 

  12. Abramoff, M. D., Tobey, D. & Char, D. S. Lessons learned about autonomous AI: finding a safe, efficacious, and ethical path through the development process. Am. J. Ophthalmol. 214, 134–142 (2020).

    PubMed  Article  Google Scholar 

  13. Khairat, S. et al. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med. Inform. https://doi.org/10.2196/medinform.8912 (2018).

  14. Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 1–9 (2019).

    CAS  Article  Google Scholar 

  15. Celi, L. A., Fine, B. & Stone, D. J. An awakening in medicine: the partnership of humanity and intelligent machines. Lancet Digit. Health 1, e255–e257 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  16. Mertz, L. From Annoying to appreciated: turning clinical decision support systems into a medical professional’s best friend. IEEE Pulse 6, 4–9 (2015).

    PubMed  Article  Google Scholar 

  17. Greenes, R. A. et al. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J. Biomed. Inform. 78, 134–143 (2018).

    PubMed  Article  Google Scholar 

  18. Wright, A. et al. Analysis of clinical decision support system malfunctions: a case series and survey. J. Am. Med. Inform. Assoc. 23, 1068–1076 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  19. Ruppel, H. & Liu, V. To catch a killer: electronic sepsis alert tools reaching a fever pitch? BMJ Qual. Saf. https://doi.org/10.1136/bmjqs-2019-009463 (2019).

  20. Downing, N. L. et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual. Saf. 28, 762–768 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  21. Giannini, H. M. et al. A machine learning algorithm to predict severe sepsis and septic shock. Crit. Care Med. https://doi.org/10.1097/ccm.0000000000003891 (2019).

  22. Moja, L. et al. Effectiveness of a hospital-based computerized decision support system on clinician recommendations and patient outcomes: a randomized clinical trial. JAMA Netw. Open 2, 1–16 (2019).

    Article  Google Scholar 

  23. Khan, S. et al. Improving provider adoption with adaptive clinical decision support surveillance: an observational study. JMIR Hum. Factors 6, 1–10 (2019).

    Article  Google Scholar 

  24. Mann, D. et al. Impact of clinical decision support on antibiotic prescribing for acute respiratory infections: a cluster randomized implementation trial. J. Gen. Intern. Med. 35, 788–795 (2020).

    PubMed  PubMed Central  Article  Google Scholar 

  25. Kwan, J. L. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ https://doi.org/10.1136/bmj.m3216 (2020).

  26. Escobar, G. J. et al. Automated identification of adults at risk for in-hospital clinical deterioration. N. Engl. J. Med. 383, 1951–1960 (2020).

    PubMed  PubMed Central  Article  Google Scholar 

  27. Schaefer, K. E., Chen, J. Y. C., Szalma, J. L. & Hancock, P. A. A meta-analysis of factors influencing the development of trust in automation: implications for understanding autonomy in future systems. Hum. Factors 58, 377–400 (2016).

    PubMed  Article  Google Scholar 

  28. Cai, C. J. et al. Human-centered tools for coping with imperfect algorithms during medical decision-making. Proc of the CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2019).

  29. Gaube, S. et al. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ Digit. Med. https://doi.org/10.1038/s41746-021-00385-9 (2021).

  30. Jacobs, M. et al. Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens. Proc of the Conference on Human Factors in Computing Systems (CHI), pp. 1–14 (2021).

  31. Shortliffe, E. H. & Sepúlveda, M. J. Clinical decision support in the era of artificial intelligence. JAMA 10025, 9–10 (2018).

    Google Scholar 

  32. Kumar, A. et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit. Care Med. 34, 1589–1596 (2006).

    PubMed  Article  Google Scholar 

  33. Seymour, C. W. et al. Time to treatment and mortality during mandated emergency care for sepsis. N. Engl. J. Med. 376, 2235–2244 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  34. Liu, V. X. et al. The timing of early antibiotics and hospital mortality in sepsis. Am. J. Respir. Crit. Care Med. 196, 856–863 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  35. Adams, R. et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat. Med. https://doi.org/10.1038/s41591-022-01894-0 (2022).

  36. Henry, K. E., Hager, D. N., Osborn, T. M., Wu, A. W. & Saria, S. Comparison of automated sepsis identification methods and electronic health record–based sepsis phenotyping: improving case identification accuracy by accounting for confounding comorbid conditions. Crit. Care Explor. https://doi.org/10.1097/cce.0000000000000053 (2019).

  37. Saria, S. & Henry, K. E. Too many definitions of sepsis: can machine learning leverage the electronic health record to increase accuracy and bring consensus? Crit. Care Med. https://doi.org/10.1097/CCM.0000000000004144 (2020).

  38. Singer, M. et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 315, 801–810 (2016).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  39. Levy, M. M., Evans, L. E. & Rhodes, A. The surviving sepsis campaign bundle: 2018 update. Crit. Care Med. 46, 997–1000 (2018).

    PubMed  Article  Google Scholar 

  40. Hospital Inpatient - Specifications Manuals - Sepsis Resources. QualityNet https://qualitynet.cms.gov/inpatient/specifications-manuals/sepsis-resources. (2021)

  41. Rhee, C. et al. Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals. JAMA Netw. Open 2, e187571 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  42. Adams, R. et al. 1405: assessing clinical use and performance of a machine learning sepsis alert for sex and racial bias. Crit. Care Med. 50, 705 (2022).

    Article  Google Scholar 

  43. Evans, I. V. R. et al. Association between the New York sepsis care mandate and in-hospital mortality for pediatric sepsis. JAMA https://doi.org/10.1001/jama.2018.9071 (2018).

  44. Guy, J. S., Jackson, E. & Perlin, J. B. Accelerating the clinical workflow using the sepsis prediction and optimization of therapy (SPOT) tool for real-time clinical monitoring. NEJM Catal. Innov. Care Deliv. https://doi.org/10.1056/CAT.19.1036 (2020).

  45. Perlin, J. B. et al. 2019 John M. Eisenberg patient safety and quality awards: SPOTting sepsis to save lives: a nationwide computer algorithm for early detection of sepsis: innovation in patient safety and quality at the national level (Eisenberg award). Jt. Comm. J. Qual. Patient Saf. 46, 381–391 (2020).

    PubMed  Google Scholar 

  46. McCoy, A. & Das, R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual. https://doi.org/10.1136/bmjoq-2017-000158 (2017).

  47. Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J. & Das, R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir. Res. https://doi.org/10.1136/bmjresp-2017-000234 (2017).

  48. Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern. Med. 48109, 1–6 (2021).

    Google Scholar 

  49. Sendak, M. P. et al. Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study. JMIR Med. Inform. 8, e15182 (2020).

    PubMed  PubMed Central  Article  Google Scholar 

  50. Khairat, S., Marc, D., Crosby, W. & Al Sanousi, A. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med. Informatics https://doi.org/10.2196/medinform.8912 (2018).

  51. Kitzmiller, R. R. et al. Diffusing an innovation: clinician perceptions of continuous predictive analytics monitoring in intensive care. Appl. Clin. Inform. https://doi.org/10.1055/s-0039-1688478 (2019).

  52. Abramoff, M. D., Lavin, P. T., Birch, M., Shah, N. & Folk, J. C. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit. Med. https://doi.org/10.1038/s41746-018-0040-6 (2018).

  53. Ginestra, J. C. et al. Clinician perception of a machine learning–based early warning system designed to predict severe sepsis and septic shock. Crit. Care Med. https://doi.org/10.1097/ccm.0000000000003803 (2019).

  54. Topiwala, R., Patel, K., Twigg, J., Rhule, J. & Meisenberg, B. Retrospective observational study of the clinical performance characteristics of a machine learning approach to early sepsis identification. Crit. Care Explor. https://doi.org/10.1097/cce.0000000000000046 (2019).

  55. Carspecken, C. W., Sharek, P. J., Longhurst, C. & Pageler, N. M. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics 131, e1970–e1973 (2013).

    PubMed  Article  Google Scholar 

  56. Bansal, G. et al. Beyond accuracy: the role of mental modelsin human–AI team performance. Proc. AAAI Conference on Human Computation and Crowdsourcing 7, 2–11. (2019).

  57. Jacobs, M. et al. How machine learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Transl. Psychiatry https://doi.org/10.1038/s41398-021-01224-x (2021).

  58. Lee, J. D. & See, K. A. Trust in automation: designing for appropriate reliance. Hum. Factors 46, 50–80 (2004).

    PubMed  Article  Google Scholar 

  59. Hoff, K. A. & Bashir, M. Trust in automation: integrating empirical evidence on factors that infuence trust. Hum. Factors 57, 407–434 (2015).

    PubMed  Article  Google Scholar 

  60. Murphy, E. V. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J. Biol. Med. 87, 187–197 (2014).

    PubMed  PubMed Central  Google Scholar 

  61. Mann, D. et al. Adaptive design of a clinical decision support tool: what the impact on utilization rates means for future CDS research. Digit. Heal. 5, 1–12 (2019).

    Google Scholar 

  62. Henry, K. E. et al. Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system. NPJ Digit. Med. https://doi.org/10.1038/s41746-022-00597-7 (2022).

  63. Rhee, C. et al. Infectious Diseases Society of America position paper: recommended revisions to the national severe sepsis and septic shock early management bundle (SEP-1) sepsis quality measure. Clin. Infect. Dis. 72, 541–552 (2021).

    PubMed  Article  Google Scholar 

  64. Rhee, C. et al. Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes. Crit. Care 20, 1–8 (2016).

    Article  Google Scholar 

  65. Rhee, C., Dantes, R. B., Epstein, L. & Klompas, M. Using objective clinical data to track progress on preventing and treating sepsis: CDC’s new ‘Adult Sepsis Event’ surveillance strategy. BMJ Qual. Saf. https://doi.org/10.1002/sca.20201 (2018).

  66. Rhee, C. et al. Sepsis surveillance using adult sepsis events simplified eSOFA criteria versus sepsis-3 sequential organ failure assessment criteria. Crit. Care Med. 47, 307–314 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  67. Jordan, M. I. & Jacobs, R. A. Hierarchical mixtures of experts and the EM algorithm. Neural Comput. 6, 181–214 (1994).

    Article  Google Scholar 

  68. Soleimani, H., Hensman, J. & Saria, S. Scalable joint models for reliable uncertainty-aware event prediction. IEEE Trans. Pattern Anal. Mach. Intell. 40, 1948–1963 (2018).

    PubMed  Article  Google Scholar 

  69. Schulam, P. & Saria, S. Can you trust this prediction? Auditing pointwise reliability subsequent to training. In Proc. of the 22nd International Conference on Artificial Intelligence and Statistics, 89:1022-1031 (2019).

  70. Pierson, E., Cutler, D. M., Leskovec, J., Mullainathan, S. & Obermeyer, Z. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nat. Med. 27, 136–140 (2021).

    CAS  PubMed  Article  Google Scholar 

  71. Subbaswamy, A. & Saria, S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 21, 345–352 (2020).

    PubMed  Google Scholar 

  72. Finlayson, S. G. et al. The clinician and dataset shift in artificial intelligence. N. Engl. J. Med. 385, 283 (2021).

    PubMed  PubMed Central  Article  Google Scholar 

  73. Subbaswamy, A., Schulam, P. & Saria, S. Preventing failures due to dataset shift: learning predictive models that transport. In Proc. of the International Conference on Artificial Intelligence and Statistics, PMLR 89:3118-3127 (2019).

  74. Henry, K., Wongvibulsin, S., Zhan, A., Saria, S. & Hager, D. Can septic shock be identified early? Evaluating performance of a targeted real-time early warning score (TREW score) for septic shock in a community hospital: global and subpopulation performance. Am. J. Respir. Crit. Care Med. 195, 7016 (2017).

    Google Scholar 

  75. Peltan, I. D. et al. ED door-to-antibiotic time and long-term mortality in sepsis. Chest 155, 938–946 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  76. Vincent, J. L. et al. The SOFA (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. Intensive Care Med. 22, 707–710 (1996).

    CAS  PubMed  Article  Google Scholar 

  77. Knaus, W. A., Draper, E. A., Wagner, D. P. & Zimmerman, J. E. APACHE II: a severity of disease classification system. Crit. Care Med. 13, 818–829 (1985).

    CAS  PubMed  Article  Google Scholar 

  78. Metcalfe, D. et al. Charlson and Elixhauser coding mapping. BMC Med. Res. Methodol. 19, 1–9 (2019).

    Article  Google Scholar 

  79. Seabold, S. & Perktold, J. statsmodels: econometric and statistical modeling with Python. Proc. 9th Python in Science Conference (2010).

  80. Norton, E. C., Miller, M. M. & Kleinman, L. C. Computing adjusted risk ratios and risk differences in Stata. Stata J. 13, 492–509 (2013).

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Y. Ahmad, A. Zhang, M. Yeo and Y. Karklin whose work significantly contributed to early iterations of the development of the deployed system. We also thank R. Demski, K. D’Souza, A. Kachalia, A. Chen and clinical and quality stakeholders who contributed to tool deployment, education and championing the work. The authors gratefully acknowledge the following sources of funding: the Gordon and Betty Moore Foundation (award 3926), the National Science Foundation Future of Work at the Human-technology Frontier (award 1840088) and the Alfred P. Sloan Foundation research fellowship (2018). This information or content and conclusions are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by the NSF the US Government.

Author information

Authors and Affiliations

Authors

Contributions

K.E.H., R.A., C.P., E.S.C., A.W.W. and S.S. contributed to the initial study design and preliminary analysis plan. S.S. led the development and deployment efforts for the TREWS software. K.E.H., H.S., A.S., R.C.L., L.J., M.H., S.M., D.N.H., A.W.W. and S.S. contributed to the system development and deployment. K.E.H., R.A., C.P., E.Y.K., S.E.C., A.R.C., E.S.C., D.N.H., A.W.W. and S.S. contributed to the review and analysis of the results. All authors contributed to the final preparation of the manuscript.

Corresponding authors

Correspondence to Albert W. Wu or Suchi Saria.

Ethics declarations

Competing interests

Under a license agreement between Bayesian Health and the Johns Hopkins University, K.E.H., S.S. and Johns Hopkins University are entitled to revenue distributions. Additionally, the University owns equity in Bayesian Health. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict-of-interest policies. S.S. also has grants from Gordon and Betty Moore Foundation, the National Science Foundation, the National Institutes of Health, Defense Advanced Research Projects Agency, the Food and Drug Administration and the American Heart Association; she is a founder of and holds equity in Bayesian Health; she is the scientific advisory board member for PatientPing; and she has received honoraria for talks from a number of biotechnology, research and health-tech companies. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict-of-interest policies. D.N.H. discloses salary support and funding to his institution from the Marcus Foundation for the conduct of the vitamin C, thiamine and steroids in sepsis trial. S.E.C. declares consulting fees from Basilea for work on an infection adjudication committee for an S.aureus bacteremia trial. The other authors declare no competing interests.

Peer review

Peer review information

Nature Medicine thanks Derek Angus, Melanie Wright and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling editor: Michael Basson, in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Retrospective predictive performance of the TREWS model.

Performance of the TREWS model on retrospective data. Figure (a) shows the receiver operating characteristic curve and Figure (b) shows the sensitivity–PPV curve (also referred to as the precision-recall curve).

Extended Data Fig. 2 Annotated screenshot of the TREWS interface.

Annotated screenshot of the TREWS provider evaluation page. Annotations show the main provider actions: reviewing the alert explanation, indicating whether the patient has a suspected source of infection and reviewing sources of organ dysfunction.

Extended Data Table 1 Time from alert to first antibiotic order among retrospectively identified sepsis patients
Extended Data Table 2 TREWS alert volume per day during the study period including re-alerts and alerts flagging patients who are candidates for escalation
Extended Data Table 3 Population characteristics
Extended Data Table 4 Model features

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Henry, K.E., Adams, R., Parent, C. et al. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med 28, 1447–1454 (2022). https://doi.org/10.1038/s41591-022-01895-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-022-01895-z

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing