Impact of a deep learning sepsis prediction model on quality of care and survival

Sepsis remains a major cause of mortality and morbidity worldwide. Algorithms that assist with the early recognition of sepsis may improve outcomes, but relatively few studies have examined their impact on real-world patient outcomes. Our objective was to assess the impact of a deep-learning model (COMPOSER) for the early prediction of sepsis on patient outcomes. We completed a before-and-after quasi-experimental study at two distinct Emergency Departments (EDs) within the UC San Diego Health System. We included 6217 adult septic patients from 1/1/2021 through 4/30/2023. The exposure tested was a nurse-facing Best Practice Advisory (BPA) triggered by COMPOSER. In-hospital mortality, sepsis bundle compliance, 72-h change in sequential organ failure assessment (SOFA) score following sepsis onset, ICU-free days, and the number of ICU encounters were evaluated in the pre-intervention period (705 days) and the post-intervention period (145 days). The causal impact analysis was performed using a Bayesian structural time-series approach with confounder adjustments to assess the significance of the exposure at the 95% confidence level. The deployment of COMPOSER was significantly associated with a 1.9% absolute reduction (17% relative decrease) in in-hospital sepsis mortality (95% CI, 0.3%–3.5%), a 5.0% absolute increase (10% relative increase) in sepsis bundle compliance (95% CI, 2.4%–8.0%), and a 4% (95% CI, 1.1%–7.1%) reduction in 72-h SOFA change after sepsis onset in causal inference analysis. This study suggests that the deployment of COMPOSER for early prediction of sepsis was associated with a significant reduction in mortality and a significant increase in sepsis bundle compliance.


Reporting on race, ethnicity, or other socially relevant groupings
Table 1 reports the race composition of the study population as extracted from the EHR.They are included to show the population makeup during the pre-intervention and post-intervention periods.

Population characteristics
Table 1 shows baseline characteristics and summary statistics for the study cohort.From January 1st, 2021 through April 30th, 2023, 6,217 encounters to the ED met Sepsis-3 criteria.Most septic patients exhibited some level of chronic comorbidity (median Elixhauser of 5) and the median SOFA score at the time of sepsis was 2.

Recruitment
We included all adult patients (age ≥ 18 years old) who presented to the ED.We excluded patients who were transitioned to comfort measures prior to their time of sepsis and patients who developed sepsis after 12 hours of hospital admission.

Ethics oversight
University of California San Diego Institutional review board (IRB) approval was obtained with waiver of informed consent (#805726) and additional approval was obtained from the Aligning and Coordinating QUality Improvement, Research, and Evaluation (ACQUIRE) Committee (project #609).
Note that full information on the approval of the study protocol must also be provided in the manuscript.

Field-specific reporting
Please select the one below that is the best fit for your research.If you are not sure, read the appropriate sections before making your selection.

Life sciences
Behavioural & social sciences Ecological, evolutionary & environmental sciences For a reference copy of the document with all sections, see nature.com/documents/nr-reporting-summary-flat.pdf

Life sciences study design
All studies must disclose on these points even when the disclosure is negative.

Sample size
We included all adult patients (age ≥ 18 years old) who presented to the ED from January 1st, 2021 through April 30th, 2023 and met Sepsis-3 criteria.This resulted in a sample size of 6,217 encounters to the ED.

Data exclusions
We excluded patients who were transitioned to comfort measures prior to their time of sepsis and patients who developed sepsis after 12 hours of hospital admission.

Replication
Inferences from the Bayesian structural time-series models are the average of 1000 MCMC runs and independent of a particular choice in hyperparameters.
Randomization Patients were temporally segregated into the pre-intervention and post-intervention periods.Covariates were controlled via a Bayesian structural time-series approach.

Blinding
Blinding was not applicable for this pre-post study design.
Behavioural & social sciences study design All studies must disclose on these points even when the disclosure is negative.

Study description
Briefly describe the study type including whether data are quantitative, qualitative, or mixed-methods (e.g.qualitative cross-sectional, quantitative experimental, mixed-methods case study).

Research sample
State the research sample (e.g.Harvard university undergraduates, villagers in rural India) and provide relevant demographic information (e.g.age, sex) and indicate whether the sample is representative.Provide a rationale for the study sample chosen.For studies involving existing datasets, please describe the dataset and source.

Sampling strategy
Describe the sampling procedure (e.g. random, snowball, stratified, convenience).Describe the statistical methods that were used to predetermine sample size OR if no sample-size calculation was performed, describe how sample sizes were chosen and provide a rationale for why these sample sizes are sufficient.For qualitative data, please indicate whether data saturation was considered, and what criteria were used to decide that no further sampling was needed.

Data collection
Provide details about the data collection procedure, including the instruments or devices used to record the data (e.g.pen and paper,

nature portfolio | reporting summary
April 2023

Data collection
computer, eye tracker, video or audio equipment) whether anyone was present besides the participant(s) and the researcher, and whether the researcher was blind to experimental condition and/or the study hypothesis during data collection.

Timing
Indicate the start and stop dates of data collection.If there is a gap between collection periods, state the dates for each sample cohort.

Data exclusions
If no data were excluded from the analyses, state so OR if data were excluded, provide the exact number of exclusions and the rationale behind them, indicating whether exclusion criteria were pre-established.

Non-participation
State how many participants dropped out/declined participation and the reason(s) given OR provide response rate OR state that no participants dropped out/declined participation.

Randomization
If participants were not allocated into experimental groups, state so OR describe how participants were allocated to groups, and if allocation was not random, describe how covariates were controlled.

Ecological, evolutionary & environmental sciences study design
All studies must disclose on these points even when the disclosure is negative.

Study description
Briefly describe the study.For quantitative data include treatment factors and interactions, design structure (e.g.factorial, nested, hierarchical), nature and number of experimental units and replicates.

Sampling strategy
Note the sampling procedure.Describe the statistical methods that were used to predetermine sample size OR if no sample-size calculation was performed, describe how sample sizes were chosen and provide a rationale for why these sample sizes are sufficient.

Data collection
Describe the data collection procedure, including who recorded the data and how.
Timing and spatial scale Indicate the start and stop dates of data collection, noting the frequency and periodicity of sampling and providing a rationale for these choices.If there is a gap between collection periods, state the dates for each sample cohort.Specify the spatial scale from which the data are taken

Data exclusions
If no data were excluded from the analyses, state so OR if data were excluded, describe the exclusions and the rationale behind them, indicating whether exclusion criteria were pre-established.

Reproducibility
Describe the measures taken to verify the reproducibility of experimental findings.For each experiment, note whether any attempts to repeat the experiment failed OR state that all attempts to repeat the experiment were successful.

Randomization
Describe how samples/organisms/participants were allocated into groups.If allocation was not random, describe how covariates were controlled.If this is not relevant to your study, explain why.

Blinding
Describe the extent of blinding used during data acquisition and analysis.If blinding was not possible, describe why OR explain why blinding was not relevant to your study.

Did the study involve field work?
Yes No

Field conditions
Describe the study conditions for field work, providing relevant parameters (e.g.temperature, rainfall).

Location
State the location of the sampling or experiment, providing relevant parameters (e.g.latitude and longitude, elevation, water depth).
Access & import/export Describe the efforts you have made to access habitats and to collect and import/export your samples in a responsible manner and in compliance with local, national and international laws, noting any permits that were obtained (give the name of the issuing authority, the date of issue, and any identifying information).

Disturbance
Describe any disturbance caused by the study and how it was minimized.

Reporting for specific materials, systems and methods
We require information from authors about some types of materials, experimental systems and methods used in many studies.Here, indicate whether each material, system or method listed is relevant to your study.Field-collected samples For laboratory work with field-collected samples, describe all relevant parameters such as housing, maintenance, temperature, photoperiod and end-of-experiment protocol OR state that the study did not involve samples collected from the field.

nature portfolio | reporting summary
April 2023

Magnetic resonance imaging
Experimental design

Design type
Indicate task or resting state; event-related or block design.

Design specifications
Specify the number of blocks, trials or experimental units per session and/or subject, and specify the length of each trial or block (if trials are blocked) and interval between trials.
Behavioral performance measures State number and/or type of variables recorded (e.g.correct button press, response time) and what statistics were used to establish that the subjects were performing the task as expected (e.g.mean, range, and/or standard deviation across subjects).

Specify in Tesla
Sequence & imaging parameters Specify the pulse sequence type (gradient echo, spin echo, etc.), imaging type (EPI, spiral, etc.), field of view, matrix size, slice thickness, orientation and TE/TR/flip angle.

Area of acquisition
State whether a whole brain scan was used OR define the area of acquisition, describing how the region was determined.

Volume censoring
Define your software and/or method and criteria for volume censoring, and state the extent of such censoring.

Statistical modeling & inference
Model type and settings Specify type (mass univariate, multivariate, RSA, predictive, etc.) and describe essential details of the model at the first and second levels (e.g.fixed, random or mixed effects; drift or auto-correlation).

Effect(s) tested
Define precise effect in terms of the task or stimulus conditions instead of psychological concepts and indicate whether ANOVA or factorial designs were used.

Specify type of analysis: Whole brain ROI-based Both
Statistic type for inference (See Eklund et al. 2016) Specify voxel-wise or cluster-wise and report all relevant parameters for cluster-wise methods.

Correction
Describe the type of correction and how it is obtained for multiple comparisons (e.g.FWE, FDR, permutation or Monte Carlo).

Models & analysis
n/a Involved in the study

Graph analysis
Report the dependent variable and connectivity measure, specifying weighted graph or binarized graph,

Functional
modeling or predictive analysis Functional and/or effective connectivity Report the measures of dependence used and the model details (e.g.Pearson correlation, partial correlation, mutual information).
Describe the research sample (e.g. a group of tagged Passer domesticus, all Stenocereus thurberi within Organ Pipe Cactus National Monument), and provide a rationale for the sample choice.When relevant, describe the organism taxa, source, sex, age range and any manipulations.State what population the sample is meant to represent when applicable.For studies involving existing datasets, describe the data and its source.
Describe all antibodies used in the study; as applicable, provide supplier name, catalog number, clone name, and lot number.Please state if this information has not been collected.Report sex-based analyses where performed, justify reasons for lack of sex-based analysis.
If you are not sure if a list item applies to your research, read the appropriate section before selecting a response.