Introduction

Sepsis is a severe life-threatening organ dysfunction caused by a dysregulated host response to infection and is associated with a high risk of mortality1. Acute respiratory failure (ARF) is a common complication of sepsis and may lead to severe pulmonary organ damage, causing serious illness or even death2. Through a series of mechanisms, sepsis induces the expansion of a systemic inflammatory response that can cause lung damage and severe respiratory distress, leading to ARF3,4. When SA-ARF develops, the prognosis is bleak, resulting in critical death5,6. An increase in mortality is associated with a higher treatment expense. According to an epidemiological report from the United States, the cost of treating respiratory failure might reach $5 billion per year7. Early assessment and prediction of disease prognosis, as well as prompt management, are critical for reducing mortality and high health care expenditures in hospitalized patients8. However, studies on the prognostic prediction of patients with SA-ARF are rare9. An optimal scoring system is needed to create accurate prognostic predictions for patients.

Scoring systems are routinely used in the ICU to evaluate the prognosis of disease10. The SOFA score was created in 199611 and is included in the current definition of sepsis12. The SOFA score is valuable for predicting the prognosis of sepsis patients13,14,15. The APS III score is a component of the Acute Physiology and Chronic Health Evaluation II (APACHE II), which is simpler than APACHEII. It lacks an age score or a chronic health score, and is more accessible for clinical use than the APACHE II score16. APSIII can predict mortality in the ICU17. The SAPS II was created in 1993, it was used to predict the risk of death in ICU18. The SAPS II has been widely used to predict the prognosis in a variety of diseases, including acute coronary syndrome (ACS)19,20. The LODS is an objective tool proposed in 1996, which has been used to assess the severity of organ dysfunction in ICU14. The LODS can also predict prognosis in patients with sepsis or ARF21,22. In 2013, Johnson et al. reported OASIS, a predictive scoring system built with machine learning methods. The OASIS helps to predict mortality in patients admitted to the ICU23. It is also a good predictor of prognosis in sepsis and respiratory disease24, and it can be used to predict mortality in the ICU25. The purpose of this study is to compare which is among those severity scores the one that better predicts mortality in hospital with SA-ARF.

Methods

Data source

All data were obtained from a large-scale, publicly available database, the MIMIC-IV database, developed by the Massachusetts Institute of Technology. The MIMIC-IV database includes data from all patients treated in the ICU at Beth Israel Deaconess Medical Center between 2008 and 2019, including information on admission examination, laboratory tests, therapeutic measures, etc. Researchers were required to pass the relevant tests on the on the National Institute of Health website and sign a statement to ultimately qualify for access to the database26. The author, Fan Shiqin, finished the online training course for data research (record ID: 52287583) and signed the agreement. The relevant Institutional Review Board in America approved the project. In addition, the program is compliant with the Health Insurance Portability and Accountability Act (HIPPA). The identifying information about the patients was removed and we were not required to obtain informed consent or ethical review.

Study population

All adult patients diagnosed with SA-ARF (first admission only) were extracted from the MIMIC-IV database. Inclusion criteria: (1) meets the criteria for the diagnosis of sepsis 3.012, patients with probable infection and SOFA score ≥ 2; (2) ARF determined using the International Classification of Diseases (icd_codes): "51,881", "J9602", "51,851", "J9601", "J9600"; (3) re-admitted patients, with only the diagnosis from the first admission was retained; (4)) admitted to ICU and age ≥ 18 years; (5) admission time > 1 day; exclusion criteria: (1) SOFA < 2; (2) missing values from the baseline data were excluded, such as PaO2/FiO2, lactate, Hb, INR, WBC, BUN, BP, T, and RR were not recorded.

Data extraction and management

We used Navicat Premium software (version 15.0) to extract relevant data from the MIMIC-IV database. MIMIC-IV provided all of the needed data, which included APSIII, SAPSII, OASIS, LODS, and SOFA scores. We also included covariates that might influence the relationship between these five scoring systems and in-hospital mortality, extracting the following basic data: age, sex, race, and whether the patients were mechanically ventilated. The laboratory parameters included hematocrit (HCT), platelet (PLT), hemoglobin (HB), white blood cell (WBC), prothrombin time (PT), creatinine (Cr), blood urea nitrogen (BUN), the international normalized ratio (INR), lactate, the ratio of partial pressure of O2 in arterial blood to the fraction of inspired oxygen (PaO2/FiO2), vital signs, such as heart rate (HR), mean arterial pressure (MBP), the respiratory rate (RR) and temperature (T). The comorbidities included hypertension, liver disease, renal disease, chronic pulmonary disease (CPD), and diabetes. If a laboratory test or vital sign was measured more than once on the first day of hospitalization, the median value was obtained.

Statistical analysis

We used R (version 4.2.2), SPSS (version 27) and MedCalc (version 20.0.22) to analyze the data. Normally distributed data were expressed as mean ± standard deviation using the independent samples t-test; non-normally distributed data were expressed as median (interquartile spacing) [M (QL, QU)] using the Mann–Whitney test. Categorical variables were analyzed using t-test or chi-square test and expressed as numbers and percentages. Regression analyses were performed to identify independent risk factors for death in the hospital. Variables with P < 0.05 in the univariate analysis were incorporated into the multivariate analysis. The AUC of the ROC curves were compared using the method of Delong et al.27 to determine the predictive ability of each scoring system for hospital death. After the optimal cutoff value was obtained from the ROC curve, Kaplan–Meier curves were plotted to assess the impact of APSIII on the survival prognosis of patients in the high and low subgroups. The log rank test was used to evaluate the difference in survival between the high-stage and low-stage patients. Finally, DCA was applied to assess the net benefits of the five scoring systems for patients with SA-ARF28. The net benefit is a decision analysis metric that combines the assessment of harms and benefits. It is a clinical judgment of the relative value of the benefits and harms of a certain examination. A series of net benefits can be obtained by plotting via a DCA, which can be used to assess diagnosis and prognosis29. The larger the area under the DCA curve is, the greater the clinical benefit of a scoring system will be.

Results

Baseline characteristics

We extracted data on 7648 patients with SA-ARF from the MIMIC-IV database. After further screening and elimination, 3874 eligible patients were included. We divided them into 1260 non-survival and 2641 survival groups. The flow chart of the data extraction process is available in Fig. 1. The average age of the survival patients was 65 years, and the non-survival group was 67 years. The average age was lower in the survivor group than in the non-survival group (P < 0.05). The length of hospital stay (LOS Hos) and length of ICU stay (LOS ICU) were longer in the survival group than in the non-survival group (P < 0.001). Among the laboratory results on the day of admission, the WBC, Cr, BUN, INR, PT, lactate, HR, and RR were significantly higher in the non-survival group than the survival group (P < 0.05). A higher PLT levels, Hb, T, BP, and MAP were found in the survival group than in the non-survival group (P < 0.001). The non-survival group was more likely to have the following coexisting comorbidities than was the survivor group: renal disease (P < 0.05). All five scoring systems were high in the non-survival group compared to the survival group. Information related to sex, age, laboratory test results, and the scoring systems used for patients with SA-ARF is shown in Table 1.

Figure 1
figure 1

Flow chart of this study. PaO2/FiO2 the ratio of partial pressure of O2 in arterial blood to the fraction of inspired oxygen; Hb haemoglobin, INR the international normalized ratio, WBC white blood cell, BUN blood urea nitrogen, BP blood pressure, T temperature, RR the respiratory rate.

Full size image
Table 1 Demographic and clinical characteristics of patients (admission variable only).
Full size table

Regression analysis

Logistic regression analysis

Multivariate analysis showed that age (OR: 1.031, 95% CI 1.025–1.037, P < 0.001), RR (OR: 1.030, 95% CI 1.009–1.051, P = 0.004), lactate (OR: 1.385, 95% CI 1.300–1.476, P < 0.05), APSIII (OR: 1.021, 95% CI 1.015–1.026, P < 0.001), LODS (OR: 1.079, 95% CI 1.037–1.122, P < 0.001), and PaO2/FiO2 (OR: 1.001, 95% CI 1.000–1.002, P = 0.036) (Table 2) were independent risk factors for death in hospitals in patients with SA-ARF. In contrast, hypertension (OR: 0.752, 95% CI 0.627–0.901, P < 0.05) and renal disease (OR: 0.751, 95% CI 0.608–0.929, P = 0.008) were protective factors.

Table 2 Binomial logistic regression analysis of in-hospital mortality in patients with SA-ARF in ICU.
Full size table

Cox regression analysis

Independent risk factors for death were obtained using Cox regression analysis. Multivariate analysis revealed that APS III (HR: 1.005, 95% CI 1.002–1.009, P = 0.004), LODS (HR: 1.030, 95% CI 1.002–1.059, P = 0.037), SAPS II (HR: 1.007, 95% CI 1.002–1.013, P < 0.001), RR (HR: 1.034, 95% CI 1.020–1.049, P < 0.001) and lactate (HR: 1.166, 95% CI 1.139–1.193, P < 0.001) (Table 3) were found to be independent predictors of in-hospital death for patients with SA-ARF.

Table 3 Cox regression analysis of risk factors for in-hospital death in patients with SA-ARF in ICU.
Full size table

Comparison of Receiver operating characteristic curves for five scoring systems

The predictive value of the scoring system for in-hospital mortality was compared using AUC. The ROC curves are shown in Fig. 2, APSIII (AUC: 0.755, 95% Cl 0.714–0.768), LODS (AUC: 0.731, 95% Cl 0.717–0.7745), SAPSII (AUC: 0.727, 95% CI 0.713–0.741), OASIS (AUC: 0.706, 95% CI 0.691–0.720) and SOFA (AUC: 0.606, 95% CI 0.590–0.621) (Table 4). The AUCs of LODS, OASIS, and APS III were more than 0.7, which were significantly higher than the AUC value of SOFA. The AUC of APS III was greater than that of LODS. APSIII had the highest AUC and was more reliable in predicting death in the hospital. The threshold of the scoring system corresponding to Youden's index was chosen as the best threshold for predicting death in the hospital. APSIII had the highest Youden’s index (0.367) and sensitivity (73.53%), its specificity (63.2%) was within tolerable bounds, and OASIS had the highest specificity (67.06%).

Figure 2
figure 2

Receiver operating characteristic curves. (1) ROC curves of APSIII, SAPSII, LODS, OASIS and SOFA. (2) APSIII acute physiology score III, SAPSII Simplified acute physiology score, OASIS Oxford acute severity of illness score, LODS Logistic organ dysfunction system, SOFA Sequential organ failure assessment.

Full size image
Table 4 Comparisons of different predictive index.
Full size table

Kaplan‒Meier curves of the APSIII scoring system

The ideal cutoff value for the APSIII score was 69 according to the ROC curve and Youden's index calculations, splitting the patients in the APSIII subgroup into two subgroups: high- and low-scoring subgroups (Fig. 3). In the low subgroup (APSIII < 69), the median survival was 102.824 days (95% CI 101.030–104.619), while in the high subgroup, it was 64.278 days (95% CI 61.773–66.783). There was a statistically significant difference in survival time between the high and low subgroups (χ2 = 539.6405, P < 0.001). The hazard ratio (HR) for the low subgroup compared to the high subgroup was 0.263 (95% Cl 0.235–0.294), indicating that the risk of death in the low subgroup was 0.263 times greater than that in the high subgroup. Therefore, the prognosis of the low subgroup was superior to that of the high subgroup.

Figure 3
figure 3

Kaplan–Meier curves for patients with SA-ARF with APSIII ≥ 69 and APSIII < 69. APSIII Acute physiology score III.

Full size image

Comparison of decision curve analysis curves

As shown in Fig. 4. The results of the DCA curve showed that the red line representing APSIII always above the other lines (LODS, SAPSII, OASIS, SOFA in descending order). Therefore, APS III has the best net benefit and provides the best clinical benefit. We can utilize the APS III score for timely clinical interventions to achieve better clinical benefits.

Figure 4
figure 4

Comparison of decision curve analysis curves. The clinical benefit of the scoring system can be assessed by assessing the net benefit (y-axis) over a range of threshold probabilities (x-axis). The grey line represents the assumption that all patients with SA-ARF present with in-hospital death, and the black line indicates the assumption that no patients with SA-ARF present with in-hospital death. It can be concluded that the APSIII curve (red) shows the greatest benefit compared to the other curves. APSIII acute physiology score III, SAPSII Simplified acute physiology score, OASIS Oxford acute severity of illness score, LODS Logistic organ dysfunction system, SOFA Sequential organ failure assessment.

Full size image

Discussion

Early identification of patients with SA-ARF is important for prognosis. We can implement mechanical ventilation, infection control and various urgent interventional treatments as soon as possible, which will help reduce the risk of death. We aim to find a scoring system that will help us predict prognosis at an early clinical stage, so that physicians can intervene early in this group of high-risk patients to reduce in-hospital mortality. APSIII, SAPSII, LODS, OASIS, and SOFA are some typical scoring systems, and the prognostic value of these scoring systems in patients with SA-ARF is worth investigating.

We have come to the following conclusions: (1) both logistic regression and Cox regression results indicated that APSIII and LODS were independent risk factors for predicting in-hospital mortality in patients with SA-ARF. (2) ROC curves suggested that the AUCs of APSIII, SAPSII, LODS, and OASIS were above 0.7, which showed some clinical predictive value. APSIII had the best predictive value for assessing in-hospital mortality, while SOFA had the poorest performance. (3) In-hospital mortality was associated with LODS, OASIS, SOFA and APSIII scores, with higher scores representing more severe organ failure and worse prognosis. (4) Kaplan‒Meier survival curves showed that patients with APSIII values greater than 69 had lower median survival days and a higher in-hospital mortality rate. The log rank test of two survival curves were P(sig.) < 0.01, and the Breslow test were P(sig.) < 0.01, suggesting that there was a difference between the APSIII score and prognosis of patients. (5) DCA curves revealed that APS III had the greatest beneficial effect on patients within the maximum threshold, with timely clinical intervention may improve clinical benefit. The other four scoring systems, in descending order of clinical benefit, were LODS, SAPSII, OASIS, and SOFA.

In addition, several laboratory indicators, lactate levels, and the RR are also independent risk factors for in-hospital death in patients with SA-ARF. The release of metabolites and lactic acid increases, and causes respiratory acidosis30. Therefore, the lactate concentration can be used to evaluate the overall outcome of patients with sepsis13. When SA-ARF occurs, it may cause acidosis, which is negative for the patient's prognosis. Some indicators can be used to document the degree of acidosis, such as the RR and lactate level31. When sepsis develops to ARF, the body is deprived of oxygen due to impaired gas exchange, causing shortness of breath and a buildup of lactic acid. The accumulation of lactic acid is associated with mortality32. In addition, we discovered that age was an important risk factor associated with death in the hospital. This might be because the risk of death from sepsis increases with age33.

APSIII is a common scoring system in the ICU. APSIII is part of the acute physiological score of the APACHE II score, which contains 12 indicators, namely, T, MAP, HR, RR, PaO2, pulmonary artery oxygen differential (A-aDO2), hematocrit (HCT), serum potassium (K +), serum sodium (Na +), pH, Cr, and WBC count; it is simpler than the APACHE II score16. APSIII can be used in pediatric ICU, neurological ICU, and acute pancreatitis for determining prognosis and predict risk of death34,35,36. Fernández et al.37 found that the APSIII score was related to 90-day fatality in patients with sepsis-related acute kidney injury (Cox regression: hazard ratio (HR): 1.01, 95% confidence interval (Cl): 1.0–1.0, p < 0.048). We found that the APSIII had the highest sensitivity and Youden’s index, and it had the largest AUC of 0.755 (95% CI 0.741–0.768). The DCA curve also showed that the APSIII provide the best clinical benefit to patients. Kaplan‒Meier curves also revealed that higher APSIII scores were associated with poorer prognosis. The reason why APSIII has the best predictive value may be related to its indicators, pH and RR, which allow it to identify disturbances in acid‒base balance. As mentioned above, ARF can lead to respiratory acidosis through deepened and accelerated breathing and accumulate lactic acid in the body31. An imbalance in alkaline balance leads to disturbances in electrolyte metabolism, which can be reflected by indicators such as Na+ and K+ in APSIII. APSIII is also an indicator of lung ventilation. When ARF occurs, the prognosis is related to oxygen uptake in the lungs. When pulmonary ventilation is impaired, this can exacerbate the progression of the disease. This is an overlooked component of other scoring systems. There are no studies on the value of the APSIII scoring system for in-hospital mortality in patients with SA-ARF. We found that the APSIII score has the optimal predictive value.

Zhu et al.38 discovered in a study of scores in patients with sepsis that the AUC for SAPS II is (AUC: 0.754, 95% CI 0.743–0.765), for OASIS is (AUC: 0.753, 95% CI 0.742–0.764), and for LODS is (AUC: 0.822, 95% CI 95.0–743.0), which is comparable to our study that LODS (AUC: 0.731, 95% CI 0.717–0.745), OASIS (AUC: 0.706, 95% CI 0.691–0.720), and SAPSII (AUC: 0.727, 95% CI 0.713–0.741). The indicators of LODS involve several systems, including the neurological, circulatory, renal, respiratory, and hepatic systems39. In severe ARF, the PaO2/FiO2 ratio can be used to indicate pulmonary ventilation, but it may be biased when accompanied by increased PaCO2. Finally, the LODS score demonstrated good predictive value in this investigation. The regression analysis showed that it was an independent risk factor for death in the hospital, and the clinical benefit of the AUC and DCA curves was second only to that of the APS III score. SAPSII includes seventeen variables, including age, physiological variables, and chronic diseases, and it also contains indicators reflecting electrolyte disturbances such as Na+, k+, HCO3− and PaO2/FiO2. However, it is not as accurate as APSIII for predicting SA-ARF, and its predictive value lies at an average level. As previously described21, OASIS is a machine learning algorithm-based model that does not have as many for determining organ failure as does APSIII. None of these three scoring systems has significant advantages over the APSIII. Moreover, these methods do not perform as well as APSIII on the DCA curve and have poorer clinical benefits than APSIII.

Notably, the SOFA score performed the worst in terms of predictive value. The SOFA score is a common indicator used to assess organ failure in patients with sepsis40. Ferreira et al.41 found that when SOFA score were assessed in the first 96 h in the ICU, the mortality increased with increasing SOFA score. Zeng et al.42 discovered that the SOFA score was related to mortality from ARF in patients with lung cancer (HR: 1.142, 95% CI 1.012–1.288, p = 0.031). However, both logistic and Cox regression analyses suggested that the SOFA score is not an independent risk factor for predicting death in the hospital. ROC curves also showed that it had the lowest AUC (AUC: 0.606, 95% CI 0.590–0.621). The DCA curve showed that the score was at the bottom of the curve. The SOFA score has the lowest predictive value and clinical benefit. It may be that it has considerably fewer indicators than a valuable scoring system for appeals.

The strength of our study is that our study is retrospective and based on the large database-MIMIC-IV. We studied a large sample size. The results of our study are also convincing. The data extracted from the database provides credibility to this study. The data extracted from the database lend credibility to this study.

This study has several limitations. First, our study was only a single-center retrospective analysis of the MIMIC-IV database, and we did not use external data to validate the conclusions. Second, the database we used has limitations. We strictly followed the ICD diagnostic codes to identify sepsis and ARF, and we ultimately obtained data on the SA-ARF. The database did not contain a specific classification of the site of infection, nor was it detailed enough to categorize the type of ARF; therefore, we were unable to perform subgroup analyses (e.g., whether it originated from pneumonia, hospital acquired pneumonia (HAP), or acute respiratory distress syndrome (ARDS)) to draw more convincing conclusions. Third, we were unable to determine whether in-hospital deaths were caused by SA-ARF or by the patient's ultimate decision to abandon treatment. Therefore, additional large clinical cohort studies are needed to validate the accuracy of these findings.

Conclusion

APSIII and LODS are independent risk factors for mortality in patients who develop SA-ARF. ROC and DCA curves showed that APSIII had the best predictive value. APSIII is an excellent predictor of prognosis in patients with SA-ARF, and it can predict in-hospital mortality more accurately. Early assessment of a patient's risk of death allows physicians to make the best clinical decisions and helps to improve the prognosis of patients.