Limited applicability of a COVID-19 specific mortality prediction rule to the intensive care setting

arising from Yan et al. Nature Machine Intelligence https://doi.org/10.1038/s42256-020-0180-7 (2020)

We read the Article by Yan et al.¹ with great interest. The COVID-19 pandemic has established itself as a major burden on healthcare services worldwide. Scores or algorithms to optimize the use of healthcare resources are of paramount importance. Against this background, Yan et al. gathered samples from a cohort of 485 infected patients in the region of Wuhan, China with a high mortality rate of almost 40% and proposed a simple and operable decision rule based on lactic dehydrogenase (LDH), lymphocytes and high-sensitivity C-reactive protein (hs-CRP) to predict the occurrence of death in the following 10 days.

Since March 2020, France has also been confronted with the COVID-19 pandemic. The decision rule of Yan et al. could be used in our patients, but external repeatability would first be required. To validate the generalizability of the rule, we used data from Outcomerea, a French multicentre cohort of intensive care units (ICUs) involved in the management of patients critically ill with COVID-19. Methods for data collection and the quality of the database have been described in detail elsewhere². Since the beginning of the COVID outbreak in France, a range of specific clinical and biological data for patients with COVID have also been recorded prospectively into this database.

We included 178 patients aged over 18 years who were admitted to the ICU from 1 March 2020 to 1 June 2020 with laboratory-confirmed COVID-19. Patients without a measurement of LDH, hs-CRP or lymphocytes during the first three days after ICU admission were excluded. The main characteristics of our cohort are reported in Table 1. Among the 178 patients, fever was the most common initial symptom (80.8%), followed by dyspnoea (74.2%), cough (63%) and fatigue (43.2%). The median time from symptoms onset to ICU admission was 10 days (range 7–12 days) and the median duration between hospital and ICU admission was 2 days (range 1–3 days). They had a median age of 61 years (range 52–69 years), a median Charlson comorbidity index of 1 (range 0–3) and a median sepsis-related organ failure assessment score (SOFA) score of 5 (range 4–8). The median LDH, hs-CRP levels and percentage of lymphocytes were 453 Ui l⁻¹ (range 352–603 Ui l⁻¹), 166 mg l⁻¹ (range 92.4–223 mg l⁻¹) and 9.6% (range 6.2–15%), respectively. The median ICU length of stay was 11 days (range 6–19 days). At days 14 and 28, the mortality rates were 18% and 34.2%. The results presented in Tables 2 and 3 show that the precision and accuracy of the decision rule were extremely low for the prediction of death. The least bad results were obtained at day 28, with a precision of 37% (positive predictive value) and an accuracy of 43%, but a recall of 93% (negative predictive value). This decision rule lacked specificity in our preselected cohort of critically ill patients, which could compromise its routine use.

Table 1 Characteristics of the 178 patients of the Outcomerea database

Table 2 Confusion matrix for the French Outcomerea dataset

Table 3 Performance of the decision rule of Yan et al. on the French Outcomerea dataset

These results could be explained by the real specificity of our cohort. Indeed, only ~5% of patients with COVID-19 are admitted to ICU for acute hypoxemic respiratory failure (AHRF)³. Consequently, our ICU population did not include (1) the vast majority of pauci-symptomatic patients with very low LDH and hs-CRP serum levels and high lymphocyte counts (these patients have good outcomes) and (2) some of the most severely ill patients with high hs-CRP and LDH serum levels and low lymphocyte counts, who are not admitted to ICU because of therapeutic limitation (these patients have the worst outcomes). Thus, it is not surprising that the predictive rule of Yan et al. was not accurate in our cohort. However, their proposed biomarkers might be interesting for predicting ICU admission and also death for patients admitted to ICU, but with other thresholds. As a result, we believe that different rules should be adapted to different stages of the illness. For example, a decision tree could be rebuilt in the ICU to predict the occurrence of death. Furthermore, death might not be the most appropriate outcome—worsening of the disease could be better. Another decision rule could be built for patients admitted to the emergency room to predict worsening, that is, the occurrence of severe or critical types of COVID (COS-COVID)⁴. Finally, as already mentioned by Yan et al., we agree that, for the development of more rigorous prediction models, collaboration and sharing of well-documented individual data for COVID-19 are needed. The predictors already identified, such as LDH, hs-CRP and lymphocyte counts, should be considered as candidate predictors for new models⁵.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.