Introduction

More than 3 years since the first Coronavirus disease 2019 (COVID-19) appeared in Wuhan, China, and started a once-in-a-century pandemic1. While advancement in the treatment and prevention of severe COVID-19 disease has progressed rapidly, this respiratory viral disease remains an important source of worldwide morbidity and mortality2. A new wave of cases is still being reported due to viral mutation, partly due to antivirals and less efficacious vaccines, which selectively produce more resistant strains. Learning from the constraints caused by the COVID-19 waves, a clinical decision tool is essential for managing outbreaks of COVID-19 cases to help in triaging patients and thus preventing scarcity of hospital beds and medical resources3.

Numerous clinical decision tools have been created and published for the purpose mentioned above, such as the COVID-GRAM and the 4C Mortality score4,5. To the best of our knowledge, the clinical decision tools available today utilize clinical parameters and laboratory data only. It is understandable because these tools must be simple and practical and have adequate accuracy to be used clinically. Understandably, there are no clinical decision tools that incorporate radiographic AI parameter of COVID-19 patients.

The role of artificial intelligence (AI) in the medical field has expanded rapidly. Mainly, this role is limited to the purpose of screening and diagnosis. For example, in the field of pulmonology, AI-aided radiographic interpretation of chest X-ray (CXR) images proved to be sensitive and accurate for pulmonary tuberculosis screening6. The role of AI in COVID-19 diagnosis has also been reported. In one study, CAD4COVID-Xray (an AI software), through the color heatmap method, had a superior COVID-19 pneumonia diagnosis compared to six radiologists7,8. In contrast, the evidence on incorporating AI-aided radiographic interpretation of CXR for predicting clinical outcomes is scarce.

This study aimed to investigate the relationship between AI-aided radiographic parameter, clinical and laboratory data, and clinical outcomes in hospitalized COVID-19 patients with confirmed RT-PCR results. Additionally, we aimed to develop and validate a clinical risk tool known as Integrated Inpatient Mortality Prediction Score for COVID-19 (IMPACT) by integrating these data.

Methods

Study design

This was a retrospective cohort study using a secondary data from medical records and Picture Archiving and Communication System (PACS) chest radiography repositories. This study was conducted at three academic hospital, i.e., Airlangga University Hospital, located in Surabaya, East Java Province, Sardjito General Hospital, located in Jogjakarta, Special Region of Jogjakarta Province, and in dr. Cipto Mangunkusumo General Hospital, located in Centre Jakarta, Special Capital Region of Jakarta. The ethics committee of the respective hospitals approved the study. The requirement for written informed consent was exempted due to the utilization of anonymized historical data. (University of Gadjah Mada, University of Airlangga, and University of Indonesia IRB). All methods were performed in accordance with the relevant guidelines and regulations and adhered to Declaration of Helsinki.

Data obtained from the first two hospitals was used for the derivation of the novel scoring system. Data obtained from the latter hospital was used for the validation of the novel scoring system.

This study enrolled a cohort of adult patients (≥ 18 years old) hospitalized with COVID-19 cases between April 2020 and April 2022 for the derivation cohort. The validation cohort, however, included only hospitalized COVID-19 cases from April 2020 to April 2021. The reasons for this approach are twofold. Firstly, we utilized data from a separate study to constitute the validation cohort. Secondly, due to the expiration of our software permit, we were unable to access the CXR software required for additional analysis. As a result, we relied solely on the available data for analysis.

The decision to utilize chest X-ray rather than more advanced imaging modalities is driven by two primary considerations. First and foremost is Indonesia's classification as a low to middle-income country (LMIC). The accessibility of chest CT-scans is constrained, predominantly concentrated in major cities, reflecting the uneven distribution of healthcare infrastructure. Furthermore, adhering to WHO guidelines, chest radiography provides higher specificity and can be performed using portable equipment at the point of care. This not only tackles the issue of restricted access but also mitigates the risk of cross-infection linked to patient transport9,10.

COVID-19 cases were verified using either positive high-throughput sequencing or real-time reverse-transcription polymerase-chain-reaction (RT-PCR) tests conducted on nasal and pharyngeal swab samples.

This study excluded cases with substandard chest radiography qualities, large lung cavities on CXR, concurrent mediastinal or lung mass, and an interval between RT-PCR and CXR acquisition of more than 7 days.

Included in the data collection including the patients’ demographic data, vaccination data, comorbidities, clinical and laboratory data in the emergency department, treatment, and discharge outcome. COVID-19 disease severity was determined on hospital admission and was stratified according to the local Indonesian Guideline, which adopted the WHO COVID-19 disease severity stratification11.

A group of internal medicine physicians carefully examined, summarized, and verified the data. Two clinicians independently reviewed each record.

Potential predictive variables

We included the following variables: gender, vaccination status, smoking history, and comorbidities (autoimmunity, obesity, arterial hypertension, diabetes mellitus, asthma, coronary heart disease, cerebrovascular disease, chronic obstructive pulmonary disease, pulmonary tuberculosis, chronic kidney disease, chronic liver disease, brain disease, immunodeficiency disease, and cancer), as well as symptoms (fever, cough, sore throat, rhinorrhea, anosmia, myalgia, headache, malaise, anorexia, diarrhea, nausea, vomiting, abdominal pain, dyspnoea, chest pain, loss of consciousness), CXR data (CXR projection, pneumonia on CXR, CXR AI probability score, CXR AI affected lung area [ALA] score), sepsis, septic shock, ARDS, co-infection, clinical data (systolic blood pressure, diastolic blood pressure, pulse rate, temperature, respiratory rate, peripheral oxygen saturation, and symptom onset), and laboratory data (haemoglobin, potassium, sodium, white blood cells, lymphocytes, neutrophil levels, thrombocyte count, neutrophil-to-lymphocyte ratio, and urea levels).

Outcomes

The primary outcome of this study was in-hospital mortality. The secondary outcome of this study was disease progression defined as at least one degree increment of disease severity (e.g., disease progression from moderate to severe disease).

AI system for chest X-ray interpretation

CAD4COVID-Xray software (Thirona, Nijmegen, Netherlands; https://covid.cad4tb.care/accounts/login/?next=/; based on CAD4TB ver. 6) was used for AI chest X-ray interpretation. The principal objective of the CAD4COVID-Xray software is to facilitate the triaging process in environments with limited resources and in areas with a high prevalence of COVID-19. This product holds a CE certification and employs the identical technical core utilized by CAD4TB, another CE-certified product registered by the FDA in Ghana. Consequently, CAD4COVID-Xray is developed to the same high-quality standard as CAD4TB, a standard substantiated by validation through over 40 academic publications. CAD4TB has been successfully deployed in 35 countries, playing a pivotal role in screening six million people globally12. Hence, its reliability, pertinence, and applicability have undergone rigorous validation, affirming that CAD4COVID-Xray is not only a dependable solution but also possesses relevance and generalizability across diverse healthcare scenarios.

The AI software relies on color heat-map method to detect parenchymal abnormality on chest X-rays (Fig. 1).

Figure 1
figure 1

Detection of lung parenchymal abnormality in the chest X-ray with the heatmap method.

Full size image

The software executed a series of steps outlined below:

  1. 1.

    Pseudoanonymization of COVID-19 Patients’ Digital Chest X-ray

    Confirmed COVID-19 patients' digital chest X-rays (in DICOM format) underwent pseudoanonymization before being uploaded using the PACS INFINITT software from INFINITT Healthcare in Seoul, South Korea.

  2. 2.

    Chest X-ray scoring process

    • The chest X-ray scoring involved four consecutive steps:

    • Normalization: Standardizing the CXR scale for consistent handling by the AI and generalization from CXRs of different sizes.

    • Segmentation of Lung Fields: Automatic separation of lungs from the surrounding areas of the image by the AI.

    • Texture Evaluation: Identification of relevant anomalies in the lung segments.

    • Area Analysis: Determination of the proportion of lung parenchyma affected.

  3. 3.

    Filter Weight Determination

    • The weights of each filter were determined, and an average filter weight was applied as a mask on the CXR picture. This created a color heat map visible only on the lung area previously segregated by the trained model.

  4. 4.

    Color Heat Map Representation:

    • The color heat map exhibited various colors based on data weight:

    • High, medium, low, and extremely low probabilities of abnormality were represented by the colors red, yellow, green, and blue, respectively.

  5. 5.

    CAD4COVID Software Scoring:

    • The CAD4COVID software received the digital CXR file and generated two AI scorings:

    • Affected Lung Area (ALA) Score: Calculated from the total lung volume with abnormalities found on the CXR, ranging from 0 to 100. A higher score indicates a greater impact on lung tissue.

    • COVID Probability Score: Determined by the average final weight of all layers, ranging from 0 to 100. A higher score suggests a higher likelihood of COVID-19 occurrence.

Variable selection and establishing a scoring system

For the variable selection and scoring system derivation, we included all 512 hospitalized COVID-19 patients in the derivation cohort. In the selection process, we entered 66 variables. We applied the Least Absolute Shrinkage and Selection Operator (LASSO) regression with the purpose to minimize the potential collinearity of measured variables from the same patient and to prevent variables over-fitting. To deal with missing values, imputation was considered if the missing values were less than 25%. We used Multiple Imputation by Chained Equations (MICE) to impute numeric, binary, and factor variables13. In our multivariable analyses, we used least absolute shrinkage and selection regression with L1 penalization and tenfold cross-validation for internal validation4. Based on its value, this logistic regression model imposes penalties on the absolute magnitudes of the regression coefficients. The estimates of weaker components are minimized towards zero by using greater penalties, leaving just the most significant predictors in the model. The factors that were shown to be the most predictive were those with the lowest value (min). The LASSO regression was carried out using the statistical program “glmnet” from the R Foundation. The risk score was then created using the consistently statistically significant factors that were included in logistic regression models after the variables found by LASSO regression analysis were included. (supplementary file).

Accuracy assessment

The IMPACT’s score accuracy was assessed using the area under the receiver-operator characteristic curve (AUROC). Statistical analysis was performed with the IBM Statistical Program for Social Science (SPSS) for Macintosh, 27.0 (IBM Corp., Armonk, NY, USA) with statistical significant set at P < 0.05.

IMPACT score validation

The IMPACT score was validated using data from Dr. Cipto Mangunkusumo General Hospital (RSCM), which included a cohort of 137 patients. RSCM is a national referral hospital situated in Jakarta, the capital city of Indonesia. As a result, the baseline characteristics of hospitalized COVID-19 patients in this study were highly diverse, showcasing the ethnic and racial heterogeneity of Indonesia. The data collected from RSCM underwent meticulous scrutiny and verification by two physicians (AS and JH). This dataset was utilized to compute the IMPACT COVID-19 mortality risk score, as mentioned earlier, for the derivation cohort.

Characteristics of derivation cohort

In the derivation cohort, we included a total of 512 patients from two academic medical centres located in Surabaya, East Java Province, and the Special Region of Jogjakarta, Jogjakarta Province, spanning from April 2020 to April 2022. Upon hospital admission, the proportion of COVID-19 disease severity, as per the Indonesian COVID-19 guideline, was as follows: mild cases accounted for 5.9%, moderate cases for 41.2%, severe cases for 30.9%, and critical cases for 22.1%. Out of 512 patients, 106 patients (20.7%) experienced clinical worsening during their hospitalization, resulting in an in-hospital mortality rate of 28.9%. The top five pre-existing conditions observed in these patients were hypertension, diabetes, obesity, coronary heart disease, and chronic kidney disease. The predominant symptoms reported were cough, fever, dyspnoea, nausea, and malaise (refer to Table 2). Abnormal chest X-rays were identified in 412 (80.47%) patients, and further details regarding laboratory findings can be found in Table 2.

Predictor selection

For the LASSO regression, we entered 66 variables measured at hospital admission (Tables 1 and 2). Through LASSO regression with, we identified 18 variables that were significant predictors for mortality (Fig. 2). These variables were gender, vaccination status, positive cancer status, sore throat, anosmia, dyspnoea, lost of consciousness, effusion on CXR, septic shock, CXR ALA score, mean arterial pressure, respiratory rate, peripheral oxygen saturation, white blood cells count, neutrophil count, neutrophil–lymphocyte-ratio, serum urea levels, and potassium levels. Subsequently, these variables were entered in logistic regression. The final variables included in final model were anosmia (OR: 0.280; 95%CI 0.095–0.826; P = 0.021), dyspnoea (OR: 1.684; 95%CI 1.049–2.705; P < 0.031), loss of consciousness (OR: 4.593; 95%CI 1.702–12.396; P = 0.003), mean arterial pressure (OR: 0.928; 95%CI 0.900–0.957; P = 0.007), peripheral oxygen saturation (OR: 0.981; 95%CI 0.967–0.996; P = 0.012), neutrophil % (OR: 1.034; 95%CI 1.013–1.055; P = 0.001), serum urea (OR: 1.018; 95%CI 1.010–1.026; P < 0.001), affected lung area score (OR: 1.026; 95%CI 1.014–1.038; P < 0.001). (Table 3).

Table 1 Demographic and clinical characteristics of the derivation cohort.
Full size table
Table 2 Clinical, laboratory and radiological findings of the derivation cohort (n[%]; mean ± SD; median [Q1–Q3]).
Full size table
Figure 2
figure 2

Variable selection for model construction using the least absolute shrinkage and selection operator (LASSO) binary logistic regression model. (a) LASSO coefficient profiles of the 66 baseline variables. (b) Tuning parameter selection for the LASSO model using tenfold cross-validation and minimum criteria.

Full size image
Table 3 Multivariate logistic regression model for predicting in-hospital mortality 512 COVID-19 inpatients.
Full size table

IMPACT mortality score construction

The IMPACT score was constructed based on the rounding of the beta coefficients acquired from the logistic model. The mortality risk score was developed by utilizing coefficients from the logistic model. We employed the following formulas in the logistic model to compute the probability and 95% confidence intervals14: The probability formula is given by exp(β × X)/[1 + exp(β × X)]. The formula for calculating the lower limit of the 95% confidence interval is exp[Xn × βn—z × SE(β)]/{1 + exp[Xn × βn—z × SE(β)]}, and the formula for calculating the upper limit of the 95% confidence interval is exp[Xn × βn + z × SE(β)]/{1 + exp[Xn × βn + z × SE(β)]}.

Therefore, the formula for calculating the mortality risk is as follows: mortality risk = 100 * (1/(1 + exp(−x))), where (x) is derived from the equation: (Anosmia*−1.273) + (Dyspnoea*0.521) + (Loss of Consciousness*1.525) + (SpO2*-0.074) + (MAP*−0.019) + (Neutrophil Percentage*0.033) + (Serum Urea*0.018) + (ALA Score*0.026) + 3.749.

IMPACT mortality score performance

The predictive value for IMPACT score from the derivation cohort was 0.815 (95% CI 0.774–0.856), which was categorized as a model with a good predictive ability (Fig. 3 panel a). The AUROCs of IMPACT sore for Airlangga University Hospital and dr. Sardjito General Hospital were 0.839 (95%CI 0.768–0.910) and 0.793 (95%CI 0.741–0.845), respectively.

Figure 3
figure 3

IMPACT in-hospital mortality scoring system area under the receiver—operating characteristic (AUROC) curve: (a) Derivation cohort, (b) validation cohort.

Full size image

IMPACT mortality score validation

The internal validation of IMPACT score in the RSCM hospital yielded AUROC score of 0.770 (95%CI 0.661–0.879) with a fair predictive ability (Fig. 3 panel b). The baseline and clinical characteristics of the validation cohort are presented in Table 4.

Table 4 Baseline characteristics of the validation cohort (n[%]; mean ± SD; median [Q1–Q3]).
Full size table

As previously stated in the Methods section, the disparity in the hospitalization periods between the validation cohort and derivation cohort for COVID-19 patients could raise concerns about the validity of our findings. To address this issue, we conducted a stratified analysis of the AUROC of the IMPACT score based on the year of admission in the derivation cohort. Our analysis revealed no significant differences in the AUROC among the admission years 2020, 2021, and 2022 (Fig. 4 and supplementary material Table 1).

Figure 4
figure 4

Area under the receiver—operating characteristic of impact score stratified according to the Admission Year.

Full size image

Ethics approval and consent to participate

Ethical approval for this study was obtained from the University of Gadjah Mada (IRB Approval Number: KE/FK/0900/EC/2022), the University of Airlangga (IRB Approval Number: 073/KEP/2022), and the University of Indonesia’s Faculty of Medicine Internal Review Board (IRB Approval Number: KET-1334/UN2.F1/ETIK/PPM.00.02/2022). The internal review board waived the consent to participate due to the retrospective nature of this study.

Discussion

In this study, we developed and internally validated the IMPACT Mortality score. The AUROC scores obtained from the derivation and validation cohorts were 0.815 and 0.770, respectively. To the best of our knowledge, this is the first scoring system that integrates clinical and laboratory data with AI-related parameters.

The developed model offers two advantages. Firstly, the scoring system incorporates basic laboratory indices that are accessible even in primary care clinics and small hospitals. This sets it apart from other scoring systems like COVID-Gram, which utilize numerous laboratory parameters that are often unavailable in small/rural hospitals, especially in developing countries. Secondly, the model includes an AI-based parameter called the ALA score, derived from analysed chest X-rays (CXRs). The ALA score is generated by the CAD4COVID software, providing rapid results after uploading the image to the cloud-based system.

While CT-scan is the preferred imaging modality for aiding COVID-19 diagnosis, it may not be accessible in many developing countries. On the other hand, CXR is readily available even in small hospitals. Additionally, logistical challenges such as patient transportation and the need for resuscitation in severe cases make CT-scan impractical15.

In the final model, several variables were identified as protective against in-hospital mortality due to COVID-19. These variables include mean arterial pressure (MAP), peripheral oxygen saturation (SpO2), and anosmia. Higher MAP and SpO2 as protective variables may indicate a milder course of COVID-1916.

Anosmia is considered a protective factor against in-hospital mortality of COVID-19 disease. A retrospective study involving 576 patients found that those with anosmia had higher levels of lymphocytes, haemoglobin, and GFR, and lower levels of D-dimer and CRP, indicating a milder immune and inflammatory response to SARS-CoV-2 infection17. Hendawy et al. also showed that anosmia is associated with mild chest infection18.

Conversely, our model identified several risk factors for mortality, including dyspnoea, loss of consciousness (LOC), higher neutrophil count, and higher serum urea levels. These factors are well-known indicators of poor prognosis, even in community-acquired pneumonia19,20. Significantly, elevated neutrophil levels contribute to COVID-19-associated coagulopathy by activating neutrophil extracellular traps (NETs)21. NETosis, the process where neutrophils release DNA structures to trap and kill pathogens, plays a crucial role in the immune response against infections, including viral infections. Excessive NET release leads to inflammation, tissue damage, and contributes to the cytokine storm observed in severe cases.

Notably, our study emphasizes the importance of quantifying parenchymal abnormalities with the help of an AI-based parameter for the purpose of determining mortality risk.

Many studies have evaluated the diagnostic ability of artificial intelligence against human readers. For example, Murphy et al. found that CAD4COVID X-ray which was trained on 24,678 radiographs, including 1540 radiographs for validation, had a comparable performance against six radiologists8.

Another study by Kapoor et al. showed that AI utilization improved triaging system of COVID-19 cases in the emergency department when faced with patients presenting with Flu-like symptoms. Combined with the high resolution CT AI analysis, CXR AI analysis had 97.9% sensitivity and 99% specificity22.

In another study, AI parameters derived from CT-scan namely CT—severity score (CT-SS) and affected lung area (%AA)23. These parameters associated with poor outcomes such as length of stay, risk of ICU admission, ICU LOS, and risk of mechanical ventilation. The CT-SS had a good predictive ability for ICU admission in COVID-19 patients (AUROC = 0.84; 95%CI 0.79–0.90).

In contrast, the evidence supporting the use of artificial intelligence for disease prognosis is limited. To date, only one published study has specifically investigated this area. In their study, Shamout et al. developed a prognostic model using an artificial intelligence system24. The model demonstrated the ability to predict deterioration within 96 h, achieving an AUROC of 0.786 (95% CI 0.745–0.830). To construct the model, the AI system utilized a dataset comprising clinical variables extracted from electronic health records and CXR images. Notably, the model effectively estimated the temporal risk evolution and considered relevant clinical endpoints, such as ICU admission, intubation, and in-hospital mortality.

Comparatively, our model and the model by Shamout et al. differ in terms of the clinical variables incorporated. Their model encompassed a larger set of clinical variables, potentially presenting challenges for replication in developing countries. In contrast, our model included a greater number of clinical variables known to be significant risk factors for COVID-19 outcomes. Consequently, our model may offer more advantages when employed in resource-limited areas.

While the WHO has lifted the pandemic status for COVID-19, this study aims to emphasize the importance of not forgetting the lessons learned from the past. Specifically, the dire need for utilizing artificial intelligence to enhance triaging capabilities, particularly during the peak of the pandemic when resources and manpower were scarce, should be highlighted.

Limitations

This study has several limitations. Firstly, the number of subjects included in both the derivation and validation cohorts is relatively small. Secondly, the study utilized retrospective data, highlighting the need for prospective validation. Thidrly, we did not utilize control group. Finally, we did not analyse the impact of COVID-19 variants on the discriminative ability of the IMPACT scoring system.

Conclusion

Our multicentre study introduces the Integrated Inpatient Mortality Prediction Score for COVID-19 (IMPACT), a clinical risk tool that integrates clinical, laboratory, and CXR data. This study provides valuable evidence on the real-world application of AI in clinical settings, demonstrating its potential for enhancing decision-making and improving patient care. The derivation and validation of IMPACT highlight the transformative role of AI in healthcare, enabling more personalized and effective treatment strategies for COVID-19 patients. Our research aims to bridge the gap between AI advancements and practical use, facilitating wider adoption of AI-based tools and revolutionizing disease prognostication in healthcare. However, it is imperative to conduct prospective validation of our findings, preferably utilizing a control group and extending the application to broader populations.