Introduction

The main issue with the COVID-19 pandemic—or any pandemic for that matter—is that healthcare systems are not designed for abrupt increase in the influx of severe cases. When this happens the hospitals fill up quickly without being able to provide the necessary care for every individual. Due to the high pressure on healthcare workers semi-autonomous systems are always welcome in the assistance of decision making. The Covid CXR Hackathon—Artificial Intelligence for COVID-19 prognosis https://ai4covid-hackathon.ing.unimore.it/ challenge was launched and run for almost an entire month with this in mind. The goal of the competition was two-fold: whether it is possible to predict COVID-19 severity (mild or severe) by CXR (chest X-ray) images produced at admission with the accompanying clinical data from blood samples and other pre-screeing factors effectively; and whether such a high performance model could be made explainable. The challenge aimed at balanced accuracy, however we evaluated our models against other metrics as well. We explored several ideas and systematically built our models to include clinical data and extra features obtained by following the insights of other COVID-19 lung studies. Compared to previous similar works where the goal was to correctly classify COVID-19 cases against different types of pneumonia1, our task was considering whether it would be possible to predict the severity of COVID-19 based on initial CXR screening alongside the clinical metadata. Obviously this is also affected by many other factors, so we didn’t expect overly confident predictions as it was previously showed by others, what is approximately achievable on such data2,3,4.

Studies have explored the use of radiography for detecting and assessing the severity of COVID-19. Imaging modalities, such as chest X-ray (CXR) and chest computed tomography (CT), have been compared for their effectiveness in diagnosing and assessing COVID-195,6,7. CXR was found to have lower sensitivity but has advantages over CT, such as lower cost, less need for patient transport, and lower radiation dosage. One study8 used synthetic X-ray images to validate their model, while two reviews9,10 analyzed CXR’s relationship with COVID-19 pneumonia.

In our work we have relied heavily on the Brixia-scoring system2. The Brixia semi-quantitative scoring system was first introduced at the beginning of the pandemic in order to aid radiologists with visual assessment and numerical scaling of the presence and severity of lung involvement amid the wide range of possible manifestation of pulmonary diseases. The prognostic value of the Brixia-score is further reinforced by other works7,11 showing that both the Brixia-score registered at admission and the highest Brixia-score registered during the hospitalization of a patient correlates with patient outcome.

Studies suggest8,12 using artificial data for COVID-19 imaging datasets, which are small compared to current deep learning datasets. However, this was not feasible in our case due to the presence of metadata and lack of a combined model to generate both COVID-19 CXRs and clinical features. In this regard, our task is also related to multi-modal learning with (CXR, free text diagnosis) data pairs, but we aim to learn from different modalities, not correspondence between diagnosis and CXR representations. Some work has proposed patch-based evaluation13 of CXR images, similar to the Brixia-scoring system. Severity prediction on CXR images without metadata has been done for COVID-19 and pneumonia14,15, but the severity metric was determined by radiologists, not disease outcome. There is also work discussing the differentiation of COVID-CXRs from regular CXRs16 and concluding for the need of more data. Studies have compared and evaluated popular CNNs17,18 for COVID-19 CXRs through accuracy metrics and input manipulation, some incorporating multiple datasets4,19 and highlighting common pitfalls20. Pre-training CNNs with novel datasets21 would have been also possible, however, we haven’t explored this, but used ImageNet pre-trained weight at initialization.

Data

The data was provided by the challenge organizers and is available upon-request https://ai4covid-hackathon.ing.unimore.it/data (archived version: https://web.archive.org/web/20220924165541/https://ai4covid-hackathon.it/data), it includes a curated, raw dataset gathered from six Italian hospitals during the first SARS-CoV-2 outbreak in Italy (March–June 2020). All patients were confirmed to have contracted the disease and later on classified with a prognosis of severe or mild outcomes, this is considered as the ground truth (GT) in our study. A severe case meant that the patient needed mechanical ventilation or died, all other cases were considered mild. In addition to the imagery data, clinical metadata were provided for all the patients. Since not all the clinical features were available for each patient, the clinical data required a lot of data imputation before it could be used for modeling. Most of the data had been available before3 but the challenge organizers expanded it with several new data points. We have examined 1103 patients in this study, which included an additional $$\sim$$ 33% of data compared to the previous study3. The held-out test set was an additional 486 samples that we didn’t use for training but pre-processed the same way as the training data.

Pre-processing

The images were converted from different types of DICOM images ranging from 12-bit to 16-bit in precision and also digitized in various ways. The resulting images which built up the training dataset varied significantly in terms of quality: some images were rotated randomly by 90$$^{\circ }$$, many were inverted and some contained a fully blacked-out or a gray margin on some edges of the scan. In order to deal with this, we selected the top two corners and the middle section of the scan in order to automatically detect whether the scan is inverted or not based on the mean brightness ratios between these regions; the inverted images have been reverted. We also manually annotated all images for inversion and concluded that our method was around 99% accurate. The left out images after the inversion method were corrected manually both in the training and the test sets. After finishing the inversion we followed with the pre-processing steps2 by applying clipping between the 2nd and 98th percentile of pixel values, quantizing the images to 8-bit, using contrast limited adaptive histogram equalization (CLAHE) and median filtering to reduce noise. Finally we scaled the images to 512 x 512 pixels in order to be appropriate for the BSNet2 network that we used to align, segment and score the images with the Brixia-scoring system.

Noise reduction and data imputation

Having the pre-processed images, next we applied the BSNet2 in order to acquire their Brixia-scores, as well as aligning them and segmenting the lung area. The alignment played a crucial role in this project since the original images differed significantly, not only from hospital to hospital, but also within the same hospital, as a result, this process ensured that the images looked more uniform than they originally did (see Fig. 1). The segmentation on the aligned images was made more reliable by feeding in also the horizontally flipped counterpart of each image, then applying horizontal flipping again (flipping it back) on the output segmentation map and finally, averaging these two predicted outcomes. This way we averaged out the predictions, achieving better (lower variance) segmentation maps (not binary) for each aligned image and boosted the accuracies on hard-to-predict areas. We applied several pre-processing steps (whitening, one-hot encoding for hospitals, etc.) for the clinical metadata too and used different imputation methods to handle the missing values. These attempts contained imputation based on age groups, hospitals, population means (modes for categorical values) and random sampling the feature distributions of the present values. For further information regarding the feature types, counts and names please see Tables S1 and S2. The extended clinical dataset with the pre-processed images was used for the training both for the performance and for the explainable models. We also included the absolute value of the 2D Fourier-transforms of the aligned images in order to capture texture sensitive features with our image-only and performance models. All models were trained with five-fold cross-validation and hyperparameter optimized with Bayesian methods. The test set was held-out until the end of the challenge, therefore we handled it as a separate entity here as well.

Methods

The authors confirm that all methods were carried out in accordance with relevant guidelines and regulations. The authors confirm that all experimental protocols were approved by the hackathon organizing team. Informed consent was obtained from all subjects or their legal guardians according to the local regulations of the hackathon organizers (COVID CXR Hackathon—Artificial Intelligence for COVID-19 prognosis: aiming at accuracy and explainabilityhttps://ai4covid-hackathon.ing.unimore.it/prizes/).

We have systematically went through input modalities with different model classes in order to answer whether chest X-ray screenings at admission aid clinical metadata also collected at admission. We know from the literature2,3,22 that prognosis prediction from clinical indicators is possible, therefore we set our baseline to be the best classical machine learning model that can also be explained and reasoned about. We then took imaging data and attempted to also predict the prognosis without clinical metadata. Moving on we set up a precision network, building on top of our image only model to also process the clinical features and integrate that information into our prediction as well. We do this in a multitask learning setting23, also predicting death as a binary output for the image only and precision networks to aid predicting power. Finally, we built a separate, interpretable model that reasons about clinical features in the form of attention maps on the image.

To sum up, we can categorize our experiments into two sets: explainable and non-explainable. In the first category, we have our baseline models for the metadata only, we also have this, with the additional Brixia-scores for each sample. Also in this category is our precision neural network which incorporates clinical metadata and images. Moving on to the explainable category, we have the baseline models with the Brixia-scores evaluated for SHAP scores and the explainable neural network that generates visual interpretations alongside its predictions (see Fig. S2 in the “Supplementary material”).

We evaluated several imputing methods, as our source training data was heavily corrupted by missing columns in specific hospitals, missing values for specific patients, etc. We turned to population sampling, population averages and also average imputing based on age groups as well as hospitals. We compare here these results between baseline models and as a result build an intuition on which imputing method could perform best in the long run for this specific dataset. The models were chosen based on popularity and general predictive power. We also explore whether adding additional information in the form of chest X-ray screenings at admission helps the predictive power or not.

Precision network

Having acquired additional features in the form of 2D Fourier transforms, Brixia-scores, segmentation masks and aligned images as well as the imputed clinical metadata, we created a precision network, that integrates all these input features into one single unit with several single-feature processing layers/heads. A convolutional backbone processed the pre-processed image and the segmentation mask and after an average pooling created a feature vector out of the visual input feature. Meanwhile, another smaller convolutional network processed the Fourier features into a Fourier-feature vector, whilst a fully connected network processed the imputed clinical metadata concatenated with the Brixia-scores. All the output feature vectors of these single-feature processing units were then concatenated together and passed through a fully connected head that did the prognosis prediction in addition to the death prediction in order to do enhance the precision of our predictions. In the inference phase we only used the prognosis output to predict the severity of COVID-19 outcomes for the test set. The major parameters, such as the type of the convolutional backbone, size of the last dense layer used before the classification output, batch size, number of epochs, learning rate, etc. (see Supplementary Fig. S2 for the full list of parameters) were tunable hyper-parameters for which we did hyperparameter-optimization.

We optimized the hyper-parameters for several models: (1) with image data only, (2) the image and metadata combined, (3) also we compared to the baseline models which were fitted on the metadata only (4) as well as the same baseline models with additional Brixia-scores. We have compared the performance of various architectures for each data modality.

Explainable models

There are multiple ways to talk about explainability. We can try to explain feature importances of baseline models with statistical methods or think of the model as a black box, but create it as such, that it outputs its own explanation. First, we looked at two baseline models that performed well in our baseline comparison and added the Brixia-scores to the meta features to introduce information from the imaging data into the explainable models. This falls into the former explainable model category, reasoning about feature importances regarding the prediction. We have created the explaining plots with the SHAP library24,25 and selected the logistic regression and the XGBoost models for the explainable baselines as these are the most popular choices amongst machine learning experts for initial evaluation and are also easy to evaluate with the SHAP package. Afterwards, we experimented with an interpretable neural network, such that it generates visual interpretations of its prediction while still being a black-box predictor.

Visual interpretations

The explainable network has two main features, one convolutional sub-network processes the image data into an unpooled feature map ($$16 \times 16$$ or $$8 \times 8$$) with many feature channels. The second sub-network is a small transformer block26 for the clinical metadata. Afterwards, a Bahdanau attention map is calculated between each processed, embedded feature (meta vs. image) and are stacked together into one large context vector, averaged along the feature maps spatial dimensions after applying the attentional weights (see Equation 1). We process the context vector (still being a high-dimensional feature vector for each meta column) further with two layers of bidirectional LSTM networks27 having it only outputting prediction of the prognosis to model the P(prognosis|imagemeta) distribution (see Fig. 3 for details). This model is optimized via stochastic gradient descent. During validation we tune hyper-parameters for the convolutional backbone, number of attention heads, dimensionality of the Bahdanau attention mechanism28, transformer encoding dimensions.

\begin{aligned} attention\_weights_{\ i} = SOFTMAX\Big (\mathbf {V_{i}} \times TANH\big ( \mathbf {W_{i}}^{image} \times encoded\_image + \mathbf {U_{i}}^{meta} \times encoded\_meta\_feature_{i} + \mathbf {b_{i}}\big )\Big ) \end{aligned}

Equation 1. Attention weights across the spatial dimensions of the encoded input image are calculated for each encoded meta feature. There are learnable weights $$\mathbf {V_{i}}, \mathbf {W_{i}^{image}}, \mathbf {U_{i}^{meta}}$$ indexed by $${\textbf{i}}$$ which corresponds to a specific meta feature. This setup makes it possible to learn different attention weights for the same image but a different input meta feature.

Every neural network model was trained with the Adam optimizer29 and exponential learning rate decay. We optimized the hyper-parameters for the validation balanced accuracy metrics with $$W \& B$$30. Hyper-parameters and specific configuration files can be found on the project’s GitHub repository https://github.com/csabaibio/ai4covid/tree/feature/paper-changes.

Results

Here we present our results featuring our baseline models, our precision network and later on we explore our explainable and our interpretable model in this scenario. We won first place [the first place was actually awarded to the authors of the baseline paper3 who resigned from the competition as they were data owners themselves] in the precision category of the challenge, and here we further explore our explainable model and look into additional explainable methods. We argue that the information contained in the CXRs at admission is present in the metadata, we tested this proposition via the trained models: we evaluated the agreement of our predictions of the logistic regression, XGBoost, image-only neural network and precision neural network models against the true labels. Creating this agreement output of each model (1 if agrees, 0 if it does not), we could create confusion matrices of these agreement vectors between the models. If two models have similar output structure: one predicts the same examples severe/mild and misses similarly, their confusion matrix will be mostly diagonal. On the other hand, if a model is superior to the other, it is agreeing with the ground truth labels more whilst the other is being wrong. We can see the latter on the first two figures of Fig. 4, where we compare the logistic and XGBoost models against the image-only neural network’s outputs. Moving on to our precision network, we show that its predictions match the baseline models extremely well, which is presented in Fig. 5.

Classical model performance on clinical metadata

We compared balanced accuracy, F1-score and AUC (area under the receiver-operator characteristics curve in Table 1 (and Supplementary Fig. S1). We found that the best performance was achieved with population average. We also validated this later on, with our precision network, and explainable network as well. The results were calculated with 5-fold cross-validation on the training data without hyperparameter tuning.

Precision network results

We conclude from the results shown in Table 1 (and Supplementary Fig. S1) that the deep learning based models are on-par with the classical methods for this dataset and that the X-ray scans at admission do not improve the prediction of the prognosis significantly. Our best, cross-validated balanced accuracy score for the baseline models was 0.773 ± 0.014 with the SVM, meanwhile the best model on the held-out test set was the logistic model scoring a BA score of 0.763 ± 0.005, both highlighted in Table 1. Compared to the model solely with image based input data which included the lung mask score-map created with BSNet alongside with the 2D Fourier transforms of the input image (Fig. 2). We hyper-optimized the parameters of this network as well with the naive Bayes method (Supplementary Fig. S3) for the validation balanced accuracy score and concluded that our best overall BA score achievable via cross-validation was 0.764 ± 0.020, but as we know that the prediction problem has high variance, we could also do an ensemble prediction of the cross-validated models and evaluate the score with bootstrapping, in which case we achieved 0.734 ± 0.021. Both of these are below the scores of the baseline models, so it seems that the different input modalities actually hurt performance and there is no additional information in the early X-ray screenings. We sum up our findings in Table 2 and conclude that including the initial X-ray screenings into the predictive models is unnecessary and even detrimental for the accuracy of the prediction of the prognosis.

Explaining the output

Based on Fig. 6 (and Supplementary Fig. S4) we can conclude that the Brixia-scores, LDH (lactate dehydrogenase concentration in blood (U/L)) levels, being in a specific hospital, age, sex and body temperature of the patient are relevant features in generating the prediction of the prognosis. We can also see that body temperature, blood oxygen level and age also being highly influential, which we have expected (see Table 2). On the other hand, most of the remaining features are not highly affecting the prediction which is a good indicator of less predicting power.

For our explainable model we averaged the output attention maps of all images and metadata features both for the training and test samples to be able to spot any difference, and created mean attention maps for each relevant meta feature with respect to the predicted outcome. These attention maps correlate well with some of the metadata features, such as coughing, hearth failure, obesity, cardiovascular disease, etc.

We superimposed the mean severe and mild prognosis images behind the attention maps in Fig. 7. This way it can be seen in Fig. 8 that some features that are in connection with the hearth (e.g. hearth failure, cardiovascular disease) put high attention to that spatial region of the lung. Also the features that cannot be spatially localized in the attention maps tend to have a spread-out attention in the whole lung, such as obesity, dementia, glucose, days of fever etc. It also appears, that features which are highly correlated with the lungs themselves, have high attention on specific regions of the lungs (bottom or sides) as in the case of arterial blood oxygen pressure, oxygen percentage, difficulty of breathing, coughing, etc.

In Fig. 8 we show the same attention maps on one of the validation subsets and the test data to highlight that the model behaves the same way during validation and testing. We also found that taking the average Brixia-scores over severe and mild prognosis cases we get high correlation with the produced attention maps (Table 3). The Brixia-scores themselves function as a $$3 \times 2$$ matrix, scoring 6 segments of the lung, we up-sampled these to the size of our attention maps and calculated the absolute correlation between them with respect to the predicted prognosis by the network.

Among the linearly highly correlating features against the Brixia-scores (see Table 3) we can highlight the straight-forward ones which are: obesity, heart related meta features (high blood pressure, atrial fibrillation, ischemic heart disease, heart failure, etc.) which have high attention in similar regions where the Birxia-scores are high as well. We can conclude that these meta-features are even more important in the severe prognosis outcome, as they suggest initially worse conditions which eventually lead to a worse outcome of COVID-19.