Introduction

Since the first cases of severe acute respiratory syndrome coronavirus 2 (SARS-COV-2) were diagnosed in December 2019 in the Chinese city of Wuhan, the coronavirus disease 2019 (COVID-19) has spread rapidly1. The strategies applied in the vast majority of countries to control the virus transmission have been ineffective. The first results concerning the safety and effectiveness of different types of vaccines have raised optimism in the scientific community due to the possibility of controlling COVID-192. But recent real-world data indicate that the effectiveness of the BNT162b2 and ChAdOx1 vaccines against infections with symptoms or high viral burden is reduced with new variants such as delta. In addition, infections in vaccinated patients have similar viral loads compared to unvaccinated patients3. This justifies the high number of new cases and a mortality rate difficult to eradicate in some countries with high vaccination rates4. On the other hand, a large part of the world population remains susceptible to SARS-COV2 infection, due to lack of access to vaccines, making herd immunity likely unachievable.

The early detection of symptoms suggestive of infection, rapid and efficient testing, contact tracing and isolation are the basis of an effective screening strategy to control transmission of COVID-19 and decrease the disease burden on healthcare systems. The Achilles' heel in the fight against this disease is the large number of patients who are asymptomatic or have only a few symptoms that are difficult to differentiate from a common cold, but who are nonetheless able to transmit the disease5,6. It was estimated that 51.9% of SARS-COV-2 infected cases were asymptomatic or had only 1 or 2 symptoms suggestive of COVID-197. The reference diagnostic tool for COVID-19 is reverse transcriptase polymerase chain reaction (RT-PCR). Its accessibility may be limited for low-resource healthcare systems and its cost and time requirements preclude its use as a mass triage tool. Recently a screening tool based on a machine learning model including clinical features and symptoms has been constructed to prioritize testing for COVID-198. It was found that a predictive model for COVID-19 that included the combination of symptoms and wearable sensor data performed better than a model based on symptoms alone9.

Olfactory dysfunction (OD) has recently been described as one of the most prevalent symptoms reaching 50% to 75% of COVID-19 patients and could be used as a means of screening to help identify people who should self-isolate10,11,12. With the delta variant OD has been described in the 39% of the cases being as well one of the most prevalent symptom13.

A symptom predictive model for COVID-19 based on a smartphone app including age, sex, loss of smell and taste, persistent cough, severe fatigue and skipped meals obtained a sensitivity of 65%14. At the time of diagnosis, a recent prospective study found that 31% of patients affected by COVID-19 presented OD15. Between 11.8 and 23% of cases presented OD before any other symptoms10,16. The validated olfactory tests are subjective and difficult to implement. A recent study showed that a simplified test based on the identification and the assessment of intensity of three different scents was able to detect unperceived OD in COVID-19 patients15.

Hydroalcoholic gels are widely distributed as they are one of the main strategies for decreasing virus transmission17. Fragrance essential oils such as lavender, eucalyptus and lemon make them more pleasant and can enhance their anti-viral effect18,19. These features make an aromatized hydroalcoholic gel a good candidate for being used as part of a simple, fast and cost-effective large-scale olfactory screening test.

The aim of this study was to develop and validate, using cross-validation techniques, a machine learning diagnostic predictive model for COVID-19 mass screening using symptoms and a simple olfactory test based on an aromatized hydroalcoholic gel, which could be especially useful when testing resources are limited.

Results

Characteristics of the study population

During the study period 3788 patients underwent RT-PCR to diagnose COVID-19 at one of the study health centres. The inclusion of cases and RT-PCRs performed per week at the centres while participating in the study can be consulted in the supporting information Fig. S1. A total of 626 patients were initially included in the study protocol. Of these, 107 patients were excluded because of incomplete data or exclusion criteria as shown in Fig. 1.

Figure 1
figure 1

Flow chart.

Full size image

The final analysis of the study included 519 patients, out of whom 341 patients (65.7%) were from primary care and 179 (34.3%) were from the hospital Emergency Department. According to the criteria for carrying out a RT-PCR test, 386 (74.4%) had at least one symptom suggestive of COVID-19, 118 (22.7%) were asymptomatic and were close contacts of a COVID-19 case, and 15 (2.9%) were asymptomatic and were tested for unknown reasons. A positive RT-PCR was found in 117 patients (22.5%) and a negative RT-PCR was found in 402 patients (77.5%).

The mean (SD) age of the study population was 42.3 (16.3) years, the age range was between 18 and 98 years and 48% were male. None of the patients requiring hospital admission died. Table 1 shows the background and clinical characteristics of the study population.

Table 1 Background and clinical characteristics of the study population.
Full size table

COVID-19 symptoms and olfactory test results

The mean (SD) number of days of the symptom evolution was 5.8 (5.6) for the COVID-19 positive patients and 5.1 (12.1) for the COVID-19 negative patients with an absolute difference of 0.75 (95% CI − 1.35 to 2.84; P = 0.48). The symptoms most strongly associated with COVID-19 were OD and GD. Fever, dry cough, asthenia, myalgia, headache, diarrhoea, OD, and GD were the eight symptoms associated with COVID-19. Table 2 shows the reported symptoms and olfactory test results in the population study.

Table 2 Patient reported COVID-19 symptoms and olfactory test results.
Full size table

In the total population study, the olfactory test 1 was positive in 267 patients (51.4%) and negative in 252 patients (48.6%). Among patients with a positive olfactory test result 112 cases (41.9%) identified the gel smell as alcohol, 57 cases (21.3%) as cologne, 27 cases (10.1%) as aromatic herbs, 10 cases (3.7%) as non-citrus fruits (3.7%), 6 cases (2.2%) as alcoholic beverages (2.2%) and 22 cases (8.2%) as other responses. In 25 cases (9.4%) participants reported that they “didn’t smell anything at all” and in 8 cases a “don’t know” response (3%) was reported. Among patients with a negative olfactory test result, 207 cases (82.1%) identified the gel smell as lemon, 26 cases (10.3%) as citrus, 13 cases (5.1%) as orange, 2 cases (0.8%) as tangerine, 2 cases (0.8%) as citronella, and 2 cases (0.8%) as lime.

A positive olfactory test 1 was associated with COVID-19 (OR 1.86; 95% CI 1.22–2.85, P < 0.01). The response “do not smell anything at all” was strongly associated with COVID-19 (OR 4.06; 95% CI 1.8–9.17). Among the 13 asymptomatic COVID-19 positive patients, 10 (76.9%) had a positive olfactory test 1 result and only 3 patients presented a negative olfactory test 1. An olfactory test 1 positive result in asymptomatic patients was associated with COVID-19 (OR 3.94; 95% CI 1.03–15.03). The detailed results of the olfactory test and the diagnostic values of the relevant symptoms, the combination of symptoms and olfactory test for predicting COVID-19 were available in the S1 and S2 Tables.

Results of the machine learning predictive model

Table 3 shows the results of the different classification trees constructed with machine learning according to the variables introduced in the model.

Table 3 Results of machine learning model.
Full size table

By only introducing the relevant symptoms into the model, the sensitivity was 0.86 (95 CI 0.79–0.92), the specificity was 0.37 (95% CI 0.33–0.42) and the AUC was 0.86 (0.81–0.9) for the total population study, and 0.97 (95%CI 0.92–0.99), 0.11 (95% CI 0.07–0.15) and 0.89 (95% CI 0.81–0.9) respectively for the symptomatic population. The sensitivity and specificity obtained was 0.94 (95%CI 0.88–0.98) and 0.32 (95% CI 0.28–0.37) when the olfactory test was introduced into the model for the total study population. The constructed sensitive classification tree only took into account the result of the olfactory test 1 and ignored the result of the olfactory test 2. Considering other clinical variables, the model also included sex and age, reaching a sensitivity of 0.97 (0.91–0.99), a specificity of 0.39 (0.34–0.44) and an AUC of 0.87 (95% CI 0.83–0.92) for the total study population, and 0.98 (95%CI 0.93–1), 0.31 (95% CI 0.26–0.37) and 0.89 (95% CI 0.84–0.93) for the symptomatic population, respectively. The specific classification tree built took into account the relevant symptoms and age and obtained a sensitivity of 0.29 (0.21–0.38), a specificity of 0.95 (95% CI 0.92–0.97) and an AUC of 0.85 (0.8–0.89) for the total study population.

The resulting receiver operating characteristic (ROC) curve is shown in the Fig. 2 and the precision-recall curve of the sensitive and specific tree algorithm are shown in Figs. S2 and S3.

Figure 2
figure 2

Sensitive and specific classification tree algorithm ROC curve.

Full size image

Discussion

The combination of symptoms and a simple olfactory test based on identifying the smell of a hydroalcoholic gel made it possible to develop a predictive model with high sensitivity, which has important clinical implications.

A predictive model based on symptoms reported on a smartphone-based app obtained a lower sensitivity of 0.65 (95% CI 0.62–0.67) and a lower AUC of 0.76 (95% CI 0.74–0.78) to predict COVID-19 than our predictive model14. Another predictive model using machine learning based on symptoms, gender, age and close contacts obtained a lower sensitivity which was between 0.85 and 0.87 depending on the possible working points and a similar AUC of 0.86 (95% CI 0.85–0.87)8. The different results of our model, depending on the variables included, show similar or even higher diagnostic values with respect to those models proposed as population screening. The model presented has the advantage that it includes asymptomatic patients and does not include close contacts in its variables as this could be difficult to determine in a situation of community transmission. To our knowledge, this is the first model including an olfactory test and built using a prospective population-based study. It is important to highlight that the symptoms combination have the higher weight in the model results, although the low false negative rate of the olfactory test among asymptomatic COVID-19 patients, helps improving the sensitivity of the model. The sample for this population-based study was obtained from patients following current indications for RT-PCR testing, and the sensitivity and specificity figures obtained make this model useful as a population-based screening.

In the same direction, new advances are being made in the development of new point-of-care, rapid, sensitive and inexpensive diagnostic methods to detect COVID-19 that can be useful to fight this pandemic and prepare for the next ones20. An effective mass screening based on antigen test detection of SARS-CoV-2 has been described21. But a recent review of several antigen tests on the market states that two-thirds had overall sensitivities (30.8%-68.9%) below the World Health Organization recommended standard of ≥ 80% raising concerns whether the antigen detection alone is sufficient for COVID-19 mass screening22. Combining our predictive model with an antigen test could be a promising mass screening strategy.

Regarding the olfactory test, it has obtained a sensitivity almost twice as high as a more complex olfactory test for predicting COVID-19 based on identifying the smell of three scented paper strips and a 4-item scale intensity rate15. In addition, the simplicity of the olfactory test means it can be implemented as a self-test, making it a more suitable population screening olfactory test than any test reported so far. The wide distribution of this predictive tool due to its low cost also contributes to improving the disease situational awareness of the population. This may be especially useful in those scenarios where preventive measures are gradually being relaxed and there is still a need to protect older and more vulnerable people due to the rapid waning of vaccine protection over time against new variants such as omicron23,24.

Our work has some limitations. Our study was conducted when the alpha variant was the most predominant variant in our area. Recent data show a high prevalence of classical symptoms such as cough, fever and olfactory and taste dysfunction among vaccinated and unvaccinated COVID-19 infected patients where the delta variant is predominant suggesting that our model may be useful in this setting3. The omicron variant has been associated with a reduced capacity to penetrate olfactory epithelial cells and produce anosmia25. A prospective study based on a focused questionnaire for assessing olfactory function found that the prevalence of OD caused by the omicron variant was 24.6%26. This drop in the prevalence of OD with this variant may affect the sensitivity of our model. Regarding the olfactory test, a high percentage of patients identified the smell of the gel as alcohol. The alcoholic matrix of the gel could hinder olfactory recognition, explaining the low specificity found. Moreover, the patient’s capacity to identify smell may decrease in an uncomfortable scenario such as an emergency department during a pandemic.

It is important to highlight that in our study no side effects related to the inhalation of the hydroalcoholic gel were reported. One study described that repeated exposure to a hydroalcoholic gel by inhalation does not increase blood ethanol levels27. The side effects described in the literature are related to the occurrence of dermatitis or are due to the ingestion of the gel28,29.

The value added by our COVID-19 predictive model in this field is its potential applications such as its inclusion in a mass testing strategy in order to save costs. Our predictive model could be useful to quickly rule out non-infected patients and for selecting the population that could benefit from a more expensive diagnostic test such as antigen testing or RT-PCR helping to reduce the costs for the health system or for companies with a rigorous occupational risk policy such as hospitals, nursing homes or large companies. It could also be especially useful for controlling transmission in those regions where testing resources are limited due to scarce economic resources or logistical difficulties.

This predictive model has been patented (EP 21 382 524.3) and is available upon request. The effectiveness of its implementation in different epidemiological settings should be tested by performing external validations; therefore, the collaboration of the scientific community is encouraged.

Conclusion

A machine learning predictive model for COVID-19 using symptoms and a simple olfactory test based on an aromatized hydroalcoholic gel showed high sensitivity for diagnosing COVID-19. The capacity of this predictive model to detect infected SARS-COV-2 patients among asymptomatic patients makes it a promising tool for the fight against COVID-19. This predictive model could be especially useful for mass screening when testing resources are limited.

Methods

Study design and setting

This is a population-based prospective cohort study conducted following the TRIPOD Statement for multivariable diagnosis prediction model30. The study was carried out in the Emergency Department of Sant Joan University Hospital of Reus, which is the reference hospital of the region, and in all five primary care centres of Reus (Catalonia, Spain) public health network.

The municipality of Reus is located in the Mediterranean area, has a surface area of 52.82 km2 and at the beginning of 2020 it had a population of 106,168 inhabitants and a density of 2010.0 inhabitants/km231. This study was approved by the Ethics Committee of the Pere i Virgili Health Research Institute (Ref: 120/2020) and the IDIAP Jordi Gol Clinical Research Ethics Committee (Codi: 20/114-PCV). The study was conducted in accordance with de Declaration of Helsinki and Good Clinical Practices. All study participants were required to sign an informed consent form.

Participants

The study included consecutive patients undergoing RT-PCR for the first time to rule out COVID-19 infection who consulted the hospital emergency department or their primary care centre between 15 June and 11 September, 2020. Patients were tested for presenting symptoms suggestive of COVID-19 or for being close contacts of a confirmed COVID-19 case. Close contacts were considered those persons who had shared an area with a positive case at a distance of less than 2 m, for more than 15 min, without protection and from 48 h prior to the onset of symptoms.

The study did not include patients under 18 years of age, patients who did not sign the informed consent form, and patients with pathologies or conditions that may interfere with the olfactory function, such as any degree of cognitive impairment, Parkinson's disease, chronic rhinosinusopathy, head trauma, nasal obstruction, treatment with high concentrations of oxygen, acute respiratory failure, patients with an altered state of consciousness, or who use inhaled corticosteroids.

Olfactory test development

A multidisciplinary cooperation was established for creating a hydro-alcoholic hand sanitizing gel that meets current requirements in terms of its composition32.

Based on the literature and habits of our Mediterranean study population, it was determined that the most suitable odoriferous substance was lemon33. Tests were carried out with different concentrations of lemon essential oil and lemon fragrances of synthetic origin. The composition of the gel was adapted to attenuate the smell of alcohol. A study was carried out to determine the most effective composition with and without thickener. Gas chromatography and mass spectrometry were used to obtain semi-quantitative results. A headspace sampling technique was used to establish the effectiveness of the volatile odoriferous substance that evaporated from the hydroalcoholic gel at 37 ºC. Finally, two hydroalcoholic gels with increasing concentrations of lemon essential oil were created as an olfactory test.

Description of the olfactory test

The olfactory test was performed by appropriately trained primary care and emergency nurses before the sample for SARS-COV-2 RT-PCR was collected. Therefore, both the patient and the healthcare personnel did not know the patient’s infection status. Firstly, the test consisted of applying 1 ml of 0.3% gel (olfactory test 1) using a dispenser onto the patient’s palm. Then the patient rubbed the gel on their hands and waited for 3 s. The patient was then asked to smell their hands and to “please, identify the smell of this gel”. The answer was recorded on the basic data collection sheet regardless of the result. If the answer was not lemon or if it was inconclusive, the same test was repeated after 30 s with the 0.5% gel (olfactory test 2). The olfactory test was considered negative if the patient recognized a citrus fruit, and the olfactory test was considered positive if the patient could not smell the gel or did not recognize a citrus fruit.

Data collection

A data collection sheet was completed by the attending nurse before taking the sample for the RT-PCR test. It included the results of the two olfactory tests when both were performed. It also included age, gender, duration of symptoms (in days), and a yes/no questionnaire to check for symptoms such as fever, dry cough, dyspnoea, anorexia, myalgia, headache, diarrhoea, asthenia, productive cough, sore throat, OD or gustatory dysfunction (GD), others or no symptoms. The RT-PCR test for detecting SARS-COV-2 was considered the gold standard for diagnosis. During our study, the RT-PCR was performed by trained personnel according to the technical considerations of the manufacturer using a double sampling of the pharynx and the nose. The conservation of the sample and the transfer to the laboratory followed the channels of the usual clinical practice of the centre. RT-PCR tests were carried out with the VIASURE SARS-COV-2 Real Time PCR Detection Kit (CerTest Biotec, Zaragoza, Spain), or with the Procleix1 method in a Panther automated extractor and amplifier (Grifols Laboratories, Barcelona, Spain). Once all the data collection sheets were completed, the medical digital records were consulted and the RT-PCR test results were recorded, as well as the patient’s background, evolution and discharge diagnosis. Regarding the severity of the disease, the patients attended and discharged immediately were considered as mild, those admitted to the hospital as moderate and those requiring ICU during hospitalization as severe.

This study was conducted at the beginning of the second wave of COVID-19 in our region34. The 14-day cumulative incidence of COVID-19 cases in the city of Reus increased gradually from 0.9 cases/100,000 inhabitants on 15 June to 376.09 cases/100,000 inhabitants on 24 August35.

Model development and internal validation

First, an analysis was conducted to explore the independent variables associated with COVID-19. The symptoms that proved to be statistically significant in a logistic regression predictive model, were fever, dry cough, myalgia, headache, diarrhoea, asthenia, altered sense of smell, and altered sense of taste. These 8 symptoms were defined as relevant and presented as well as their combinations the strongest associations with the predicted event. Diagnostic values were calculated for each symptom separately and their combinations for the total population and the symptomatic population. The productive cough variable was also included as a relevant symptom.

In order to facilitate the search for the best combination of variables to predict the diagnosis of COVID-19, we decided to build a model based on a decision tree constructed by machine learning that could also facilitate its clinical use following guidelines36. Other modelling methodologies such as random forests or artificial neural networks were discarded because they need larger training datasets and also because their interpretability is not as straightforward as that of decision trees. Priority was given to the construction of a parsimonious model using as few variables as possible, robust by minimising missing data, transparent and simple. Moreover, minimising false negatives was also a priority in the predictive model construction to allow its use as a population screening.

The 8 relevant symptoms and the result of the olfactory test were variables significantly associated with COVID-19. Sex and age were as well sequentially introduced into the model as these variables were considered clinically relevant10. The final model had 11 independent variables therefore the study sample complied with the standard rule of ten clinical events per predictive variable37.

The number of relevant symptoms was counted for each patient and this new variable was used to develop the model based on classification trees using a recursive partitioning algorithm38. The growth of the trees was controlled to avoid overfitting the data. Trees were pruned to the size that minimized the cross-validated error. In addition, these classification trees were built using the following parameters: the splitting index was the Gini coefficient; the minimum number of patients in any node of a tree for a split to be attempted was set at 30; the minimum number of patients in any terminal node of a tree was set at 10; node splits were only attempted if they improved the fit by a factor of 0.01; and the number of cross-validations to be run was set at 10. The sizes of the trees obtained using this strategy range between six and seven leaves (terminal nodes), which proves that overfitting has been successfully avoided.

In order to obtain different values of sensitivity and specificity in the resulting classification trees, distinct costs of false positives and false negatives were used in the loss matrix parameter that drives the splitting function of the classification tree algorithm. In particular, the specific classification tree was grown using equal cost values for false positives and false negatives, while the sensitive classification tree was grown using a cost value for false negatives that was eight times the cost value for false positives.

The internal model validation was carried out using the R package cross validation techniques in machine learning.

Statistical analysis

The quantitative variables used in this study were described using the mean, the standard deviation (SD), the median and the first and third quartiles. The differences between means and their corresponding 95% confidence interval (CI) were also used to compare groups of patients. Categorical variables were described using the number of cases, percentages and 95% CI. Comparisons between groups of patients were performed using Student’s T test for quantitative variables, while the chi-squared test was used for categorical variables. Groups of patients were also compared in terms of the risk difference and odds ratio (OR) of the binary variables, and their corresponding 95% CI. All tests were two-tailed and P-values lower than 0.05 were considered statistically significant. Diagnostic values in terms of sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio and negative likelihood ratio, as well as their corresponding 95% CI, were calculated for the binary variables and smell tests. Several predictive models were analysed to handle missing data in the study protocol. A data-complete analysis was adopted over other strategies due to the low relevance of the missing data in the final results of the predictive machine learning model. All statistical analyses were performed using R software version 4.0.