Comparing the clinical efficacy of COVID-19 vaccines: a systematic review and network meta-analysis

New Coronavirus Disease 2019 (COVID-19) vaccines are available to prevent the ongoing severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic. We compared the efficacy of new COVID-19 vaccines to prevent symptomatic and severe disease in the adult population and to prevent symptomatic COVID-19 among the elderly. Leading medical databases were searched until August 30, 2021. Published phase 3 randomized controlled trials (RCTs) evaluated efficacy of the vaccine to prevent symptomatic and sever COVID-19 in adults were included. Two reviewers independently evaluated the literature search results and independently extracted summary data. The risk of bias was evaluated using the Cochrane Risk of Bias Assessment Tool. We performed a network meta-analysis (NMA) according to PRISMA-NMA 2015 to pool indirect comparisons between different vaccines regarding their relative efficacy. The primary outcomes were the efficacy of the vaccine against symptomatic COVID-19 in adults (PROSPERO registration number: CRD42021235364). Above 200,000 adult participants from eight phase 3 RCTs were included in NMA, of whom 52% received the intervention (active COVID-19 vaccine). While each of nine vaccines was tested in the unique clinical trial as compared to control, based on indirect comparison, BNT162b2 and mRNA-1273 vaccines were ranked with the highest probability of efficacy against symptomatic COVID-19 (P-scores 0.952 and 0.843, respectively), followed by Gam-COVID-Vac (P-score 0.782), NVX-CoV23730 (P-score 0.700), CoronaVac (P-score 0.570), BN02 (P-score 0.428), WIV04 (P-score 0.327), and Ad26.COV2.S (P-score 0.198). No statistically significant difference was seen in the ability of the vaccines to prevent symptomatic disease in the elderly population. No vaccine was statistically significantly associated with a decreased risk for severe COVID-19 than other vaccines, although mRNA-1273 and Gam-COVID-Vac have the highest P-scores (0.899 and 0.816, respectively), indicating greater protection against severe disease than other vaccines. In our indirect comparison, the BNT162b2 and mRNA-1273 vaccines, which use mRNA technology, were associated with the highest efficacy to prevent symptomatic COVID-19 compared to other vaccines. This finding may have importance when deciding which vaccine to use, together with other important factors as availability of the vaccines, costs, logistics, side effects, and patient acceptability.

www.nature.com/scientificreports/ process, lasting for decades before the product has been approved for clinical use 3 . Shortly after the start of the SARS-Cov-2 outbreak, scientists began racing to develop an effective and safe vaccine against SARS-CoV-2, based on new and old vaccines technologies 4 . Within less than two years period, there are more than 300 vaccine candidates globally, 117 vaccines in different clinical stages of development, including 30 of them in phase 3 5 5 .
The vaccines with EUA use various vaccine technologies, including mRNA 6,7 , virus vector [8][9][10] , and adjuvanted recombinant protein nanoparticles 11 . Each technology has its advantages and limitations 12 . mRNA-1273 7 and BNT162b2 6 are the newest generations of mRNA vaccines. mRNA vaccines do not contain the antigen itself but deliver the genetic information for the antigen, and vaccinated individual synthesizes antigens in the host cells 13 . In this technology, all components are produced via chemical synthesis, which allows fast-track development in the event of a pandemic. The advantages associated with mRNA vaccines include high efficacy and relatively low severity of side effects. Before the current pandemic, mRNA vaccine technology seems promising in several diseases such as cytomegalovirus and Zika virus 14 , however, mRNA vaccines were not licensed for human use before the SARS-Cov-2 pandemic 15 . Thus, there are relatively short-term efficacy and safety data of COVID-19 mRNA vaccines, including recently published short-term real-world studies 6,7,[16][17][18][19][20] . NVX-CoV2373 is an adjuvanted recombinant protein vaccine that contains Matrix-M1 adjuvant and a recombinant full-length wild-type SARS-CoV2 spike glycoprotein 21 . The same technology platform was used in the recently EU-approved Janssen Ebola vaccine 22 . ChAdOx1, Ad26CoV2.S, and Gam-COVID-Vac are viral vector-based vaccines [8][9][10] . The technology uses antigen cloned into a viral vector that cannot reproduce. The viral vector imitates the viral infection disease state and can produce more robust cellular immune responses compared to the recombinant protein vaccine. Adenoviral vector vaccines' safety has been extensively studied, and adenoviral vector-based therapeutic drugs are used in clinical practice 23 . In parallel with new technologies, recently published RCT reported the efficacy of three new whole-virus inactivated vaccines 24,25 .
For most new SARS-CoV-2 vaccines the efficacy data are based on the results of single phase 3 RCT, together with recently published real-world data for some of them [6][7][8][9][10][11]17,19,24,25 . Widespread vaccination programs have commenced in several countries, while the long-term effectiveness of COVID-19 vaccines is lacking. Recently published meta-analysis of eight COVID-19 vaccines, that have published the data of phase 3 randomized controlled trials (RCTs), reported excellent efficacy (pooled Risk Ratio (RR) to prevent symptomatic disease of 0.17; 95% Confidence Interval (CI): 0.09-0.32) 26 . While all new COVID-19 vaccines were found to be very effective to prevent symptomatic disease as compared to control, no study compared the efficacy between different vaccines.
The conventional meta-analysis approach can only compare two interventions at a time. Using the network methods enables the evaluation of multiple treatments in a single analysis. In the absence of a trial that directly compared two different treatments, an indirect comparison can be performed. Indirect evidence refers to the evidence obtained through a common comparator 27 . Network meta-analysis published on March 2021 included data about four COVID-19 vaccines and provided the following rank of effectiveness: BNT162b2 ≈ mRNA-1273 > Gam-COVID-Vac > > ChAdOx1 28 .
We aimed to integrate updated published data from phase 3 RCTs about different COVID-19 vaccines and provide an indirect comparison between vaccines' clinical efficacy to prevent symptomatic and severe disease, using network meta-analysis. Our results may provide additional evidence-based information to help choose the best policy to achieve the most significant public health benefit.

Methods
Data sources and search strategy. We performed a comprehensive database search which included PubMed/Medline, Embase, including Mesh/Emtree terms search, Clinical Trials Registry Clinicaltrials.gov, and The Cochrane Library using the following keywords: COVID-19, severe acute respiratory syndrome coronavirus, Coronaviridae Infections, coronavirus, sudden acute respiratory syndrome, vaccines, vaccine, randomized controlled trial, controlled clinical trial, clinical trial, phase II/III, phase III. The search strategies incorporated index terms (Mesh) and text words for the search concepts. The search words are detailed in online-only supplements. Databases were searched up to August 30, 2021, without language or date restrictions.
The primary outcomes were the clinical efficacy of the vaccine against symptomatic laboratory-confirmed COVID-19. Secondary outcomes were the efficacy to prevent severe COVID-19 infection and vaccine efficacy among the elderly.
The systematic review and network meta-analysis were performed following Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) 2020 framework guidelines 29 . The protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) on February 5, 2021 (CRD42021235364).
Inclusion and exclusion criteria. We included published phase 3 RCTs to evaluate the vaccine's efficacy to prevent symptomatic COVID-19. The following publications were excluded from analysis: phase 1 and phase 2 RCTs, non-randomized trials, observational studies, duplicated reports, pharmacokinetic studies in healthy adults, reviews, expert opinion, editorials, letters to the editor, and comments. examined the list of titles, the abstracts, and finally, the full-text articles for eligibility using the Rayyan web software for systematic reviews 30 . Disagreements were resolved through consensus.

Data collection.
The following data were extracted by two independent reviewers: study details (identifier, study design, geographical location, study period, publication year, length of follow up), participant details (number of participants, study population, age and gender, co-morbidities, SARS-Cov-2 variants), intervention details (vaccine name, vaccine platform, vaccine regimen), details about efficacy outcomes: number of cases of symptomatic disease, number of cases of severe disease, number of cases of symptomatic disease in participants above the age of 60 years (raw data). Disagreements between reviewers were resolved through consensus.
Quality assessment and risk of bias. The risk of bias of the randomized control trials was assessed by two independent reviewers using the Cochrane tool for assessing the risk of bias for randomized control trials (RCT) 31 .

Statistical analysis.
We implemented a network meta-analysis according to PRISMA-NMA 2015 32 . To investigate the differences in efficacy between various vaccines, we performed a pairwise network meta-analysis, using a random-effects model [33][34][35][36] . In the absence of trials that directly compared two different vaccines, only indirect comparisons have been performed. The network incorporated raw data of vaccine efficacy compared to control from each included study. RRs and 95% CIs for indirect comparisons between different vaccines regarding their relative efficacy was calculated using the pairwise method. Vaccine efficacy was ranked using P-scores derived from network point estimates. The P-score is a frequentist equivalent to the Bayesian network surface under the cumulative ranking curve. The P-score of intervention can be interpreted as the mean extent of certainty that one intervention is better than another intervention, and can be used to rank an intervention within a range of interventions, measured on a scale from 0 (worst) to 1 (best) 37 .
To compare vaccines efficacy to prevent severe disease, we incorporated raw data of severe cases among vaccinated and control groups, as reported in each study. RRs and 95% CIs for indirect comparisons between different vaccines regarding their relative efficacy was calculated using the pairwise method. Vaccine's efficacy to prevent severe disease was ranked using P-scores derived from network point estimates.
We applied pairwise network meta-analysis, using a random-effects model to compare vaccines' efficacy to prevent symptomatic disease among the elderly. The network incorporated raw data of vaccine efficacy compared to control in patients above 60 years old from each included study. Vaccine's efficacy to prevent symptomatic disease among the elderly was ranked using P-scores derived from network point estimates.
Analysis was performed using R Version 3.4.3 and the "netmeta" package Version 0.9-8 38 .

Results
We identified eight phase-3 RCTs that reported primary or preliminary CODIV-19 vaccine efficacy, with contributory data from nine publications [6][7][8][9][10][11]24,25,39 . The search and selection processes are illustrated in eFigure 1. The characteristics of included studies are summarized in Table 1. Data from above two hundred thousand participants are included in our network
Development of severe disease. Additionally, we evaluated the efficacy of the vaccines to prevent clinically significant severe COVID-19. The data of severe disease were available from five studies, a total of 107 cases of severe disease (eTable 1) 6,7,[9][10][11]24 . eTable 2 summaraizes sever COVID-19 definitions, as defind in the inclided studies. When the indirect comparison between the seven vaccines was performed, Gam-COVID-Vac and mRNA-1273 vaccines were ranked with the highest efficacy to prevent a severe COVID-19 (P-scores 0.899 and 0.816, respectively), followed by NVX-CoV23730 (P-score 0.531), BNT162b2 (P-score 0.500), Ad26.COV2.S (P-score 0.34), WIV04 and HB02 (P-score 0.384) ( Table 2). However, no vaccine was statistically significantly associated with a decreased risk compared to other vaccines, although there was a trend present with mRNA-1273 and Gam-COVID-Vac vaccines compared to the other vaccines for a lower risk for severe disease (Fig. 3).

Risk of bias.
The risk of bias was evaluated for all published studies. It was classified as having some concerns for four studies [6][7][8]11 and it was deemed moderate for other studies 9,10,24,25 (eFigure 2).  26   were associated with the highest decrease in the relative risk for symptomatic COVID-19 compared to the other vaccines. BNT162b2 vaccine was associated with an 85% decreased relative risk of symptomatic disease than ChAdOx1 and Ad26.COV2.S (RR 0.15, 95% CI 0.07-0.31 and RR 0.15, 95% CI 0.07-0.31, respectively). the mRNA-1273 vaccine was 79% more effective in preventing symptomatic COVID-19 than ChAdOx1 and Ad26. COV2.S (RR 0.21, 95% CI 0.11-0.41 and RR 0.21, 95% CI 0.11-0.41, respectively) (Fig. 1). Ranking BNT162b2 and mRNA-1273 vaccines as best interventions over other competing vaccines to prevent symptomatic disease (P-score 0.95 and 0.84, respectively) ( Table 2). Our results are consistent with previously published data, provided the following rank of effectiveness: BNT162b2 ≈ mRNA-1273 > Gam-COVID-Vac > > ChAdOx1 28 . We did not find any statistically significant difference between the vaccines' efficacy to prevent symptomatic disease among the elderly.
The Ad26.COV2.S, ChAdOx1, and Gam-COVID-Vac are DNA vaccines encoding the SARS-CoV-2 spike (S) protein 40 . In our analysis, the Gam-COVID-Vac vaccine was more effective in preventing symptomatic COVID-19 as compared to Ad26.COV2.S and ChAdOx1 vaccines (RR 0.25, 95% CI 0.14-0.46 and RR 0.25, 95% CI 0.14-0.46, respectively) (Fig. 1). One possible explanation for the reduced efficacy of Ad26.COV2.S vaccine is a single-dose regimen compared to the two-dose regimen of Gam-COVID-Vac. A study is evaluating a two-dose administration of Ad26.COV2.S vaccine began participant recruitment during November 2021 41 . Also, higher efficacy of Gam-COVID-Vac as indirectly compared to ChAdOx1 vaccine may be explained by two different vectors' technology used in former. Using heterologous viral vectors for each dose allows the minimization of host immune responses against the vector components 42 . Three inactivated vaccines (HB02, WIV04, and CoronaVac) were developed from different SARS-CoV-2 strains isolated in China 24,25 (Fig. 1).
Implications. Based on the indirect comparison method, the BNT162b2 and mRNA-1273 were associated with the highest efficacy in preventing symptomatic COVID-19. Our finding may have importance when deciding which vaccine to use, although this is not the only consideration that should be considered. Availability of the vaccines, costs, logistics, side effects, and patient acceptability, amongst others, are also factors to be considered.

Strengths and limitations.
To the best of our knowledge, this is the most comprehensive network metaanalysis to compare the efficacy of nine new COVID-19 vaccines to prevent symptomatic and severe disease in the adult population. Previously published network meta-analysis reported indirect comparisons across fore COVID-19 vaccines 28 . The results of our indirect comparison between the new vaccines showed that mRNA vaccines (BNT162b2 and mRNA-1273) were associated with a more significant decrease in the risk for symptomatic COVID-19 compared to other vaccines. We also found a trend to increased the efficacy of mRNA vaccines to prevent severe COVID-19. However, the results did not reach statistical significance because of the relatively low rate of severe disease.
However, our indirect comparison has several limitations. Firstly, our network meta-analysis includes one study for each intervention arm. In addition, the results of the two studies are not peer-reviewed, while reported data originated from press releases and reports submitted to FDA 43,44 . There are several significant differences between studies' protocols, which may be partially responsible for the differences between the vaccine efficacies. As mentioned above, Ad26.COV2.S efficacy is based on a singledose regimen, while other vaccines were administered as a two-dose regimen, including ChAdOx1 (AZD1222) vaccine, whose protocol was adapted to a two-dose regimen after the study had been started 45 . Moreover, vaccines were examined under non-equivalent conditions, including countries with unlike socio-economic conditions and various stages of COVID-19 outbreak, different seasons, and different SARS-CoV-2 variants. All mentioned above may influence vaccines' efficacy. Recently published data support that the B.1.1.17 variant, known as the UK strain, is susceptible to the immunity induced bytheBNT162b2 and mRNA-1273 vaccines 46,47 . However, the B.1.351 variant, primarily identified in South Africa, is less susceptible to mRNA-1273 vaccine-induced neutralizing antibodies 47 . It remains to determine if the reduction in antibody susceptibility will be associated with decreased vaccine effectiveness. There is also a high probability that the virus will acquire new mutations that will change its susceptibility to vaccines, and some vaccines might be influenced more than others. As a result, the efficacy of the different vaccines is expected to be affected. Besides, the current data on vaccine efficacy is based on short-term data, so we could not compare the effectiveness and immunity duration of different vaccines. Presently, it is not known which vaccine will induce longer immune responses. Also, as seen with other vaccines, booster doses may be required every few years to maintain immunity.
Secondly, our meta-analysis compares the efficacy of the studied vaccines in two hundred thousand participants of phase 3 RCTs without data from observational studies. So far, millions of people have been vaccinated around the world. One study from Clalit Health Services, a large health maintenance organization in Israel, compares the efficacy of the BNT162b2 mRNA vaccine in about 600,000 vaccinated persons to that of a similarsized group of unvaccinated controls 48 . In this study, the efficacy of the BNT162b2 mRNA vaccine was similar to that seen in the phase 3 RCT 7 .
Finally, the safety outcomes of the vaccines were beyond the aims of the current network meta-analysis. Currently, available safety results are based on short-duration follow-up, and a very low rate of severe adverse reactions has been observed in the short term. As the mRNA vaccine technology is new and it is still unclear which issues will emerge in the long term, real-world data will be needed to assess the safety of prospective vaccines.

Conclusion
In our indirect comparison, the BNT162b2 and mRNA-1273 vaccines, which use mRNA technology, were associated with the highest efficacy in preventing symptomatic COVID-19 compared to the other vaccines. The compared vaccines were not different in efficacy to prevent severe disease. We found no difference between vaccines' efficacy to prevent symptomatic COVID-19 among the elderly.