Real-world analysis of the incidence of SARS-CoV-2 infection post vaccination is important in determining the comparative effectiveness of the available vaccines. In this retrospective cohort study using deidentified administrative claims for Medicare Advantage and commercially insured individuals in a research database we examine over 3.5 million fully vaccinated individuals, including 8,848 individuals with SARS-CoV-2 infection, with a follow-up period between 14 and 151 days after their second dose. Our primary outcome was the rate of Covid-19 infection occurring at 30, 60, and 90 days at least 14 days after the second dose of either the mRNA-1273 vaccine or the BNT162b2 vaccine. Sub-analyses included the incidence of hospitalization, ICU admission, and death/hospice transfer. Separate analysis was conducted for individuals above and below age 65 and those without a prior diagnosis of Covid-19. We show that immunization with mRNA-1273, compared to BNT162b2, provides slightly more protection against SARS-CoV-2 infection that reaches statistical significance at 90 days with a number needed to vaccinate of >290. There are no differences in vaccine effectiveness for protection against hospitalization, ICU admission, or death/hospice transfer (aOR 1.23, 95% CI (0.67, 2.25)).
Both the emergence of variants of concern (VOC) and reports of infections post vaccination emphasize the need to study the relative real-world effectiveness of the available vaccines against Covid-19. The mRNA-1273 (Moderna) vaccine and the BNT162b2 (Pfizer–BioNTech) vaccine have both proven highly effective in preventing severe disease, hospitalization, and death from Covid-19. However, data suggest that the humoral antibody response to the vaccines differs, with two doses of the mRNA-1273 vaccine providing significantly higher humoral antibody response compared with two doses of the BNT162b2 vaccine in both uninfected and previously infected individuals across all age categories1. This is clinically relevant as several studies have demonstrated that higher humoral antibody responses correlate with enhanced protection against Covid-192.
Data also suggest that infection rates follow the same trend as the humoral antibody response. Observational data incorporating cases of Delta VOC have suggested a higher rate of infection in individuals immunized with two doses of the BNT162b2 vaccine compared with two doses of the mRNA-1273 vaccine, suggesting that the real-world performance of the vaccines may differ from one another3.
We therefore conducted a retrospective cohort study that examined the incidence and severity of Covid-19 infections as a function of the time from vaccination in over 3.5 million individuals across the United States who were fully vaccinated via either the mRNA-1273 or the BNT162b2 vaccine, including over 8800 infections; see Fig. 1. Understanding the clinical performance of each vaccine is critically important to determine the comparative effectiveness of the vaccines.
Our primary outcome is the rate of Covid-19 infection occurring at 30, 60, and 90 days at least 14 days after the second dose of either the mRNA-1273 vaccine or the BNT162b2 vaccine.
In a population of over 3.5 million fully vaccinated individuals, 8848 experienced documented Covid-19 infections. Of those, 3090 (35%) received mRNA-1273 and 5758 (65%) received BNT162b2 (Table 1 and Fig. 1).
Median age and prevalence of many comorbid conditions were similar across groups considered (Table 1). Models for the seven negative outcomes, included to assure there were no unaccounted-for differences between groups receiving BNT162b and mRNA-1273, exhibited negligible mean bias arising from systemic errors (Table S3).
Kaplan–Meier curves report that the estimated time to infection and time to severe negative outcomes defined as any hospitalization, ICU admission, death, or discharge to hospice, whichever occurs first, are reported in Fig. 2. In the raw data, a statistically significant difference (p-value < 0.001) with a diverging trend over time is observed between BNT162b and mRNA-1273 (Fig. 2C). This difference persists in the PH model-based predictions for a general patient (Fig. 2A). There are no statistically significant differences between the predicted probabilities of the two vaccines (Fig. 2B, D) for the composite outcome of hospitalization, ICU admission, death, or transfer to hospice.
Adjusted odds ratios for likelihood of infection 30, 60, and 90 days post vaccination (14 days after the second dose) show that the mRNA-1273 vaccine is associated with lower odds of infection. For the unadjusted model with IPTW as sampling weights, the NNV to observe this difference ranges from 1,047 over 30 days to 290 over 90 days (Fig. 3), signaling superior mRNA-1273 vaccine effects over time in reducing the likelihood of infection. This difference is consistent across all the models considered. However, for severe adverse outcomes (ICU admission, composite ICU admission/death/referral to hospice, or composite hospitalization/ICU admission/death/referral to hospice), while events are rare, no statistically significant differences between the two vaccines were observed (Fig. S4–6). As shown in Fig. 4, time-to-event analysis assuming right censoring, was performed and provided similar results to the binary outcome results for different model specifications, including for outcomes of infection (aHR 0.69, 95% CI (0.66, 0.72)), composite ICU admission/death/referral to hospice (aHR 0.76, 95% CI (0.50, 1.16)), and composite hospitalization/ICU admission/death/referral to hospice (aHR 0.67, 95% CI (0.51, 0.89)). Risk differences between mRNA-1273 and BNT162b2 for both infection and composite outcomes increase over time (Fig. 4b), this observation is in alignment with the binary outcome results (Fig. 3).
We further considered time-to-event analyses among all those experiencing infections in our sample for ICU admission, composite ICU admission/death/referral to hospice, and composite hospitalization/ICU admission/death/referral to hospice. No statistically significant differences between vaccines were observed for adverse composite outcomes (Figs. S12–14).
The results are similar for the stratified analyses of (1) including only patients with no prior diagnosis of Covid-19 (Fig. S11), (2) including only patients who are aged 65 or older (Fig. S9), and (3) including only patients who are aged 64 and younger (Fig. S10). Since the population considered is different in each model, between-model comparisons are not valid, however, both models show directionally the same results.
It is known that certain comorbidities worsen prognosis in an unvaccinated population. Our models indicate that in a vaccinated population, congestive heart failure (aHR 1.52, 95% CI (1.03, 2.26)), hypertension (aHR 2.17, 95% CI (1.30, 3.62)), and lymphoma (aHR 7.03, 95% CI (4.31,11.47)) increase the likelihood of experiencing the composite adverse event of hospitalization, ICU admission, or death/ transfer to hospice. This difference persisted even after adjusting for which vaccine a person received, sociodemographic variables, transfer from nursing facility, timing of vaccination, place of residence, and historical comorbidities (Fig. S7).
Multiple studies have shown that the incidence of SARS-CoV-2 infection in vaccinated individuals has increased over the last year. This increase may be related to the higher transmissibility of the Delta and Omicron VOC and/or the potential contribution of waning immunity post vaccination. These data underscore the need to understand the comparative effectiveness of the available vaccines.
Two important questions must be answered to address the comparative effectiveness of the BNT162b and mRNA-1273 vaccines. The first is whether each vaccine is equivalent in its ability to prevent severe disease from Covid-19. The data available from our large population of patients suggest that at 90 days out from vaccination, there is no significant difference between BNT162b and mRNA-1273 in terms of the risk of the composite outcome of hospitalization, ICU admission, or death/transfer to hospice (aOR [95% confidence interval]: 1.23 [0.67, 2.25], Fig. S6). We analyzed data both from a binary and time-to-event outcome point of view, each quantifying the association between patient outcomes and vaccine types with different model specifications. Conclusions remain similar, irrespective of modeling framework, highlighting the robustness of our methods. The second question is whether there is a higher incidence of SARS-CoV-2 infections with the BNT162b vaccine compared with the mRNA-1273 vaccine, and here there is a statistically significant difference in favor of the mRNA-1273 vaccine. This difference appears very early post vaccination and increases over time. The NNV with the mRNA-1273 vaccine compared with the BNT162b vaccine to prevent one case of SARS-CoV-2 is 1047 at thirty days post vaccination (aOR [95% CI]: 0.67 [0.63, 0.71]), but is 290 at ninety days post vaccination (aOR [95% CI]: 0.66 [0.60, 0.73]). Although this incremental risk is small at the individual level, it is meaningful at the population level. Our results suggest that for every 1 million individuals vaccinated with the BNT162b vaccine compared with the mRNA-1273 vaccine, this would represent 3,448 additional care-seeking cases of Covid-19 at 90 days.
Delta infection was of very low prevalence when these data were collected, and therefore unlikely affected our comparative effectiveness results. However, as new VOCs emerge, each of these may have unique characteristics that would impact their ability to evade vaccine-induced immunity. Periodic updates to this analysis will allow us to track the comparative effectiveness of mRNA-1273 and BNT162B in the setting of the emergence of new VOCs, including Delta and Omicron.
This study has limitations. First, the analysis is restricted to commercially insured and Medicare Advantage beneficiaries from a single U.S. insurer, a group that is unevenly distributed across the United States geographically and demographically. Nevertheless, this study reflects a large and comprehensive sample of U.S. vaccinations. Second, we are unable to measure SARS-CoV-2 infection that is not apparent in medical claims or via laboratory testing, which likely results in an overestimate of the vaccines’ protective effects. This includes the fact that our “time-to-infection” represents date of infection from our data (positive PCR test, or ICD-10 code of U07.1 in claims), rather than the date on which SARS-CoV-2 was contracted. Notably, asymptomatic or mild disease for which an individual did not seek care are not captured. However, serious adverse events, which pose the most strain on the healthcare system and for individuals, are reliably observable in our data and are less affected by diagnosis-dependent biases over time. Finally, we do not include an analysis accounting for calendar time as a proxy to infer the likely variant of infection as this additional confounder would further complicate already-nuanced results. While not undertaken here, further study to understand the relative effectiveness against all variants of concern is needed and will be conducted when our data on infection with Delta and Omicron are more complete. Basic counts are included in Table S8.
This study also has strengths. It represents a geographically and sociodemographically diverse group of 3,966,004 patients, allowing confidence in the estimation of individual-level patient factors associated with documented breakthrough SARS-CoV-2 infection and the resulting serious adverse events.
The incidence of SARS-CoV-2 infection in this large cohort of individuals post full vaccination with mRNA-1273 or BNT162b suggests that the effectiveness of the mRNA-1273 vaccine exceeds that of the BNT162b vaccine by a small margin. However, both vaccines compared equally with respect to the incidence of severe disease at 90 days defined by hospitalization, ICU admission, discharge to hospice, or death (Fig. S6; aOR 1.23, 95% CI (0.67, 2.25)).
This paper follows STROBE reporting guidelines for cohort studies. This study was deemed exempt from institutional board review by the UnitedHealth Group Office of Human Research Affairs due to using deidentified retrospective data.
We used administrative deidentified claims for Medicare Advantage and commercially insured individuals in a research database, including vaccination status through May 31, 2021. This study was reviewed and deemed exempt by the institutional review board of UnitedHealth Group.
Starting with all Medicare Advantage or commercial enrollees 18 years or older vaccinated after emergency-use authorization (EUA) for either mRNA-1273 or BNT162b2 who were fully vaccinated (received their second dose with 14 days of additional observation) on or before May 31, 2021. Those who received a second dose earlier than EUA-approved timeframe (21 days for BNT162b2 and 28 days for mRNA-1273) or more than 42 days after their first dose were excluded from analyses. We then required valid zip-code information and excluded any patients who experienced a Covid-related event (ICD-10 code U07.1 in inpatient or outpatient setting or positive PCR laboratory test for SARS-CoV-2) within 13 days of their second dose and none after “full vaccination,” which is considered 14 days after the second dose, to eliminate those whose infections occurred before they were fully vaccinated (Fig. 1). For subsequent analyses, we subdivided this group into those who had at least 30, 60, and 90 days of follow-up observation. In the supplementary information, we also look in particular at the subset of 8848 fully vaccinated individuals who became infected with Covid-19 (identified either by a positive PCR test or ICD-10 code U07.1, including ER, outpatient, and inpatient hospital visits).
Our main outcome was SARS-CoV-2 infection identified either by a positive PCR test or Covid-19 ICD-10 code U07.1 in claims, including ER, outpatient, and inpatient hospital visits. We define this as a “care-seeking” population. We also considered ICU admission, hospital admission, and the composite of either inpatient mortality or discharge to hospice within 30, 60, and 90 days of initial admission for Covid-19. We considered this composite measure a more complete representation of the outcome of interest than mortality alone as it reflects an outcome closer to any-site mortality, and given known racial differences in hospice use4.
Binary outcomes such as SARS-CoV-2 infection, hospital admission, composite outcomes such as ICU, death, or hospice transfer with and without hospitalization that occurred within 30-, 60-, and 90 days post vaccination were considered. A series of multivariable logistic regressions4 were performed to estimate the odds of experiencing events for those vaccinated with mRNA-1273 compared with BNT162b2 adjusting for risk factors (demographic, nursing facility admission source, socioeconomic status (SES) index, comorbidities, time of vaccination, residence by state, prior Covid diagnosis, and urban/rural location). To mitigate any potential selection bias, sample propensity scores, signaling conditional probability of receiving mRNA-1273 and BNT162b2, were estimated via multivariable generalized linear model with logit link function and inverse probability of treatment weights (IPTWs) were calculated using stabilization5,6. Both weighted univariate and multivariable logistic regressions7,8,9,10,11,12 were performed to validate the results of unweighted samples. As a sensitivity, propensity-score subclassification models were fitted with respect to quintiles of the estimated propensity scores13,14.
Kaplan–Meier analysis was performed for time-to-infection and time-to-composite outcomes for individuals fully vaccinated by June 1, 2021; log-rank-based p-values were provided. Multivariable Cox proportional hazard (PH) models15 were fitted with and without IPTWs. We performed three-sensitivity analyses by (a) stratifying data for vaccinated individuals aged 65 and above, (b) stratifying data for vaccinated individuals under age 65, and (c) excluding patients with a history of any prior Covid-19 diagnoses before complete vaccination. PH model-based predictions of experiencing events for a mRNA-1273- and a BNT162b2-vaccinated individual were provided.
Weighted standardized mean differences and statistical significance tests based on weighted regressions16 were used to check balances within measured covariates. Standard errors of the parameters were estimated by robust sandwich covariance estimators for all models. Mean predicted marginal probabilities (risk) were calculated via “recycled” predictions17,18,19, assuming everyone in the sample received each vaccine, then multiplied by 1000 to convert to predicted marginal risk per 1000 individuals, and number needed to vaccinate (NNV) was obtained with respect to absolute risk differences based on each model19,20.
As sensitivity, we reported “E-values”21 to assess the magnitude of unmeasured confounding. Summary measures22 to determine the strength of selection bias were provided for main outcomes of interest. Multivariable generalized linear models with respect to seven negative-control outcomes were fitted and empirical null distribution of systematic error was used to quantify the mean of bias23,24. All statistical tests were two-sided, with a significance level of 5%. All analyses were conducted using R, version 3.6.325. Technical details are included in the Supplementary Information.
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
The raw data are protected and are not available due to data-privacy laws. The processed data are available at reasonable request to the corresponding author.
Steensels, D., Pierlet, N., Penders, J., Mesotten, D. & Heylen, L. Comparison of SARS-CoV-2 antibody response following vaccination with BNT162b2 and mRNA-1273. JAMA 326, 1533–1535 (2021).
Khoury, D. S. et al. Neutralizing antibody levels are highly predictive of immune protection from symptomatic SARS-CoV-2 infection. Nat. Med. 27, 1205–1211 (2021).
Puranik A., et al. Comparison of two highly-effective mRNA vaccines for Covid-19-19 during periods of Alpha and Delta variant prevalence. medRxiv https://doi.org/10.1101/2021.08.06.21261707 (2021).
Cohen, L. L. Racial/Ethnic disparities in hospice care: a systematic review. J. Palliat. Med. 11, 763–768 (2008).
Lee, B. K., Lessler, J., Stuart, E. A. Weight trimming and propensity score weighting. PLoS ONE. 6, e18174 (2011).
Cole, S. R. & Hernan, M. A. Constructing inverse probability weights for marginal structural models. Am. J. Epidemiol. 168, 656–664 (2008).
Imbens, G. W. & Wooldridge, J. M. Recent developments in the econometrics of program evaluation. J. Econ. Lit. 47, 5–86 (2009).
Guo, S., Fraser, M. & Chen, Q. Propensity score analysis: recent debate and discussion. J. Soc. Soc. Work Res. 11, 463–482 (2020).
DuGoff, E. H., Schuler, M. & Stuart, E. A. Generalizing observational study results: applying propensity score methods to complex surveys. Health Serv. Res. 49, 284–303 (2014).
McCaffrey, D. F., Ridgeway, G. & Morral, A. R. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol. Methods 9, 403–425 (2004).
Austin, P. C. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar. Behav. Res. 46, 399–424 (2011).
Joffe, M. M. & Rosenbaum, P. R. Invited commentary: propensity scores. Am. J. Epidemiol. 150, 327–333 (1999).
Rubin, D. B. Matched Sampling for Causal Effects (Cambridge University Press, 2006).
Guo, S. & Fraser, M. W. Propensity Score Analysis: Statistical Methods and Applications 2nd ed., Vol. 11 (SAGE publications, 2015).
Cox, D. R. & Oakes, D. Analysis of Survival Data 1st ed. (Chapman and Hall/CRC, 2018).
Austin, P. C. & Stuart, E. A. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat. Med. 34, 3661–3679 (2015).
Muller, C. J. & MacLehose, R. F. Estimating predicted probabilities from logistic regression: different methods correspond to different target populations. Int. J. Epidemiol. 43, 962–970 (2014).
Stoltz, D. S. & Williams, R. A. In SAGE Research Methods Foundations (SAGE Publications Ltd, 2020).
Zhang, Z. et al. Estimate risk difference and number needed to treat in survival analysis. Ann. Transl. Med. 6, 120 (2018).
Cook, R. J. & Sackett, D. L. The number needed to treat: a clinically useful measure of treatment effect. BMJ 310, 452–454 (1995).
VanderWeele, T. J. & Ding, P. Sensitivity analysis in observational research: introducing the E-value. Ann. Intern. Med. 167, 268 (2017).
Smith, L. H. & VanderWeele, T. J. Bounding bias due to selection. Epidemiology 30, 509–516 (2019).
Schuemie, M. J., Ryan, P. B., DuMouchel, W., Suchard, M. A. & Madigan, D. Interpreting observational studies: why empirical calibration is needed to correct p‐values. Stat. Med. 33, 209–218 (2014).
Gruber, S. & Tchetgen Tchetgen, E. Limitations of empirical calibration of p-values using observational data: Limitations of p-value calibration. Stat. Med. 35, 3869–3882 (2016).
R Core Team. R: A Language and Environment for Statistical Computing, http://www.R-project.org/ (R Foundation for Statistical Computing, 2013).
The authors wish to thank Alyssa Wong for her contributions to helping us identify negative outcomes.
The authors declare no competing interests.
Peer review information
Nature Communications thanks the anonymous reviewers for their contribution to the peer review of this work. Peer reviewer reports are available.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Islam, N., Sheils, N.E., Jarvis, M.S. et al. Comparative effectiveness over time of the mRNA-1273 (Moderna) vaccine and the BNT162b2 (Pfizer-BioNTech) vaccine. Nat Commun 13, 2377 (2022). https://doi.org/10.1038/s41467-022-30059-3
This article is cited by
Predictors of long-term neutralizing antibody titers following COVID-19 vaccination by three vaccine types: the BOOST study
Scientific Reports (2023)
Intranasal COVID-19 vaccine induces respiratory memory T cells and protects K18-hACE mice against SARS-CoV-2 infection
npj Vaccines (2023)
Rheumatology International (2023)
Environmental Science and Pollution Research (2022)