Severe adverse events (AEs) after COVID-19 vaccination are not well studied in randomized controlled trials (RCTs) due to rarity and short follow-up. To monitor the safety of COVID-19 vaccines (“Pfizer” vaccine dose 1 and 2, “Moderna” vaccine dose 1 and 2, and “Janssen” vaccine single dose) in the U.S., especially regarding severe AEs, we compare the relative rankings of these vaccines using both RCT and the Vaccine Adverse Event Reporting System (VAERS) data. The risks of local and systemic AEs were assessed from the three pivotal COVID-19 vaccine trials and also calculated in the VAERS cohort consisting of 559,717 reports between December 14, 2020 and September 17, 2021. AE rankings of the five vaccine groups calculated separately by RCT and VAERS were consistent, especially for systemic AEs. For severe AEs reported in VAERS, the reported risks of thrombosis and GBS after Janssen vaccine were highest. The reported risk of shingles after the first dose of Moderna vaccine was highest, followed by the second dose of the Moderna vaccine. The reported risk of myocarditis was higher after the second dose of Pfizer and Moderna vaccines. The reported risk of anaphylaxis was higher after the first dose of Pfizer vaccine. Limitations of this study are the inherent biases of the spontaneous reporting system data, and only including three pivotal RCTs and no comparison with other active vaccine safety surveillance systems.
The COVID-19 vaccines (BNT162b2 mRNA vaccine, or “Pfizer”, mRNA-1273 vaccine, or “Moderna”, and Ad26.COV2.S vaccine, or “Janssen”) are critical components of the response to the pandemic globally. The efficacy and safety of the three COVID-19 vaccines administered in the US were reported in a series of randomized controlled trials (RCTs)1,2,3,4. These RCTs reported risks of mild and common local and systemic adverse events (AEs). For some other severe but rare AEs, risks were not explicitly estimated in the RCTs, likely due to few cases or short follow-up. However, the consequence of these AEs may be more substantial. For example, complications such as thrombosis5 or Guillain–Barré syndrome (GBS) may occur after vaccination6,7. Despite their severity, the incidence of these rare AEs are difficult to estimate in RCTs.
Post-marketing surveillance of COVID-19 vaccines offers larger sample sizes from which to estimate rare AEs, but may create bias because the data are collected less systematically than in RCTs. For example, the Vaccine Adverse Event Reporting System (VAERS), maintained by the Centers for Disease Control and Prevention (CDC) and FDA, provides real-world post-marketing evidence of vaccine safety and has been used to monitor safety of vaccines, including COVID-19 vaccines5,8,9,10. But because it organically accepts reports from healthcare professionals as well as any member of the population, it is unclear whether VAERS reports can replace or anticipate results from more systematic assessments, or what biases may be present.
In this paper, we compare the relative rankings of three COVID-19 vaccines available in the U.S. using both RCT and VAERS data to assess how consistently these data sources perform. We first analyzed common local (i.e. pain, erythema and swelling) and systemic (i.e. headache, fever, chill, fatigue, nausea/vomiting, arthralgia and myalgia) AE signals after COVID-19 vaccines from VAERS data and then compared the results with RCT data. We then “extrapolated” the consistency and further ranked the COVID-19 vaccines regarding rare but severe AEs, including shingles, hearing impairment, thrombosis, facial paralysis, anaphylaxis, pulmonary embolism, myocarditis, and GBS. Though these AEs have drawn more public attention and may result in more severe outcomes, they were not explicitly evaluated in the RCTs due to the rarity of events and/or short follow-up.
We extracted 569,072 reports submitted to VAERS related to COVID-19 vaccines from December 14, 2020 to September 17, 2021. After excluding 1952 entries reporting co-administration with other vaccines, 1198 reports with an unknown manufacturer, and 6297 reports without associated symptoms; 559,717 reports were used for analysis, including 237,735 for the Pfizer, 270,754 for the Moderna, and 51,228 for the Janssen vaccines. The characteristics of these VAERS reports are presented in Table 1.
VAERS reports were more common after administration of the Janssen vaccine (3.40 per 1000 doses) than Pfizer or Moderna vaccines (1.06 or 1.79 per 1000 doses, respectively). Fewer reports were related to the elderly population (60 +) after Janssen vaccine (17.4%), compared to Moderna (35.4%) and Pfizer (25.0%). Females were much more likely to submit VAERS reports than males, irrespective of vaccine type. The AEs reported were slightly more likely to occur on the same day as vaccination for the Pfizer vaccine (41.9%, compared to 36.7% and 36.6% for Moderna and Janssen respectively). More reports were related to the first dose than the second dose for Pfizer (50.6% vs 34.7%) and Moderna (60.2% vs 26.3%) vaccines.
The main results are presented in Table 2, which contains the risks of AEs after COVID-19 vaccination as reported in either RCTs or VAERS, and the rank of vaccines and their consistency. The risks of AEs after COVID-19 vaccination as reported in VAERS are also shown in Fig. 1. As expected, the AE risks from systematically observed RCTs (the first rows) are substantially different than the AE risks spontaneously reported in the VAERS database (the second rows). However, the rank order of vaccine AEs are highly consistent between RCTs and VAERS, especially for systemic AEs. Specifically, for Pfizer and Moderna vaccines, the reporting risks of systemic AEs after the first dose are always lower than those after the second dose in either RCTs or VAERS. Results based on age groups are presented in Tables S1 and S2 in the Supplementary Materials. In general, the AE risks for elderly population is slightly lower than young population, while the rank of vaccines remains to be consistent for each of the two age groups.
Table 2 also reports the ranks of vaccines for other rare but severe AEs available in VAERS but not in RCT data. To this end, VAERS provided large-scale post-marketing evidence for comparing the three vaccines. For example, the risks of thrombosis and GBS after Janssen vaccine are the highest. Concerns about excessive risk of thrombosis after the Janssen vaccine resulted in a 10-day suspension (April 13 to April 23) of its administration in April 2021. The excessive reporting risk of GBS after Janssen vaccine was also reported by several studies11,12. For shingles, the VAERS data suggested that the 1st shot of Moderna vaccine was associated with the highest reporting risk, followed by the 2nd shot of Moderna vaccine. For myocarditis, higher reporting risks was observed for the second dose of Pfizer and Moderna vaccines. There are some post-marketing studies on myocarditis after COVID-19 vaccination13,14 but no comparison between vaccines exists. For anaphylaxis higher reporting risks were observed for the first dose of Pfizer vaccine. One study8 showed that the incidence of anaphylaxis associated with the Pfizer vaccine is higher than other non-COVID vaccines, but no comparison among COVID-19 vaccines exist. There were no substantial differences among vaccines regarding risks of other AEs (Fig. 1). We emphasize that the rank consistency for local and systemic AEs adds credibility to the relative safety among multiple vaccines for the rare but severe AEs, and this is a unique contribution of our study.
In sensitivity analyses examining temporal changes on reporting risk we found decreasing risks of mild and common AEs such as pain and fever (Fig. S1a, b), and increasing risks of more severe and rare AEs such as thrombosis and GBS (Fig. S1c, d). The ranking of vaccines remained stable, and the correlations show consistency (i.e. correlation > 0) of the vaccines’ ranks between RCTs and VAERS, by using the VAERS reports as early as the first 2 months after administration. For more visualization see https://chongliang-luo.shinyapps.io/covid_vaers/, where the temporal changes on reporting risk are shown with user-specified age group, sex and AEs.
We analyzed COVID-19 vaccine safety data from published RCTs and the VAERS database. RCTs and VAERS rank COVID-19 vaccines similarly by risk of AE, supporting the validity of VAERS for assessing relative AE risk in this context. To the extent we can extrapolate from AEs reported in both data sources to AEs reported only in VAERS, we can use VAERS to estimate the relative safety of the vaccines against serious side effects too rare to be measured effectively in RCTs, and in real time.
Our investigation has several strengths. First, our study investigated the evidence consistency by comparing safety signals across multi-source data. The consistency in evidence across datasets is an important component of evidence quality, which cannot be obtained using a single dataset. Second, severe AEs are better monitored by post-marketing data than RCTs. Our analysis using VAERS provides valuable evidence of the relative safety of the COVID-19 vaccines regarding severe AEs, and circumvents the reporting bias of the spontaneous reporting system.
Our study has several limitations. First, our analysis focuses on the comparison across vaccines, and evidence from RCTs and VAERS, but does not provide information on absolute incidences of AEs among the general population. Also, the risk calculations in RCTs and VAERS do not take into account background risks in the reporting populations. Second, as a spontaneous reporting system, reporting into VAERS varies over time and is subject to shifts in the population undergoing vaccination and to publicity stimulation, hence the inherent limitations/biases when using VAERS data for vaccine safety evaluation. For example, before vaccines were available to the general public, they were administered according to scheduled phases depending on age, occupation, and living situation; AEs and reporting practices may vary accordingly. These novel vaccines drew extraordinary public attention, especially when first administered, which may also affect reporting practices. Third, our study only includes three pivotal RCTs1,2,3,4 for the three vaccines but more trials may be included in the future. Furthermore, it lacks comparison with other safety systems such as the Biologics Effectiveness and Safety (BEST15), the Safety Platform for Emergency Vaccines (SPEAC16), the Clinical Immunization Safety Assessment (CISA17) project and the Vaccine Safety Datalink (VSD18). The current investigation is less robust unless more trials and safety surveillance databases are included in the comparison.
The validity of this study is largely dependent on the assumption that the reporting behavior to VAERS (i.e. reporting sensitivity19) is similar across different type of vaccines. This assumption is likely to be the case for Pfizer and Moderna vaccines, as they were authorized and became available at similar times. However, the Janssen vaccine was authorized about 3 months later and could have drawn less public attention. It is thus possible that people with only mild AEs after the Janssen vaccine may have chosen not to report to VAERS and as a result, the risks of more severe, systemic AEs is higher than those after Pfizer and Moderna vaccines. This could explain the inconsistency of the rank of Janssen vaccine between RCT and VAERS regarding systemic AEs in Table 2. We have previously seen such an example in our evaluation of GBS as a potential post-vaccination effect after influenza vaccines6, where people who reported GBS tend not to report mild AEs, such as pain, erythema and swelling etc.
Post-market surveillance is essential for continuously evaluating and monitoring vaccine safety. In addition to VAERS, other active surveillance databases such as BEST15, SPEAC16, CISA17 and VSD18 have been used for evaluating the safety of COVID-19 vaccines7,20,21,22. In general, data collected by surveys23,24 or social media platforms25,26 could also provide evidences for vaccine safety or hesitancy. Despite the limitation of being a passive reporting system, VAERS, with proper statistical analyses, still provides valuable information about vaccine safety, especially in the early phase of general population administration and vaccine surveillance. For example, in the first several months (Dec 2020–April 2021) of the Pfizer and Moderna COVID-19 vaccine administration, VAERS-based studies5,8,9,10 emerged to generate safety signals for anaphylaxis, sudden hearing loss and cerebral venous sinus thrombosis. These studies contribute unique time-sensitive real-world evidence on safety signals that were examined rigorously by later randomized studies. Our study provides a novel method utilizing the publicly available RCT and VAERS data for monitoring the safety of COVID-19 vaccines. It is reassuring that our analyses based on the VAERS data produced evidence consistent with existing RCTs, and demonstrated the ability to identify rare AEs that were not recognized in the RCTs. Our analysis can be applied to the post-market surveillance of ongoing (e.g. COVID-19 vaccine boost dose and adolescent dose) and future vaccine administration using VAERS, especially for the severe AEs. The website https://chongliang-luo.shinyapps.io/covid_vaers/ will be continuously updated for timely monitoring the COVID-19 vaccines in the future.
We collected vaccine safety data from published RCTs for the three vaccines authorized in the U.S. For all RCTs1,2,3,4, local AEs (i.e. pain, erythema and swelling) and systemic AEs (i.e. headache, fever, chill, fatigue, nausea/vomiting, arthralgia/joint pain, and myalgia/muscle pain) were assessed within seven days after vaccination.
We also used data from VAERS27,28,29 to seek post-marketing evidence on the safety of the three vaccines. The VAERS data is publicly available and no institutional approval is required. The analyses were performed in accordance with VAERS guidelines and regulations27. We extracted AE reports after COVID-19 vaccination up to September 17, 2021. Detailed definitions of the AEs in MedDRA preferred term (PT) level are listed in Table S1. We grouped some of the similar PT terms to their Standardized MedDRA Queries (SMQs)30. Reports with co-administration with other vaccines, or unknown manufacturer of COVID-19 vaccine, or invalid symptoms (e.g. “no adverse events”) were excluded. To be consistent with the RCTs, we used VAERS reports with age 18 + (age 16 + for the Pfizer vaccine), and onset time within 7 days after vaccination for the solicited AEs (i.e. local and systemic AEs). For the unsolicited AEs (i.e. other rare but severe AEs including shingles, hearing impairment including tinnitus, thrombosis, facial paralysis including Bell’s palsy, anaphylaxis, pulmonary embolism, myocarditis, and GBS), we used VAERS reports with onset time within 42 days after vaccination. Analysis was also repeated for age groups, i.e. young and elderly populations. We used the age cutoffs from the original trials (i.e. 55 for Pfizer trial, 65 for Moderna trial and 60 for Janssen trial) for the RCTs, and 60 for VAERS.
The vaccines and doses were grouped in 5 categories, i.e. Pfizer-1st (P1), Pfizer-2nd (P2), Moderna-1st (M1), Moderna-2nd (M2) and Janssen (J). Figure 2 illustrates the overall rationale of our statistical analysis plan using data from RCTs and VAERS. We calculated AE risks for each vaccine type and dose in VAERS. Since the AEs are self-reported in the VAERS database, AE risks are not directly comparable to event risks in the RCTs. However, a unique opportunity in our investigation of VAERS data on COVID-19 vaccines is that the Pfizer and Moderna vaccines were released around the same time and both have received similar public attention. Thus, it is reasonable to assume that the reporting behavior of AEs is not strongly dependent on the type of vaccine received by the vaccinee. Under this assumption, despite the VAERS data being subject to reporting bias, a similar reporting risk applies to both vaccinees, hence the ranks of AE reporting risks across vaccines calculated from VAERS are meaningful statistics, which can be cross-compared with the ranks of AE risks across vaccines calculated from RCTs.
Given the above rationale, we compared AE ranking by vaccine in both the VAERS and RCT databases, and evaluated their consistency. The rank consistency was measured by the Spearman’s rank correlation coefficient \(\rho \)31. Spearman’s \(\rho \) ranges from − 1 to 1 and a positive value indicates consistency of the two ranks. This rank correlation measure is commonly used to assess agreement between two rating approaches.
Finally, based on the consistency between RCTs and VAERS, we further ranked the vaccines regarding the risks of other rare but severe AEs (i.e. shingles, hearing impairment including tinnitus, thrombosis, facial paralysis including Bell’s palsy, anaphylaxis, pulmonary embolism, myocarditis, and GBS) in VAERS.
Polack, F. P. et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N. Engl. J. Med. 383, 2603–2615 (2020).
Baden, L. R. et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N. Engl. J. Med. 384(5), 403–416 (2021).
Sadoff, J. et al. Safety and efficacy of single-dose Ad26.COV2.S vaccine against Covid-19. N. Engl. J. Med. 384(23), 2187–2201 (2021).
Thomas, S. J. et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine through 6 months. N. Engl. J. Med. 385(19), 1761–1773 (2021).
See, I. et al. US case reports of cerebral venous sinus thrombosis with thrombocytopenia after Ad26.COV2.S vaccination, March 2 to April 21, 2021. JAMA 325, 2448–2456 (2021).
Luo, C. et al. Prediction of post-vaccination Guillain–Barré syndrome using data from a passive surveillance system. Pharmacoepidemiol. Drug Saf. 30(5), 602–609 (2021).
Hanson, K. E. et al. Incidence of Guillain–Barré syndrome after COVID-19 vaccination in the vaccine safety datalink. JAMA Netw. Open 5(4), e228879 (2022).
Shimabukuro, T. & Nair, N. Allergic reactions including anaphylaxis after receipt of the first dose of Pfizer-BioNTech COVID-19 vaccine. JAMA 325(8), 780–781 (2021).
Shimabukuro, T. T., Cole, M. & Su, J. R. Reports of anaphylaxis after receipt of mRNA COVID-19 vaccines in the US—December 14, 2020–January 18, 2021. JAMA 325(11), 1101–1102 (2021).
Shimabukuro, T. T. et al. Preliminary findings of mRNA Covid-19 vaccine safety in pregnant persons. N. Engl. J. Med. 384, 2273–2282 (2021).
Woo, E. J. et al. Association of receipt of the Ad26. COV2. S COVID-19 vaccine with presumptive Guillain–Barré syndrome, February–July 2021. JAMA 326(16), 1606–1613 (2021).
Alimchandani, M. Guillain–Barré syndrome (GBS) after Janssen COVID-19 vaccine: Vaccine Adverse Event Reporting System (VAERS) (2021).
Witberg, G. et al. Myocarditis after Covid-19 vaccination in a large health care organization. N. Engl. J. Med. 385, 2132–2139 (2021).
Diaz, G. A. et al. Myocarditis and pericarditis after vaccination for COVID-19. JAMA 326(12), 1210–1212 (2021).
The Biologics Effectiveness and Safety (BEST) Initiative. https://bestinitiative.org/. Accessed May 11, 2022.
The Safety Platform for Emergency vACcines (SPEAC). https://brightoncollaboration.us/speac/
Centers for Disease Control and Prevention. The Clinical Immunization Safety Assessment (CISA) Project. https://www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/cisa/index.html
Centers for Disease Control and Prevention. The Vaccine Safety Datalink (VSD). https://www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/vsd/index.html
Miller, E. R., McNeil, M. M., Moro, P. L., Duffy, J. & Su, J. R. The reporting sensitivity of the Vaccine Adverse Event Reporting System (VAERS) for anaphylaxis and for Guillain–Barre syndrome. Vaccine 38(47), 7458–7463 (2020).
Klein, N. P. et al. Surveillance for adverse events after COVID-19 mRNA vaccination. JAMA 326(14), 1390–1399 (2021).
Blumenthal, K. G., Phadke, N. A. & Bates, D. W. Safety surveillance of COVID-19 mRNA vaccines through the vaccine safety datalink. JAMA 326(14), 1375–1377 (2021).
Groom, H. C. et al. Monitoring vaccine safety using the vaccine safety Datalink: Assessing capacity to integrate data from Immunization Information systems. Vaccine. 40(5), 752–756 (2022).
Lee, K. L. M. N. et al. Investigating trends in those who experience menstrual bleeding changes after SARS-CoV-2 vaccination. MedRXiv. https://doi.org/10.1101/2021.10.11.21264863 (2022).
Boekel, L. et al. Adverse events after first COVID-19 vaccination in patients with autoimmune diseases. Lancet Rheumatol. 3(8), e542–e545 (2021).
Du, J. et al. Use of deep learning to analyze social media discussions about the human papillomavirus vaccine. JAMA Netw. Open 3(11), e2022025–e2022025 (2020).
Du, J. et al. Leveraging deep learning to understand health beliefs about the human papillomavirus vaccine from social media. NPJ Digit. Med. 2(1), 1–4 (2019).
Centers for Disease Control and Prevention. VAERS: Vaccine Adverse Event Reporting System.
Chen, R. T. et al. The vaccine adverse event reporting system (VAERS). Vaccine 12(6), 542–550 (1994).
Shimabukuro, T. T., Nguyen, M., Martin, D. & DeStefano, F. Safety monitoring in the vaccine adverse event reporting system (VAERS). Vaccine 33(36), 4398–4405 (2015).
Standardized MedDRA Queries (SMQs). https://www.meddra.org/standardised-meddra-queries
Sedgwick, P. Spearman’s rank correlation coefficient. BMJ 349, g7327 (2014).
We thank Dr. Anne M. Butler for suggestions on writing and visualization.
C.L., J.D., C.T. and Y.C. were supported in part by National Institutes of Health (1R01AI130460). Y.C. was supported in part by National Institutes of Health (1R01AG073435, 1R56AG074604, 1R01LM013519, 1R56AG069880).
Dr. Cuker has served as a consultant for Synergy; has received authorship royalties from UpToDate; and his institution has received research support on his behalf from Alexion, Bayer, Novartis, Novo Nordisk, Pfizer, Sanofi, Spark, and Takeda. Dr. Asch is partner and part owner of VAL Health. Dr. Poland is the chair of a Safety Evaluation Committee for novel investigational vaccine trials being conducted by Merck Research Laboratories. Dr. Poland provides consultative advice on vaccine development to Merck & Co., Medicago, GlaxoSmithKline, Sanofi Pasteur, Johnson & Johnson/Janssen Global Services LLC, Emergent Biosolutions, Dynavax, Genentech, Eli Lilly and Company, Kentucky Bioprocessing, Bavarian Nordic, AstraZeneca, Exelixis, Regeneron, Janssen, Vyriad, Moderna, and Genevant Sciences, Inc. These activities have been reviewed by the Mayo Clinic Conflict of Interest Review Board and are conducted in compliance with Mayo Clinic Conflict of Interest policies. Dr. Poland holds patents related to vaccinia, influenza, and measles peptide vaccines. Dr. Poland has received grant funding from ICW Ventures for preclinical studies on a peptide-based COVID-19 vaccine. This research has been reviewed by the Mayo Clinic Conflict of Interest Review Board and was conducted in compliance with Mayo Clinic Conflict of Interest policies. The rest of the authors report no conflicts of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Luo, C., Du, J., Cuker, A. et al. Comparability of clinical trials and spontaneous reporting data regarding COVID-19 vaccine safety. Sci Rep 12, 10946 (2022). https://doi.org/10.1038/s41598-022-13809-7