Comparability of clinical trials and spontaneous reporting data regarding COVID-19 vaccine safety

Severe adverse events (AEs) after COVID-19 vaccination are not well studied in randomized controlled trials (RCTs) due to rarity and short follow-up. To monitor the safety of COVID-19 vaccines (“Pfizer” vaccine dose 1 and 2, “Moderna” vaccine dose 1 and 2, and “Janssen” vaccine single dose) in the U.S., especially regarding severe AEs, we compare the relative rankings of these vaccines using both RCT and the Vaccine Adverse Event Reporting System (VAERS) data. The risks of local and systemic AEs were assessed from the three pivotal COVID-19 vaccine trials and also calculated in the VAERS cohort consisting of 559,717 reports between December 14, 2020 and September 17, 2021. AE rankings of the five vaccine groups calculated separately by RCT and VAERS were consistent, especially for systemic AEs. For severe AEs reported in VAERS, the reported risks of thrombosis and GBS after Janssen vaccine were highest. The reported risk of shingles after the first dose of Moderna vaccine was highest, followed by the second dose of the Moderna vaccine. The reported risk of myocarditis was higher after the second dose of Pfizer and Moderna vaccines. The reported risk of anaphylaxis was higher after the first dose of Pfizer vaccine. Limitations of this study are the inherent biases of the spontaneous reporting system data, and only including three pivotal RCTs and no comparison with other active vaccine safety surveillance systems.


Results
We extracted 569,072 reports submitted to VAERS related to COVID-19 vaccines from December 14, 2020 to September 17, 2021. After excluding 1952 entries reporting co-administration with other vaccines, 1198 reports with an unknown manufacturer, and 6297 reports without associated symptoms; 559,717 reports were used for analysis, including 237,735 for the Pfizer, 270,754 for the Moderna, and 51,228 for the Janssen vaccines. The characteristics of these VAERS reports are presented in Table 1.
VAERS reports were more common after administration of the Janssen vaccine (3.40 per 1000 doses) than Pfizer or Moderna vaccines (1.06 or 1.79 per 1000 doses, respectively). Fewer reports were related to the elderly population (60 +) after Janssen vaccine (17.4%), compared to Moderna (35.4%) and Pfizer (25.0%). Females were much more likely to submit VAERS reports than males, irrespective of vaccine type. The AEs reported were slightly more likely to occur on the same day as vaccination for the Pfizer vaccine (41.9%, compared to 36.7% and 36.6% for Moderna and Janssen respectively). More reports were related to the first dose than the second dose for Pfizer (50.6% vs 34.7%) and Moderna (60.2% vs 26.3%) vaccines.
The main results are presented in Table 2, which contains the risks of AEs after COVID-19 vaccination as reported in either RCTs or VAERS, and the rank of vaccines and their consistency. The risks of AEs after COVID-19 vaccination as reported in VAERS are also shown in Fig. 1. As expected, the AE risks from systematically observed RCTs (the first rows) are substantially different than the AE risks spontaneously reported in the VAERS database (the second rows). However, the rank order of vaccine AEs are highly consistent between RCTs and VAERS, especially for systemic AEs. Specifically, for Pfizer and Moderna vaccines, the reporting risks of systemic AEs after the first dose are always lower than those after the second dose in either RCTs or VAERS. Results based on age groups are presented in Tables S1 and S2 in the Supplementary Materials. In general, the AE risks for elderly population is slightly lower than young population, while the rank of vaccines remains to be consistent for each of the two age groups. Table 2 also reports the ranks of vaccines for other rare but severe AEs available in VAERS but not in RCT data. To this end, VAERS provided large-scale post-marketing evidence for comparing the three vaccines. For www.nature.com/scientificreports/ example, the risks of thrombosis and GBS after Janssen vaccine are the highest. Concerns about excessive risk of thrombosis after the Janssen vaccine resulted in a 10-day suspension (April 13 to April 23) of its administration in April 2021. The excessive reporting risk of GBS after Janssen vaccine was also reported by several studies 11,12 . For shingles, the VAERS data suggested that the 1st shot of Moderna vaccine was associated with the highest reporting risk, followed by the 2nd shot of Moderna vaccine. For myocarditis, higher reporting risks was observed for the second dose of Pfizer and Moderna vaccines. There are some post-marketing studies on myocarditis after COVID-19 vaccination 13,14 but no comparison between vaccines exists. For anaphylaxis higher reporting risks were observed for the first dose of Pfizer vaccine. One study 8 showed that the incidence of anaphylaxis associated with the Pfizer vaccine is higher than other non-COVID vaccines, but no comparison among COVID-19 vaccines exist. There were no substantial differences among vaccines regarding risks of other AEs (Fig. 1). We emphasize that the rank consistency for local and systemic AEs adds credibility to the relative safety among multiple vaccines for the rare but severe AEs, and this is a unique contribution of our study. In sensitivity analyses examining temporal changes on reporting risk we found decreasing risks of mild and common AEs such as pain and fever (Fig. S1a, b), and increasing risks of more severe and rare AEs such as thrombosis and GBS (Fig. S1c, d). The ranking of vaccines remained stable, and the correlations show consistency (i.e. correlation > 0) of the vaccines' ranks between RCTs and VAERS, by using the VAERS reports as early  www.nature.com/scientificreports/ Our investigation has several strengths. First, our study investigated the evidence consistency by comparing safety signals across multi-source data. The consistency in evidence across datasets is an important component of evidence quality, which cannot be obtained using a single dataset. Second, severe AEs are better monitored by post-marketing data than RCTs. Our analysis using VAERS provides valuable evidence of the relative safety of the COVID-19 vaccines regarding severe AEs, and circumvents the reporting bias of the spontaneous reporting system.
Our study has several limitations. First, our analysis focuses on the comparison across vaccines, and evidence from RCTs and VAERS, but does not provide information on absolute incidences of AEs among the general population. Also, the risk calculations in RCTs and VAERS do not take into account background risks in the reporting populations. Second, as a spontaneous reporting system, reporting into VAERS varies over time and is subject to shifts in the population undergoing vaccination and to publicity stimulation, hence the inherent limitations/ biases when using VAERS data for vaccine safety evaluation. For example, before vaccines were available to the general public, they were administered according to scheduled phases depending on age, occupation, and living situation; AEs and reporting practices may vary accordingly. These novel vaccines drew extraordinary public attention, especially when first administered, which may also affect reporting practices. Third, our study only includes three pivotal RCTs 1-4 for the three vaccines but more trials may be included in the future. Furthermore, it lacks comparison with other safety systems such as the Biologics Effectiveness and Safety (BEST 15 ), the Safety Platform for Emergency Vaccines (SPEAC 16 ), the Clinical Immunization Safety Assessment (CISA 17 ) project and the Vaccine Safety Datalink (VSD 18 ). The current investigation is less robust unless more trials and safety surveillance databases are included in the comparison.
The validity of this study is largely dependent on the assumption that the reporting behavior to VAERS (i.e. reporting sensitivity 19 ) is similar across different type of vaccines. This assumption is likely to be the case for Pfizer and Moderna vaccines, as they were authorized and became available at similar times. However, the Janssen vaccine was authorized about 3 months later and could have drawn less public attention. It is thus possible that people with only mild AEs after the Janssen vaccine may have chosen not to report to VAERS and as a result, the risks of more severe, systemic AEs is higher than those after Pfizer and Moderna vaccines. This could explain the inconsistency of the rank of Janssen vaccine between RCT and VAERS regarding systemic AEs in Table 2. We have previously seen such an example in our evaluation of GBS as a potential post-vaccination effect after influenza vaccines 6 , where people who reported GBS tend not to report mild AEs, such as pain, erythema and swelling etc.
Post-market surveillance is essential for continuously evaluating and monitoring vaccine safety. In addition to VAERS, other active surveillance databases such as BEST 15 , SPEAC 16 , CISA 17 and VSD 18 have been used for evaluating the safety of COVID-19 vaccines 7,20-22 . In general, data collected by surveys 23,24 or social media platforms 25,26 could also provide evidences for vaccine safety or hesitancy. Despite the limitation of being a passive reporting system, VAERS, with proper statistical analyses, still provides valuable information about vaccine safety, especially in the early phase of general population administration and vaccine surveillance. For example, in the first several months (Dec 2020-April 2021) of the Pfizer and Moderna COVID-19 vaccine administration, VAERS-based studies 5,8-10 emerged to generate safety signals for anaphylaxis, sudden hearing loss and cerebral venous sinus thrombosis. These studies contribute unique time-sensitive real-world evidence on safety signals that were examined rigorously by later randomized studies. Our study provides a novel method utilizing the publicly available RCT and VAERS data for monitoring the safety of COVID-19 vaccines. It is reassuring that our analyses based on the VAERS data produced evidence consistent with existing RCTs, and demonstrated the ability to identify rare AEs that were not recognized in the RCTs. Our analysis can be applied to the post-market surveillance of ongoing (e.g. COVID-19 vaccine boost dose and adolescent dose) and future vaccine administration using VAERS, especially for the severe AEs. The website https:// chong liang-luo. shiny apps. io/ covid_ vaers/ will be continuously updated for timely monitoring the COVID-19 vaccines in the future.

Methods
Data sources. We collected vaccine safety data from published RCTs for the three vaccines authorized in the U.S. For all RCTs 1-4 , local AEs (i.e. pain, erythema and swelling) and systemic AEs (i.e. headache, fever, chill, fatigue, nausea/vomiting, arthralgia/joint pain, and myalgia/muscle pain) were assessed within seven days after vaccination.
We also used data from VAERS [27][28][29] to seek post-marketing evidence on the safety of the three vaccines. The VAERS data is publicly available and no institutional approval is required. The analyses were performed in accordance with VAERS guidelines and regulations 27 . We extracted AE reports after COVID-19 vaccination up to September 17, 2021. Detailed definitions of the AEs in MedDRA preferred term (PT) level are listed in Table S1. We grouped some of the similar PT terms to their Standardized MedDRA Queries (SMQs) 30 . Reports with coadministration with other vaccines, or unknown manufacturer of COVID-19 vaccine, or invalid symptoms (e.g. "no adverse events") were excluded. To be consistent with the RCTs, we used VAERS reports with age 18 + (age 16 + for the Pfizer vaccine), and onset time within 7 days after vaccination for the solicited AEs (i.e. local and systemic AEs). For the unsolicited AEs (i.e. other rare but severe AEs including shingles, hearing impairment including tinnitus, thrombosis, facial paralysis including Bell's palsy, anaphylaxis, pulmonary embolism, myocarditis, and GBS), we used VAERS reports with onset time within 42 days after vaccination. Analysis was also repeated for age groups, i.e. young and elderly populations. We used the age cutoffs from the original trials (i.e. 55 for Pfizer trial, 65 for Moderna trial and 60 for Janssen trial) for the RCTs, and 60 for VAERS. Statistical analysis. The vaccines and doses were grouped in 5 categories, i.e. Pfizer-1st (P1), Pfizer-2nd (P2), Moderna-1st (M1), Moderna-2nd (M2) and Janssen (J). Figure 2 illustrates the overall rationale of our   www.nature.com/scientificreports/ Given the above rationale, we compared AE ranking by vaccine in both the VAERS and RCT databases, and evaluated their consistency. The rank consistency was measured by the Spearman's rank correlation coefficient ρ 31 . Spearman's ρ ranges from − 1 to 1 and a positive value indicates consistency of the two ranks. This rank correlation measure is commonly used to assess agreement between two rating approaches.
Finally, based on the consistency between RCTs and VAERS, we further ranked the vaccines regarding the risks of other rare but severe AEs (i.e. shingles, hearing impairment including tinnitus, thrombosis, facial paralysis including Bell's palsy, anaphylaxis, pulmonary embolism, myocarditis, and GBS) in VAERS.

Data availability
The RCT data analyzed in this study are available from the published papers [1][2][3][4] ; the VAERS data are publically available from https:// vaers. hhs. gov/ data. html.