Main

Vaccination is one of the most powerful tools available for improving public health1, and motivating higher rates of vaccination is currently a major global challenge2. Vaccine lotteries have rapidly risen to prominence as a means of promoting vaccination in the face of a global pandemic. Both politicians and scientists anticipated that such lotteries would be effective motivators3,4,5,6,7, and between May and July 2021, at least 21 US states launched vaccine lotteries in an effort to boost inoculation rates against COVID-19, most with jackpots of US$100,000 or more8.

There was good reason for optimism. Prior research with relatively modest rewards and convenience samples has shown that lottery incentives can change people’s health decisions9,10,11. Lottery incentives attempt to capitalize on the finding that giving people a small chance at a large pay-off can be a more cost-effective persuasion tool than providing direct payments for an action. This effectiveness stems from individuals’ tendency to overweight small probabilities12,13,14, which leads them to overvalue their long odds of winning a lottery15. Alone and in combination with other strategies, lotteries have been used to successfully motivate weight loss16, physical activity17,18, adherence to medical treatments or protocols19,20,21 and the completion of health surveys or assessments22,23. While lotteries often translate to low expected values, some early evidence suggested that even small cash payments might motivate COVID-19 vaccination. In one widely publicized survey, almost a third of unvaccinated American adults reported that payments as small as US$25 would make them more willing to get a vaccine24. A random-assignment experiment in Sweden offered early evidence that small rewards on the order of US$24 could increase COVID-19 vaccination rates25. Lotteries offering large jackpots with even lower expected values could be presumed to be effective motivators given people’s tendency to dramatically overweight small probabilities15,26.

We carried out a pre-registered, citywide experiment in Philadelphia designed to assess the effects of three high-pay-off (up to US$50,000), geographically targeted lotteries to motivate adult residents to get their first dose of a COVID-19 vaccine. Specifically, we partnered with the City of Philadelphia to launch the ‘Philly Vax Sweepstakes’ in June 2021, which gave away nearly US$400,000 in cash prizes to vaccinated Philadelphia county residents over a six-week period between 7 June and 19 July 2021 when three separate drawings were held (one every two weeks). In each of the three drawings, 12 prizes were awarded to Philadelphia adults who had received a first dose of their COVID-19 vaccine: six US$1,000 prizes, four US$5,000 prizes and two US$50,000 ‘grand prizes’. The sweepstakes included an experiment: the residents of three randomly selected Philadelphia zip codes were given a 50 to 100 times higher probability of winning prizes than other Philadelphians, and this feature of the lottery was highlighted in all media communications. The three ‘treatment’ zip codes in this experiment were chosen at random from a set of 20 prioritized Philadelphia zip codes with the lowest vaccination rates (the zip codes comprised 587,508 adult residents), and half of the total prizes from each lottery drawing were allocated to a given treatment zip code.

Previous research suggests that providing up to a 100-fold increase in the chance of winning a large cash prize should be highly motivating26,27,28,29. This is because increasing the chances of winning a lottery changes the lottery’s expected value, which both theory and prior research have shown changes behaviour18,27,28. Here we experimentally test the value of a geographically targeted, lottery-based approach to incentivizing vaccination by varying citizens’ chances of winning a prize.

Our experiment contributes to a growing body of concurrent evaluations of COVID-19 lotteries around the United States that reach varied conclusions30,31,32,33, but our work differs from other studies in several key ways. First, our study provides experimental variation in incentives at a local (zip code) level, so we are not forced to rely on comparisons between neighbouring states whose COVID-19 policies and underlying trends in vaccination can meaningfully differ. Second, our study follows a pre-registered analysis plan, limiting the opportunity to select a preferred analysis strategy after sampling results. Finally, we conducted a lottery in which all participants had a chance to win, but participants in randomly selected treatment zip codes had much higher chances of winning; other concurrent studies measured the overall impact of introducing a vaccine lottery, addressing a different research question.

Results

Defining the primary outcome and study population

Because only adults were eligible to win the Philly Vax Sweepstakes, our primary outcome was the number of first-dose vaccinations of Philadelphia residents ages 18 and older in a given zip code in a given week per 100,000 people. We calculated weekly vaccinations per 100,000 people by dividing the total first-dose vaccinations for adults in the geography of interest for a given week by the total adult population in that region according to the American Community Survey34 and then multiplying by 100,000. First-dose vaccinations included first doses of the Moderna and Pfizer vaccines as well as the single-dose Johnson & Johnson vaccine. The number of weekly first-dose vaccinations in each Philadelphia zip code for residents ages 18 and older during our study period was provided by the Philadelphia Department of Public Health35 on 18 August 2021.

The 20 zip codes included in our experiment included 587,508 adult Philadelphians, and these zip codes had an average vaccination rate of 33% before the start of the Philly Vax Sweepstakes (that is, as of 6 June 2021). Figure 1 shows a map of Philadelphia and its surrounding counties, highlighting the locations of our three treatment zip codes in Philadelphia (19126, 19133 and 19142), whose residents were offered elevated chances of a lottery win, as well as our 17 control zip codes. These treatment and control zip codes comprised the 20 Philadelphia zip codes with the lowest per capita vaccination rates as of 27 May 2021. Table 1 provides summary statistics on the demographic composition of the residents of these communities as well as the percent of each population with at least one COVID-19 vaccination dose before the start of the Philly Vax Sweepstakes.

Fig. 1: Map of Philadelphia County (where the Philly Vax Sweepstakes occurred).
figure 1

Treatment zip codes in Philadelphia are shown in dark green, control zip codes are shown in medium green and all other Philadelphia zip codes are shown in light green. This map was created using border data from Google Earth.

Full size image
Table 1 Population summary statistics for Philadelphia County and for zip codes of interest
Full size table

Ex ante simulations based only on pre-treatment data reported in Supplementary Section 1 demonstrate the effect size that we could anticipate being 90% powered to detect with our primary, pre-registered difference-in-differences regression analysis of the effect of the lottery on each treatment zip code, and in the three treatment zip codes analysed together, using a two-tailed test when α is set to 0.05 and clustering standard errors by zip code. These simulations indicate that in our first selected zip code (19126), we had 90% power to detect an effect size of 539 additional vaccinations per 100,000 people per week, corresponding to one additional vaccination for every US$463 spent (or less). In our second selected zip code (19133), we had 90% power to detect an effect size of 468 additional vaccinations per 100,000 people per week, corresponding to one additional vaccination for every US$334 spent (or less). In our third selected zip code (19142), we had 90% power to detect an effect size of 533 additional vaccinations per 100,000 people per week, corresponding to one additional vaccination for every US$283 spent (or less). Finally, our pooled model had 90% power to detect an average effect size of 569 additional vaccinations per 100,000 people per week, corresponding to one additional vaccination for every US$326 spent (or less). These simulations suggest that we were fairly well powered to detect practically meaningful effect sizes, as most direct incentives from policymakers for COVID-19 vaccination in the United States have been on the order of US$100 or less8. Ex post data reveal that our power to detect effects was better than expected because vaccination rates declined precipitously during the pre-treatment period before levelling off during treatment.

The impact of treatment assignment on sweepstakes sign-ups

A necessary condition for our zip code treatments to be effective was citizens’ awareness of the incentives they faced. To assess the effect of our marketing campaign, which was executed with the Philadelphia Department of Public Health (Methods), and, in particular, to determine whether we successfully created differential expectations about the chances of a sweepstakes win among residents in treated zip codes versus control zip codes, we examine the distribution of manual registrations for the sweepstakes across zip codes over time.

Although manual registration for the sweepstakes wasn’t required to win (because the sweepstakes drawing pool was seeded with the names and contact information for 1,064,805 Philadelphia adults36; Methods), residents were encouraged to actively register online at phillyvaxsweeps.com or by phone to ensure they were included. A total of 6% of adult Philadelphians manually registered for the sweepstakes during its six-week run.

The vast majority of media attention and manual registrations for the sweepstakes occurred before the first-round drawing. Seventy-one per cent of active registrants (or 4% of adult Philadelphians) registered in the two weeks leading up to the first of the three drawings, which was the period when the sweepstakes received the most press coverage (Supplementary Table 1). The remaining 2% of Philadelphia adults who registered for the sweepstakes did so over the remaining four weeks (Fig. 2).

Fig. 2: Daily manual registrations for the Philly Vax Sweepstakes at phillyvaxsweeps.com.
figure 2

Daily manual registrations are plotted as a function of the total city population over the six weeks following the sweepstakes’ launch (on 7 June 2021) and up to (and including) the day before the final drawing (on 19 July 2021).

Full size image

As shown in Fig. 3a, 5.59% of adult residents of the first treated zip code (19126) actively registered for the Philly Vax Sweepstakes in the two weeks leading up to the first drawing compared with 2.74% of adult residents of the control zip codes (z = 19.081, P < 0.001). The fact that registrations were 2.04 times higher in the first treatment zip code than in the control zip codes in the two weeks before the first drawing suggests that we successfully raised awareness among treatment zip code residents of their heightened chances of winning the lottery. Notably, although we advertised that registration was not required to participate, in just two weeks more than 1 in 18 adults living in 19126 navigated to the Philly Vax Sweepstakes website and filled in their contact information to be sure they were included in the first drawing, suggesting a very high level of awareness about the lottery in this zip code.

Fig. 3: Manual registrations for the Philly Vax Sweepstakes at phillyvaxsweeps.com by region and treatment period.
figure 3

a–c, The percentage of manual registrations for the Philly Vax Sweepstakes is calculated based on the total city population for each treated zip code (19126 in a, 19133 in b and 19142 in c) and for the control zip codes during each of the three treatment periods, highlighting the relevant treatment period for each treated zip code.

Full size image

Registrations from subsequent treatment zip codes were far lower, with smaller absolute gaps in registration emerging between treatment zip code residents and control zip code residents, though significant and large relative gaps in registration suggest that in each treatment zip code, there was meaningful awareness of the higher odds faced. As shown in Fig. 3b, 1.17% of adult residents of the second treated zip code (19133) registered in the two weeks leading up to the second drawing compared with 0.55% of adult residents in the control zip codes (z = 11.310, P < 0.001), meaning that the registration rate was 2.13 times higher in the treatment zip code than in the control zip codes. As shown in Fig. 3c, 1.75% of adult residents in the third treated zip code (19142) registered in the two weeks leading up to the third drawing compared with 0.62% of adult residents in the control zip codes (z = 19.602, P < 0.001), meaning that the registration rate was 2.82 times higher in the treatment zip code than in the control zip codes.

Overall, we saw the most registrants in a treatment zip code in the two weeks leading up to a drawing in the first treatment zip code (5.59%)—a number that dwarfed registrations in the treatment zip codes during both the second (1.17%; z = 23.139, P < 0.001) and third treatment periods (1.74%; z = 19.261, P < 0.001). See Supplementary Section 2 for more information about registrations.

The impact of treatment assignment on vaccinations

Figure 4 presents raw weekly vaccination data for each of the three treatment zip codes compared with the average of the 17 control zip codes during each week of the experiment, highlighting the relevant two-week treatment period for each treatment zip code as well as a synthetic control counterfactual for each treatment zip code constructed following Abadie et al.37 (Fig. 4a,c,e). It also presents the difference in weekly vaccinations per 100,000 people in the treatment and control zip codes (Fig. 4b,d,f). As shown in Fig. 4a,b, there was an initial but unsustained uptick in vaccinations of roughly 40% (from a pre-sweepstakes baseline of 569 per 100,000 people) in the first treatment zip code (19126) following the announcement of the sweepstakes compared with the control zip codes. However, as shown in Fig. 4c,d, the second treatment zip code (19133) did not experience any such uptick compared with our control zip codes. Neither did the third treatment zip code (19142), as shown in Fig. 4e,f.

Fig. 4: Vaccinations in the treatment and control zip codes.
figure 4

a–f, The panels on the left present the weekly number of first-dose vaccinations per 100,000 adult Philadelphians in each of the treated zip codes (19126 in a, 19133 in c and 19142 in e) versus the pooled 17 control zip codes and a synthetic control group. The panels on the right present the difference in the raw number of weekly first-dose vaccinations per 100,000 adult Philadelphians between the treated and control zip codes (19126 in b, 19133 in d and 19142 in f). The weekly data are plotted on the last day of a given week (for example, the data for the week 31 May–6 June is plotted on 6 June).

Full size image

To assess the significance of the trends visualized in Fig. 4, for each of our three treatment zip codes, we estimated a separate, pre-registered difference-in-differences ordinary least squares regression predicting weekly first-dose adult vaccinations per 100,000 people in that zip code and in all control zip codes. The difference-in-differences literature has emphasized the importance of clustering standard errors by geography38. However, these standard errors can be biased when the number of clusters is small39. Following Dube et al.40 and our pre-registered analysis plan40, we began by running each model three times and reporting the standard errors produced when clustering by week (15 clusters), clustering by zip code (18 clusters) and using HC3 robust standard errors, noting that the model with the largest standard errors was the most conservative. We supplemented our pre-registered analyses in two ways. First, we conducted permutation tests to evaluate the sharp null hypothesis that the impact of each sweepstakes was what would be expected due to chance41 (Supplementary Section 3; these results are consistent with those presented below). Second, we reran our analysis using a synthetic control approach with significance tests produced through randomization inference37.

Models 1 to 3 in Table 2 report the regression-estimated effects of the treatment on the first, second and third treatment zip codes, relative to our 17 control zip codes. We also conducted equivalence tests, using a one-sided t-test procedure on the upper bound42,43, for each estimated treatment effect. The upper bound that we test is one that would be generated by a marginal US$1,000 spent per vaccine—an effect that our study was well powered to detect (we could, in fact, detect smaller effects, such as those generated by a marginal US$100 spent per vaccine, which represents the upper end of what US policymakers spent in 2021 to reward COVID-19 vaccination8). See Supplementary Section 4 for more on these equivalence tests.

Table 2 Regression-estimated impact of being selected as a treatment zip code in the Philly Vax Sweepstakes on weekly first-dose COVID-19 vaccinations per 100,000 people
Full size table

Model 1 shows that the first treatment zip code (19126) experienced an estimated average weekly increase of 61 vaccinations per 100,000 residents (or 11%) in the two weeks of the relevant treatment period, when 19126 residents had 98 times other Philadelphians’ chances of winning a prize (most conservative: t(237) = 0.940; P = 0.350; ∆ = 10.7%; 95% confidence interval (CI) (−67, 188); least conservative: t(17) = 3.020; P = 0.008; ∆ = 10.7%; 95% CI (18, 103)). An equivalence test performed on our most conservative pre-registered model allows us to reject the null hypothesis that our treatment effect generated more than 250 additional vaccinations per 100,000 people, or one vaccine per US$1,000 spent (t(250) = 4.501, P < 0.001). We also generated a synthetic control estimate of the impact of the odds boost on vaccinations in the first treatment zip code. This method estimates an average weekly increase in vaccinations of 52 vaccinations per 100,000 people in our first treatment zip code (∆ = 9.1%, P = 0.588).

The second treatment zip code (19133) experienced an estimated non-significant average weekly increase of 19 vaccinations per 100,000 people in the two weeks of the relevant treatment period, when 19133 residents had 61 times other Philadelphians’ chances of winning a prize (most conservative: t(237) = 0.350; P = 0.728; ∆ = 1.6%; 95% CI (−87, 125); least conservative: t(17) = 1.100; P = 0.288; ∆ = 1.6%; 95% CI (−17, 55)). An equivalence test performed on our most conservative pre-registered model allows us to reject the null hypothesis that our treatment effect generated more than 156 additional vaccinations per 100,000 people, or one vaccine per US$1,000 spent (t(156) = 3.057, P = 0.001). We also generated a synthetic control estimate of the impact of the odds boost on vaccinations in the second treatment zip code; this model estimates that our treatment produced an average weekly increase in vaccinations of 43 vaccinations per 100,000 people in 19133 (∆ = 3.7%, P = 0.765).

Finally, the third treatment zip code (19142) experienced an estimated average weekly decrease of 102 vaccinations per 100,000 people in the two weeks of the third treatment period, when 19142 residents had 59 times other Philadelphians’ chances of winning a prize (most conservative: t(237) = −1.150; P = 0.253; ∆ = −9.6%; 95% CI (−276, 73); least conservative: t(17) = −2.910; P = 0.010; ∆ = −9.6%; 95% CI (−175, −28)). An equivalence test performed on our most conservative pre-registered model allows us to reject the null hypothesis that our treatment effect generated more than 151 additional vaccinations per 100,000 people, or one vaccine per US$1,000 spent (t(151) = 2.673, P = 0.004). We also generated a synthetic control estimate of the impact of the odds boost on vaccinations in the third treatment zip code; this model estimates that our treatment produced an average weekly increase in vaccinations of 45 vaccinations per 100,000 people in 19142 (∆ = 4.2%, P = 0.294).

The impact of the lottery in the third treatment zip code is statistically non-significant using both the difference-in-differences model and the synthetic control model. However, the directions of the estimates produced by these two models are noticeably different. Why is that? Difference-in-differences models rely on an assumption of parallel trends in pre-treatment data, while synthetic control models weight the control data in a manner that produces parallel trends. We tested for violations of the parallel trends assumption inherent in each of our difference-in-differences models by adding interaction terms between each week indicator and the indicator for the relevant treatment zip code to each regression model. We then examined the significance of the coefficient estimates on the pre-treatment interaction terms and conducted an F-test of their joint significance (Supplementary Fig. 1 depicts these estimates visually). In the first treatment zip code (19126), the coefficients on the individual interaction terms and on the joint F-test were all non-significant (all P ≥ 0.354), suggesting that we cannot reject parallel trends leading up to the treatment period for the first treatment zip code. In the second treatment zip code (19133), the coefficients on the individual interaction terms and on the joint F-test did not reach standard levels of significance (all P ≥ 0.218), which again suggests that we cannot reject parallel trends leading up to the treatment for the second treatment zip code. In the third selected treatment zip code (19142), however, the coefficients on the interaction terms between the weeks ending on 23 May and 30 May and the treatment zip code are significant (P = 0.041 and P = 0.033, respectively), and the joint F-test also reveals significant differences in pre-treatment trends (P = 0.020). This suggests that we can reject parallel trends leading up to the treatment of the third treatment zip code and that our difference-in-differences estimation here should be interpreted with caution, which may explain why our synthetic control treatment estimate is so different for the third treatment zip code.

The effect of the sweepstakes on the pooled treatment zip codes

In Table 2, Model 4, we present our pre-registered pooled difference-in-differences model assessing the combined effect of all three treatment periods on vaccinations in all three treatment zip codes. The estimated pooled weekly vaccination rate in the treatment zip codes during the two weeks following the announcement of their elevated chances to win declined by a non-significant 4 vaccinations per 100,000 people compared with the control zip codes during the same period (most conservative: t(14) = −0.080; P = 0.936; ∆ = −0.4%; 95% CI (−99, 92); least conservative: t(19) = −0.090; P = 0.929; ∆ = −0.4%; 95% CI (−79, 71); see Supplementary Section 5 for the parallel trends tests). An equivalence test performed on our most conservative model allows us to reject the null hypothesis that our treatment effect generated more than 186 additional vaccinations per 100,000 people, or one vaccine per US$1,000 spent (t(186) = 3.942, P < 0.001). We also generated a synthetic control estimate of the pooled impact of the odds boost on vaccinations in the three treatment zip codes; this model estimates that the treatment produced an average weekly increase in vaccinations of 33 vaccinations per 100,000 people in the Philly Vax Sweepstakes (∆ = 3.5%, P = 0.882). Again, the point estimate from our pooled difference-in-differences model differs somewhat from the point estimate from our synthetic control model, probably because of the parallel trends issues described above. Notably, however, both estimates indicate that the odds boosts given to the treatment zip codes during the Philly Vax Sweepstakes did not significantly increase vaccinations.

The effect of the overall sweepstakes on Philadelphia

Although the overall Philly Vax Sweepstakes was not implemented experimentally, we can still attempt to estimate its effect on Philadelphia County following a pre-registered difference-in-differences, county-by-county analysis. We present two different pre-registered difference-in-differences analyses that separately compare Philadelphia County vaccinations with the vaccination rates in (1) surrounding counties and (2) Pittsburgh’s Allegheny County following Roberto et al.44 (we substitute Pittsburgh for Baltimore in our analysis because Baltimore held a concurrent vaccine lottery, making it an inappropriate control). We use county-level vaccination data from the Centers for Disease Control and Prevention45 combined with population data from the American Community Survey34. This analysis relies on the same regression formulation as our difference-in-differences zip code analysis but aggregates up to the county level and focuses on Philadelphia County as the treated unit.

As reported in Supplementary Table 3, we estimated that the pooled weekly vaccination rate in Philadelphia County during the six-week sweepstakes increased by 383 vaccinations per 100,000 people compared with surrounding counties but decreased by 116 vaccinations per 100,000 people during the same period compared with Allegheny County (compared with surrounding counties, most conservative: t(14) = 1.890; P = 0.080; ∆ = 27.0%; 95% CI (−52, 819) and least conservative: t(3) = 4.830; P = 0.017; ∆ = 27.0%; 95% CI (131, 636); compared with Allegheny County, most conservative: t(13) = −0.300; P = 0.772; ∆ = −8.2%; 95% CI (−960, 729) and least conservative: t(14) = −0.320; P = 0.756; ∆ = −8.2%; 95% CI (−899, 667)). When we tested for violations of the parallel trends assumption inherent in our difference-in-differences models by adding interaction terms between each week indicator and the indicator for Philadelphia County to our difference-in-differences regression model comparing Philadelphia with surrounding counties, we found that the coefficient estimates on the pre-treatment interaction terms are jointly significant (F-test P value 0.010). Our Allegheny County comparison had too few clusters to run this test, but visual inspection suggested a clear parallel trends violation (Supplementary Fig. 2). The overall impact of the Philly Vax Sweepstakes was therefore ambiguous. We conducted robustness checks of these estimates by comparing Philadelphia with its surrounding New Jersey counties, as well as re-estimating our pre-registered analysis using an alternative data source in Supplementary Sections 6 and 7, respectively.

We also ran an exploratory synthetic control estimate of the impact of the Philly Vax Sweepstakes on Philadelphia County vaccinations compared with vaccinations in Pennsylvania’s 63 other counties with available data (there are too few counties surrounding Philadelphia for a synthetic control estimate to yield reliable results41). As shown in Supplementary Fig. 3, Philadelphia’s vaccination rate exceeded its counterfactual throughout the treatment period. However, in the week before the start of the sweepstakes, Philadelphia experienced an uptick in vaccinations that the synthetic control was unable to match accurately. Our synthetic control estimate is therefore likely upwardly biased. This model estimates that the sweepstakes produced an average weekly increase in vaccinations of 309 vaccinations per 100,000 people (∆ = 21.8%, P = 0.079).

Discussion

Our study presents an experimental evaluation of a large-scale, geographically targeted lottery that varied the odds of a win across zip codes in an attempt to incentivize COVID-19 vaccination. Overall, the findings suggest that when residents are given ~100× the chances of their neighbours in other zip codes to win up to US$50,000 for getting a vaccine (roughly 1 in 2,000 odds) and when this makes the front page of the local newspaper and is featured on the local nightly news, it may generate a very small boost in vaccinations. Specifically, our first treatment zip code, which benefited from far more media coverage than the subsequently announced treatment zip codes, saw an estimated 11% uptick in vaccination in the two weeks leading up to a drawing. In this zip code of 12,485 adult residents (Table 1), the odds boost to residents in our lottery produced an estimated 15 extra vaccinations over two weeks at a marginal cost of roughly US$62,376, or US$4,158 per vaccination (an exorbitant price tag; see Supplementary Section 8 for the background on this estimate). No other treated zip code experienced plausibly significant benefits (that is, the other treatment effects were smaller, and equivalence tests allowed us to reject the null hypothesis that any of our treatments increased vaccinations for a price tag of US$4,690 per person or less). Pooling across the treated zip codes, we can tightly bound the upside of geographically targeted vaccine lotteries across our study: the upper bound on the effect of treating a zip code from our most conservative 95% CI is a 9% boost in vaccination rates for two weeks. (The 95% CI in our treatment zip codes bounds the effects between reducing weekly vaccinations by 93 per 100,000 people and increasing weekly vaccinations by 85 per 100,000 people.)

This study builds on past research by experimentally evaluating a series of geographically targeted vaccine lotteries at a large scale, encompassing over 500,000 Philadelphia residents in undervaccinated communities (or well over 1,000 times as many participants as typical in past experiments with lotteries and health behaviour change16,17,18,19,20,21,22,23). A major difference between our study and past experimental research on lottery incentives is that the participants in this experiment were probably less aware of the incentives they faced. Our results tentatively suggest that awareness or salience may play a crucial role in the impact of lottery incentives. It is possible that handing out actual, physical lottery tickets to all citizens would have increased awareness, but it would have been logistically challenging.

We also experimentally vary people’s odds of winning a lottery and not whether people face lottery incentives at all, which is a key distinction between our work and past research that could explain the discrepancy between our findings and the findings from smaller-scale studies of lottery incentives. Economic theory suggests that people should respond similarly to the introduction of lottery incentives, changes in their odds of winning a lottery (so long as the odds remain low enough to provoke the overweighting of small probabilities15,26) and changes in the value of lottery prizes, because all of these variations simply change the expected value of receiving a vaccine46. Future research exploring this assumption would be valuable, as changing the odds of a lottery win might matter more when contrasted against no chance of a win in surrounding zip codes. It would also be valuable to explore whether changing a lottery’s jackpot would have a larger impact than varying citizens’ odds of a lottery win. Left-digit bias suggests that reducing the size of our US$50,000 jackpot to US$49,000 would have reduced its potency more than increasing the jackpot to US$51,000 would have boosted its impact47, but finding the optimal combination of jackpot size and probability of a win is an intriguing open question.

Our paper adds to a growing, concurrent set of studies examining the effects of large COVID-19 vaccination sweepstakes launched across the United States in 2021, which have reported primarily null results30,31,32,33. While other sweepstakes evaluations have focused on non-experimental, typically post hoc evaluations of the overall impact of running statewide lotteries, we focus on a pre-registered analysis of an experimentally implemented vaccine sweepstakes. Furthermore, our sweepstakes varied citizens’ odds of a win across zip codes rather than varying the overall presence of a lottery. We thus test a very different question from other, concurrent studies. That said, like other recent papers that have attempted to pinpoint the effects of vaccine lotteries without the benefit of experimental designs, we sought to generate an estimate of the overall impact of the Philly Vax Sweepstakes on Philadelphia County compared with surrounding and comparable counties. We present pre-registered, inconclusive results from our attempts to do so. Taken together, this research paints a discouraging picture of the potential for either introducing a vaccine lottery or vastly boosting citizens’ odds of winning a vaccine lottery as a means of encouraging vaccination. These tools do not appear to offer a reliable, cost-effective route to boosting immunizations, despite a rich set of smaller-scale research that suggests that lotteries can be a low-cost tool for changing health behaviours in other contexts9,10,11,16,17,18,19,20,21,22,23.

Our study has several limitations worth noting. First, our sample (Philadelphia residents) is not representative of the broader US population, let alone the global population. We also cannot directly compare how well geographically targeted lotteries perform relative to other policy tools, nor can our results rule out the possibility that geographically targeted lotteries with substantially larger jackpots might have had more robust, positive effects. Similarly, we cannot rule out the possibility that enhanced local marketing or coordination with community health organizations to improve awareness, comprehensibility of geographically targeted incentives and trust could have increased the impact of our lotteries. It is also possible that offering lottery incentives only to those who had not yet made a vaccination decision could have increased their potency (albeit raising valid fairness concerns). We further cannot compare how well geographically targeted lotteries would have worked earlier in the vaccine rollout, when individuals’ motivations may have differed. Consistent with this possibility, recent work by Rabb et al.48 suggests that text reminders to get a COVID-19 vaccine may have been less potent in later stages of the vaccine rollout than they were at earlier stages49. Finally, we cannot account for the possibility that statewide jackpots across the nation involving multiple million-dollar prizes may have created a reference point that psychologically diminished the impact of our US$50,000 and smaller lottery prizes.

Our goal in selecting the 20 treatment-eligible zip codes with the lowest vaccine uptake before the sweepstakes was to close gaps in vaccine demand and uptake. This approach was based on an assumption that the types of barriers or hesitancy faced by the residents of these zip codes might be addressed by a financial reward with low expected value. Our results suggest that, at least in the context of Philadelphia’s COVID-19 pandemic and vaccine roll-out in the summer of 2021, unvaccinated individuals required something more. As demonstrated through other efforts in Philadelphia and elsewhere, effective vaccine promotion requires input from trusted leaders in the undervaccinated areas, an understanding of vaccine distrust and demonstrated efforts to mitigate the impact of the pandemic. Despite its limitations, this pre-registered experiment gave us the opportunity to causally evaluate the benefits of concentrating rewards in undervaccinated treatment zip codes. To our knowledge, no other vaccine lottery has incorporated experimentation of any kind, and doing so made it possible to obtain a very precise estimate of the limited effect of geographic targeting.

In conclusion, we found that giving the residents of certain zip codes massively higher odds of winning a lottery jackpot did not meaningfully alter their vaccination decisions, and we estimate a fairly tight 9% upper bound on the benefits of such treatments. We add to a growing literature on vaccine lotteries, with implications for policymakers seeking behaviour change at scale whether in the context of COVID-19 or in that of other health-promoting activities. As the COVID-19 pandemic continues to evolve, we hope that demonstrating the limited effectiveness of our three zip-code-targeted vaccine regret lotteries will encourage policymakers to look for other, more impactful ways to encourage vaccination.

Methods

Ethics approval and pre-registration

Prior to implementation, the design of the Philly Vax Sweepstakes was reviewed and approved by the institutional review boards of the University of Pennsylvania and the City of Philadelphia. Informed consent was waived by both institutional review boards because the study was deemed to pose minimal risk to the participants, it could not be practicably carried out without a waiver of informed consent and waiving consent did not adversely affect the participants. Our study’s analysis plan was also pre-registered on 7 June 2021 at https://osf.io/z7gqj, and an updated version revised to address the fact that a planned control city had launched its own vaccination lottery was submitted on 17 June 2021 at https://osf.io/26c9z.

Sweepstakes design and implementation

All adult residents of Philadelphia who received at least one dose of the COVID-19 vaccine were eligible to win a prize in the Philly Vax Sweepstakes. The sweepstakes drawing pool was seeded with the names and contact information for 1,064,805 Philadelphia adults from a purchased commercial database36. In addition, to ensure they were included, residents could actively register for the sweepstakes online at phillyvaxsweeps.com or by phone. Both registration channels were managed by a professional sweepstakes vendor, Universal Promotions, Inc. By the close of the sweepstakes, 75,356 people (6% of adult Philadelphians) had actively registered (see Fig. 2 for registration volume over time and Supplementary Information for more on how the sweepstakes was advertised). The winners’ names were drawn from a deduplicated database.

Residents whose names were drawn but who had not received their first dose of the COVID-19 vaccine before the day of the drawing were ineligible to claim a prize. We reached out to each resident whose name was drawn using all available contact information. Residents had at least 48 hours to claim their prize after being successfully contacted. If they did not, or if they could not be successfully contacted after all available means were exhausted, a new name was drawn. Furthermore, when residents were reached, if proof of first-dose vaccination could not be verified by the jurisdiction in which a resident received their shot, a new name was drawn for that prize. By design, this feature of the lottery created the potential for regret16,17,18,19,20,21,22,23.

The experimental component of the Philly Vax Sweepstakes was designed as follows: the 20 Philadelphia zip codes with the lowest per capita vaccination rates as of 27 May 2021 (11 days before the sweepstakes) were included in the experiment50, and 3 Philadelphia zip codes were randomly selected from this set for treatment, defined as vastly increased odds of winning the sweepstakes. The other 17 zip codes from this pool became controls. Each treatment zip code was announced two weeks before the drawing in which its residents would have heightened odds of winning prizes, and the residents of this zip code enjoyed 59 to 98 times higher chances of winning a prize for that drawing (depending on the zip code’s population) compared with other Philadelphians. Specifically, each resident in the first treated zip code (19126) had a 1 in 2,081 chance of winning, or 98 times the chances of residents in other parts of Philadelphia (1 in 203,542). Each resident in the second treated zip code (19133) had a 1 in 3,304 chance of winning, or 61 times the chances of residents in other parts of Philadelphia (1 in 202,307). Finally, each resident in the third treated zip code (19142) had a 1 in 3,427 chance of winning, or 59 times the chances of residents in other parts of Philadelphia (1 in 202,184). Our experiment’s design allowed us to causally evaluate the impact of increasing some Philadelphians’ chances of winning and targeted more lottery resources towards underserved communities.

The Philly Vax Sweepstakes officially launched on Monday, 7 June 2021, with a press conference featuring the mayor of Philadelphia and the announcement of the first treatment zip code and its residents’ vastly elevated odds of winning. Following the 7 June announcement of the first treatment zip code, a new treatment zip code was randomly selected fortnightly and announced on the Philly Vax Sweepstakes website, with the second treatment zip code announced on Monday, 21 June, and the third announced on Tuesday, 6 July (Monday, 5 July, was a national holiday).

On the day the sweepstakes was announced (7 June 2021), it was a featured story on at least five local news channels51,52,53,54,55, and it was a front-page story in the most prominent local newspaper, the Philadelphia Inquirer56, on 8 June 2021. Over 35 different news outlets published original stories about the sweepstakes at some point during the six-week programme (see Supplementary Table 1 for the full list of news articles), which ended with the third and final drawing of winners on Monday, 19 July 2021. Sixty-two per cent of those stories were published during the first treatment period, 29% were published during the second treatment period and 9% were published during the third treatment period. Sixty-three per cent of these stories discussed the elevated chances of a win for the residents of the treatment zip codes.

The sweepstakes was marketed by the City of Philadelphia through twice-weekly press releases emphasizing the current treatment zip code, Nextdoor posts, text messages from the emergency information notification system, automated phone calls, a press conference announcing the first-round winners and a press release with quotes from the second-round winners. There were also radio advertisements on Philadelphia’s two largest Hip-Hop and R&B stations and emails and text messages to patients about the lottery from Penn Medicine, a large, regional health system (see Supplementary Section 9 for more information about how and when the sweepstakes was advertised).

Google search trends data for ‘Philly Vax Sweepstakes’ indicate that considerably greater attention was focused on the Philly Vax Sweepstakes immediately following its launch on 7 June 2021 compared with later in the summer (Supplementary Fig. 4).

Statistical analysis

We evaluate the impact of our geographically targeted lottery treatment using pre-registered difference-in-differences analyses57, and we compare weekly vaccinations per 100,000 adult residents over time. Specifically, to evaluate the impact of randomly assigning the residents of certain Philadelphia zip codes to have ~50× to 100× other Philadelphians’ chances of winning our vaccine lottery, we compared the difference in weekly first-dose vaccinations per 100,000 people in each of our 3 treatment zip codes and in each of our 17 control zip codes. We did this comparison before versus during the two weeks leading up to a drawing when treatment zip code residents had vastly elevated odds of winning (73 times higher than those residing in the control zip codes, on average).

To execute these difference-in-differences analyses, we ran ordinary least squares regressions to predict weekly first-dose vaccinations per 100,000 people. Our predictors in these regressions were zip code fixed effects, week fixed effects and an indicator variable that took on a value of 1 for the two weeks when Philadelphians in the treatment zip code in question had higher odds of winning a prize. Following Dube et al.40 and our pre-registered analysis plan40, and as described in our Results section, we ran each model three times and report the standard errors produced when clustering by week (15 clusters), clustering by zip code (18 clusters) and using HC3 robust standard errors, noting that the model with the largest standard errors is the most conservative.

Finally, we report the results of synthetic control analyses to assess the impact of treatment assignment58. The synthetic control method37 constructs a counterfactual of our treatment zip code by creating a weighted average of the 17 control zip codes such that the counterfactual closely tracks the selected zip code during the pre-treatment period, for our outcome of interest. For our pooled model, we followed Cavallo et al.59, which extends the synthetic control method of Abadie et al.37 for analysing multiple groups over different treatment periods59. Our matching procedure used the three weeks of pre-treatment data immediately prior to the start of each lottery for our outcome variable (weekly vaccinations per 100,000 people in a given zip code). We estimated an average two-week treatment effect from being selected as a treatment zip code by taking the difference in weekly vaccinations between our treated zip code and the counterfactual generated during the treatment period. The significance of our results was determined using permutation tests of ratios of the root mean squared prediction error (treatment-period RMSPE over pre-treatment RMSPE) following Abadie et al.60.

As a robustness check, which we report in Supplementary Section 3, we generated an alternative set of standard errors for our difference-in-differences models using permutation tests. In permutation tests41, each control zip code in a given model was treated as a placebo treatment zip code for the respective treatment period. We then ran our difference-in-differences model for each of these treatment assignments and used the coefficients to form a placebo distribution of the effects of the sweepstakes58. The estimated two-sided P value was the proportion of times we observed an effect greater than or equal to the actual observed treatment effect within our placebo distribution, and CIs were constructed using test statistic inversion59. Note that because we only have 17 control clusters and 1/17 = 0.0588, our 95% CIs for these permutation tests are actually 94.12% CIs.

We also present a pre-registered attempt to evaluate how the Philly Vax Sweepstakes affected vaccinations in Philadelphia County overall versus surrounding counties and separately versus Pittsburgh’s Allegheny County using a difference-in-differences approach. In these analyses, our predictor variables in difference-in-differences ordinary least squares regressions to predict weekly first-dose vaccinations per 100,000 people were county fixed effects, week fixed effects and an indicator variable that took on a value of 1 for the six weeks of the Philly Vax Sweepstakes. Our standard error estimation procedure follows Dube et al.40, like our zip-code-level standard error estimation procedure, both of which follow our pre-registered analysis plan40.

Finally, we report the results of synthetic control analyses for the overall impact of the sweepstakes on Philadelphia. The counterfactual was a weighted average of the 63 Pennsylvania counties besides Philadelphia with available first-dose vaccination data from the Centers for Disease Control and Prevention such that the counterfactual closely tracks the Philadelphia County first-dose vaccination rate per week per 100,000 residents during the pre-treatment period. Our matching procedure used the three weeks of pre-treatment data immediately prior to the start of the sweepstakes for our outcome variable (weekly vaccinations per 100,000 people in a given county). We estimated an average six-week treatment effect for Philadelphia by taking the difference in weekly vaccinations between Philadelphia County and the counterfactual generated during the treatment period. The significance of our results was again determined following the same procedure used by Abadie et al.60 as described for our other synthetic control analyses60.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.