Introduction

Motivating individuals to contribute to public goods is a central focus of public policy at every level of governance, with examples ranging from road safety1,2,3, maintenance of public spaces and institutions4, security of persons and property5, civic engagement6, tax compliance7, air and water quality8,9,10, and climate change mitigation11. The traditional economic approach to motivating individual contributions emphasizes the use of financial incentives12,13,14 such as prizes, fines, taxes, and subsidies. However, a growing body of scientific literature shows that when it comes to motivating contributions to public goods, financial incentives have a mixed track record. A theoretical literature suggests financial incentives can sometimes reduce people’s motivation to contribute to public goods by, e.g., interfering with non-monetary motivations such as reputation15. Meanwhile, an empirical literature documents a great deal of variation in their impact, and includes a perhaps surprisingly large number of examples in which financial incentives failed to promote contributions to public goods16,17,18,19,20,21,22,23,24 (e.g., when 2, 10, and 25 USD incentives failed to increase voter turnout in a 2010 election in Lancaster, CA25 or pay-for-performance failed to improve the work performance of U.K. public sector employees26), as well as some striking examples in which financial incentives backfired18,27 (e.g., when the introduction of performance pay led volunteers to work less hard28, or the introduction of a fine led to less compliance29).

Little is known about policy makers’ knowledge about the effectiveness of such financial incentives. A systematic review reveals no research on this topic (see SM, Section 2.1). There are reasons to believe that policy makers are not accounting for the potential pitfalls identified in the literature. One reason is that neoclassical approaches remain central in policy education. For instance, we reviewed curricula of all 17 public management programs for federal and state civil servants and the top 10 law schools in Germany—two of the most popular degrees held by German mayors30,31—and found that none of these curricula included the above ‘behavioral’ literature on the potential limitations of financial incentives. Another reason is that policy makers make frequent and ongoing use of financial incentives to promote contributions to public goods. Examples include the prominent role of material rewards in the Chinese social credit system32, subsidies for the adoption of green technologies within the U.S.’s 2022 infrastructure bill33, Germany’s suite of energy efficiency and electric vehicle subsidies34, as well as vaccine lotteries in many areas of the United States35. Finally, there are a number of behavioral and organizational barriers and biases that might slow adoption of evidence related to financial incentives, e.g., it is difficult to evaluate such evidence36, it is common to place too much weight on positive effects or local experts’ perspectives37,38, it is more natural to rely more upon prior attitudes and beliefs than on research39, political beliefs can influence how one views the research40, or, even upon updating one’s own beliefs, it might be hard to adopt evidence due to organizational inertia41. The lack of policy education about the potential pitfalls of financial incentives, incentives’ prominent role in policy, and the plethora of barriers and biases standing in the way of evidence adoption all suggest that policy makers might place more confidence in the impact of financial incentives than is warranted by the scientific consensus.

In this study, we present a stark example in which one should be uncertain about whether financial incentives would successfully motivate contributions to a public good, yet policy makers expected they would work. We focus on the case of COVID-19 vaccination. This is a setting where the scientific literature would suggest uncertainty about the effectiveness of financial incentives. Empirically, while there have been several high-profile studies42,43 that have found financial incentives to be effective in increasing COVID-19 vaccination, one meta-analysis44 and one systematic review45 have found that financial incentives’ impacts on COVID-19 vaccination varied substantially, having had a small positive effect on average across all studies, but having had no impact in a relatively large share of studies. The meta-analysis concludes that the effect sizes of financial and non-financial treatments do not differ significantly44. In a few cases, financial incentives even backfired46,47,48. There are also theoretical reasons to believe that financial incentives may have limited effectiveness in this context. First, it is a setting where concerns over the public good are paramount: although vaccination confers clear benefits to the individual, there are also important benefits to the public good, via reduced transmission49,50,51 and mutation of the virus52,53,54,55. Moreover, a goal of public policy in this domain is to motivate individuals to vaccinate even when they might not have done so purely out of concern for their own health56. Second, non-financial motivations in general, and reputational motivations in particular play an outsized role in motivating COVID-19 vaccination57 and evasion thereof58. For instance, in survey data from Germany, 65.8% of respondents cited “I want to contribute to eradicating the virus” as one of their top three motivations for vaccination, and 21.2% cited “My social environment exerts pressure”59. Third, proposed financial incentives for vaccination tend to be relatively small—typically the equivalent of about 25 USD. For instance, one prominent study employed a 20 Euro incentive in increasing vaccination43, and the author team themselves characterized this incentive as “small.” As has been argued15 and shown empirically28, moderate to small financial incentives can fail or backfire because they can undermine other motivations without providing sufficient motivation of their own right15,29. Finally, financial incentives for vaccinations may backfire because they lead people to infer that the vaccine is risky. Zhang and Lane47 found that even large monetary incentives reduced Chinese nationals’ willingness to get vaccinated against COVID-19 and they speculated the monetary incentives were construed as signals about the vaccine’s safety, and in a study of willingness to participate in a clinical trial, increasing financial incentives for participation increased the prospective participants’ perception of study risk60.

Yet, in the midst of the COVID-19 pandemic, as governments at all levels considered policies to motivate vaccination against the virus, financial incentives were prominently discussed and promoted by numerous academics61,62, policy makers63 including by prominent public health institutions such as the US Centers for Disease Control and Prevention64, in the media65,66, and even in congressional testimony67. In prediction studies with scientific experts, financial incentives were predicted to be particularly effective61. Many localities ultimately implemented financial incentives68, including nearly half of U.S. states68 and major cities such as Philadelphia69; Ohio and California are noteworthy in that they offered large lottery incentives of up to 1–1.5 million USD, respectively68.

During this period, our team collaborated with the German town of Ravensburg to test policy makers’ predictions of a policy that relied on financial incentives to promote COVID-19 vaccination. We also evaluated the actual impact of this policy on vaccination uptake via a randomized controlled trial using the entire population of Ravensburg. Ravensburg is a city in southern Germany, in the state of Baden-Wuerttemberg, approximately two hours by car or train from Munich, Stuttgart, and Zurich. It has 41,548 adult inhabitants living in 10,032 distinct addresses. The mean net household income is 49,107 Euros, 2.7 percent above Germany’s national average70. In November and December 2021, Ravensburg organized a series of seven COVID-19 vaccination events and an accompanying marketing campaign including a personalized letter to every adult resident to advertise the events. At the time the city commenced its campaign, most residents were eligible to be vaccinated or boosted: the vaccination rate in Ravensburg and the surrounding county was 61% and the administration of booster vaccinations just started for people 70 years and older, and high-risk groups (these rates were below those of the state and Germany as a whole, which were 62.8%71 and 66.8%72, respectively). In the hopes of motivating first and follow-up (booster) vaccination, Ravensburg recruited corporate donations, which it used to provide incentives of 20 Euros (approximately $20 USD) per vaccination—88% of Germany’s median hourly wage, and approximately double the minimum wage at the time73. We also included a community-level financial incentive in the letters, which we discuss in detail in the Supplemental Materials (SM, Section 1.2). This incentive meant that, if enough people attended the vaccination events, the financial incentive would be doubled to 40 Euros; this threshold was ultimately met and people were indeed paid 40 Euros for vaccinating. The city’s total cost for these financial incentives amounted to 6,640 Euros, roughly 9% of the total budget for the vaccination drive (the city incurred costs of approximately 42,000 Euros in printing and mailing costs, and 30,000 Euros to staff and secure the events). The vaccination events were advertised via letters, designed according to best practices74, signed by the mayor and two local health officials, and mailed to each resident. There were two versions of the letters: one which included a voucher which entitled the recipient to the financial incentive, and one which did not.

In addition to the incentive-based randomized evaluation, and consistent with recommendations in the literature75, we conducted a prediction exercise in which we asked experts to predict whether the financial incentive would successfully motivate vaccination in Ravensburg. Crucially, we targeted local policy makers with the authority to implement interventions in their localities, contacting 815 county heads and mayors or municipal government representatives of towns with 30,000 or more residents in Germany—these are policy experts who have the authority and capacity to implement policies similar to the Ravensburg vaccination campaign. In our prediction exercise, we explained that Ravensburg was testing the effect of financial incentives on vaccination in a randomized control trial (RCT), and stated the exact dates when the RCT was implemented. After providing some relevant background information including the rate of first, second, and booster vaccinations, we showed participants copies of both letters (see Fig. 1), and, for each, asked respondents to use their experience as local policy makers to predict “what percentage of adult residents who received a letter with [without] a voucher got vaccinated on one of the seven dates?”. We then asked the same question, but for booster shots (for the survey instrument in German and English, see SM, Section 2.5). Survey responses were anonymous, and participants were made aware of this.

Figure 1
figure 1

Randomized Control Trial of the Effect of a Financial Incentive on COVID-19 Vaccination. We sent all 41,548 adult residents of the town of Ravensburg, Germany an invitation to be vaccinated at one of seven vaccination events, randomly assigning 5,016 individuals to receive a 20 Euro financial incentive. (A) Timeline of the invitations and events. (B) Map of Ravensburg indicating the location of the vaccination events. (C) Reproductions of the invitation letters. Left: Letter sent to the intervention group, which offered the 20 Euro financial incentive. The text associated with the incentive is highlighted in yellow. Right: Letter sent to the control group, which is identical except that it omits text associated with the financial incentive. For a translation of these letters into English, see SM, Sect. 1.3.

Full size image

Results

Of the policy makers contacted, 10.8% (i.e., 88 in total) completed our prediction exercise (for more detail, see SM, Section 2.4). Respondents in our prediction study reported having authority over at least 5.7 million German residents. Overwhelmingly, policy experts believed that financial incentives would meaningfully increase vaccinations in Ravensburg. On average, they estimated that 27.5% of adults who were not offered the incentive would get vaccinated, whereas 42.8% who were offered the incentive would get vaccinated, a difference of 15.3 percentage points (SD = 15.7, t(86) = − 9.06, p < .001; for an alternative specification of this analysis, please see SM, Section 2.3 for details). Responses for booster shots displayed a similar pattern: respondents predicted that 32.3% of adults who were not offered the incentive would get boosted, whereas 48.2% of adults who were offered the incentive would get boosted, a difference of 15.9 percentage points (SD = 17.7, t(87) = − 8.38, p < .001). Only twelve (13.8%) respondents expected a null effect of financial incentives, and four (4.6%) a negative effect. For boosters, 13 (14.8%) expected a null effect, and two (2.3%) a negative effect.

To test whether financial incentives actually motivated vaccination, we worked with Ravensburg to implement a preregistered RCT testing the same letters that were presented in the survey (see Fig. 1). We randomly assigned half of Ravensburg’s 10,032 unique addresses to be eligible for a 20 Euro financial incentive. Then, in these addresses, we randomly selected a single adult in the household to receive the financial incentive (N = 5016; intervention group). All other adults received an identical letter without the incentive (N = 36,532; control group). Our power calculations suggested that we would be able to detect a difference in vaccination rates of 1.3 percentage points across the two groups with 80% power. See SM, Sections 1.2, 1.4, and 1.9 for further details on the experimental design and power calculations.

In stark contrast to policy makers’ predictions, we found no effect of financial incentives on vaccination (see Fig. 2). The proportion of Ravensburg residents assigned to the control group who were vaccinated at one of the seven vaccination events was 1.94%. In the intervention group, 1.73% of residents were vaccinated at the events. The difference in vaccination rates between individuals that received the intervention and individuals that did not was − 0.26 percentage points when controlling for the covariates we pre-registered (95% CI [− 0.553, 0.027], p = .076) (see Table S5 for alternative specifications that yield similar results). When we compare policy makers’ predictions to these results, we see that, for instance, over 86% of policy makers’ predictions were outside the 95% confidence interval of the randomized trial, and that over 95% of policy makers’ predictions were greater than the 75th percentile of estimated individual effects.

Figure 2
figure 2

Policy Makers’ Estimates of the Effects of Financial Incentives Compared to the Actual Effects of the Incentive. We asked German mayors and other policy makers to predict “what percentage of adult residents who received a letter with [without] a voucher got vaccinated on one of the seven dates?”. Here, we present their estimates of the increase in vaccination rates from our 20 Euro financial incentive [as a percentage point difference of their estimate of the vaccination rate in the control group]. The average estimated increase was 15.3 percentage points (119.1%). Estimates highlighted in orange fall within the 95% confidence interval of the actual increase in the randomized trial (− 0.26 percentage points, 95% CI [− 0.55, 0.03]). For comparison, the green density plot presents estimated individual treatment effects for all of Ravensburg’s 41,548 adult residents from a causal forest76. We find no evidence of treatment effect heterogeneity in our field experiment (an omnibus significance test fails to reject the null hypothesis of homogeneity at p value = 0.9376). We include the causal forest predictions to illustrate that even with some underlying treatment effect heterogeneity, policy makers overwhelmingly fail to predict the actual treatment effects.

Full size image

One possible explanation for the discrepancy between policy makers’ predictions and the results of the field trial is that in making their predictions, policy makers’ focus on a particular constituency that is not representative of the average. Thus, we investigated the possibility that incentives work for some residents but not others. We found no evidence that this was the case: the effect of the incentives does not significantly vary by residents’ age, gender, and nationality. We test for heterogeneity using an omnibus treatment effect heterogeneity significance test76; we fail to reject the null hypothesis of treatment effect homogeneity at p value = .93 (see SM, Section 1.5.3 for further details). We also consider, in particular, the possibility that the policy worked just for first-time vaccinations or for boosters, especially since individuals who had not yet obtained a first vaccination by this point might have been vaccine-hesitant or otherwise more difficult to motivate to obtain a vaccine. However, we found a null result for first-time vaccinations (-0.02 percentage and a small but statistically significant negative effect for boosters vaccinations (-0.30 percentage points, p value = .006; see SM, Table S2 for more detail on both).

Another possibility is that policy makers did not have a precise sense of the impacts of interventions on vaccination, because, for instance, they find it difficult to mentally simulate their effects or have poor intuitions for statistics77,78,79,80,81,82,83; our comprehensive review of the curricula of public management programs and law schools revealed little in terms of statistical training. Consistent with this, we find that policy makers did not accurately predict the ‘base’ response rate in the control group. This finding is shared with those of other, recent prediction studies69,83,84,85. In such cases, it can be informative to consider alternative outcome measures, such as relative rankings, which provide an indication of which intervention the respondent expects to work better. These are reported above. Another outcome that is often used in such cases69,83,84,85 is percentage changes—relative to the base rate—which provide some indication of whether the respondent thought the difference between the interventions would be, relatively speaking, big or small. We, therefore, repeat our main analysis using percentage changes and find similar results: most policy makers expected a big boost in the vaccination rate amongst those who received the incentive, not just in absolute terms, but also relative to their guess of the base rate (see SM, Section 1.5.2 for details).

Discussion

Overall, these findings show policy makers drastically over-predicted the effect that financial incentives had on contributions to a public good in a high-stakes setting, earmarking 6,640 Euros, or roughly 9% of the total vaccine drive budget to incentives that ultimately proved completely ineffective. To the extent that these results are indicative of a broader tendency by policy makers to rely on financial incentives when they are likely to fail, we emphasize the importance of educating policy makers on the potential pitfalls of employing financial incentives when attempting to promote contributions to public goods. Doing so should, in our view, be a major focus of policy communication.

While our results and discussion have emphasized that financial incentives can fail to promote contributions to public goods, they do not always fail43,86, nor are they always expected to. The literature points to two key conditions that reduce the likelihood financial incentives will work. The first is whether the behavior involves a contribution to the public good. Particularly for contributions to public goods, one should be cautious when considering employing financial incentives. If, however, the behavior in question involves a mostly personal decision (e.g., whether to lose weight 87, go to the gym 88, sleep more 89, or to be screened for cancer90) then there are fewer concerns with using financial incentives. This is because non-financial motivations such as reputation are particularly important in motivating contributions to public goods91,92,93, and these can be ‘crowded out’ by financial incentives27,28,29. This is admittedly complicated by the fact that the same behavior may simultaneously involve private benefits while also being a contribution to the public good, as is the case for vaccination59; when this is the case, one should try to anticipate people’s motivations, e.g., whether most people are primarily motivated to protect themselves from COVID-19, or whether they are not so worried about COVID-19 for themselves, and might instead be motivated by the protective effects of vaccination on others, and the potential to achieve herd immunity.

Second, the literature points to small financial incentives as particularly likely to fail or backfire. To quote the title of one well-known paper, it is advised to “pay enough or don’t pay at all” 28. This is because, to be effective, incentives must be sufficiently large to not only compensate individuals for the costs—in time, effort, discomfort, etc.—of contributing to the public good, but also to make up for the fact that these incentives undermine other, non-financial motives for contributing. Relatively large incentives have been shown to work where smaller incentives fail. For instance, in a field experiment, paying volunteers 1% of the revenue they raised led them to work less hard than when they received no payment, but paying 10% of revenue increased their effort28. Many of the financial incentives for adoption of electric vehicles and energy-efficient technologies in the U.S., Europe, and Asia are quite large (e.g., Germany offers a subsidy of up to 6,000 Euros per electric vehicle). Such large subsidies are likely large enough to be effective; in the past, incentives of this magnitude have been successful at motivating uptake of hybrid and electric vehicles94,95, as well as energy efficiency technologies such as heat pumps96.

Ethics

The Zeppelin University ethics committee reviewed and approved this study on 29th October 2022. All vaccination events were conducted in strict adherence to the vaccination procedures stipulated by the German public health authorities, which includes a mandatory consultation by a doctor prior to vaccination and the participant’s signed confirmation of consent to get vaccinated. The Zeppelin University ethics committee waived participants’ informed consent as it relates to being included in the study and the collection of data as to whether or not a participant was observed consenting to be vaccinated at the public event. The study was performed in accordance with relevant guidelines and regulations.