arising from H. Dai et al. Nature https://doi.org/10.1038/s41586-021-03843-2 (2021)

Simple messages derived from behavioural science have increased the uptake of the seasonal flu vaccine1,2,3,4,5, and early studies from the coronavirus disease 2019 (COVID-19) vaccine rollout have found that this strategy works for recently eligible older adults6 and healthcare workers7. However, it is unknown whether messaging on its own will encourage vaccination against COVID-19 among reluctant populations. In a randomized controlled trial (RCT) five to eight weeks after all adults in the study population (n = 142,428) were eligible for vaccination, we find that the best-performing nudge in previous studies2,6 and seven additional messages—stressing vaccines’ safety, efficacy, minimization of bad outcomes, accessibility (free, no identification required), protection of recipients’ families or widespread adoption—had no detectable effect among people who had not been vaccinated according to state records. This suggests an important boundary condition for nudges that is consistent with a recent result from late in the flu season8. Public health authorities should consider simple messages to encourage vaccination at key inflection points (for example, rollout of paediatric COVID-19 vaccines and full Food and Drug Administration approval for adults), but may see diminishing returns if using them to encourage the more hesitant.

After a strong initial push, the rate of COVID-19 vaccinations declined in the USA. Efforts to encourage vaccination have run the gamut from free doughnuts and marijuana to million-dollar lotteries and rare experiences such as driving at a superspeedway. Recently, Dai et al.6 reported promising results from an RCT evaluating another tactic—sending people short messages informed by behavioural science. The appeal of this approach is clear: it is cheap and minimally invasive. It is also well supported by convergent evidence: email messages increased COVID-19 vaccination appointment sign-ups among healthcare workers7, and SMS1,2,3, mail4 and email5 messages have increased seasonal flu vaccinations. Moreover, it has garnered considerable media attention9, with pieces advocating it in The Washington Post, Fortune, The Guardian, U.S. News & World Report and this journal10. Policymakers also took note, as several states implemented SMS campaigns9.

The Dai et al. study was conducted early in the COVID-19 vaccine rollout with recently eligible older adults. Although the results show the potential of nudges, it is unknown whether short messages can change motivations in the population that did not get vaccinated immediately. Indeed, Dai et. al. distinguish burden reduction (helping people to follow through on pre-existing intentions) from demand creation (changing intentions), and numerous reviews find limited and mixed evidence on what drives demand11,12,13,14.

To test whether these findings generalize beyond the initial stages of COVID-19 vaccination, we evaluated the efficacy of text messages sent by the Rhode Island Department of Health (RIDOH) to increase uptake in May and June 2021. The messages included the best-performing ‘ownership’ language from Dai et al. and a related flu study2. This language was supplemented in most conditions with information about safety, efficacy or access, for example. This study offers a strong test of direct messaging because recipients were unvaccinated five to eight weeks after becoming eligible. It is also a realistic test of what a government can and, more importantly, cannot do (for example, craft messages containing false claims and send excessive communications).

RIDOH maintains separate databases of individuals who have been vaccinated and tested for COVID-19. Our study population is the difference of these lists (tested but not yet vaccinated) matched through a series of quasi-identifiers and excluding people under 18 when tested (final n = 142,428; see Extended Data Fig. 1 for randomization scheme). The primary outcome was vaccination by the end of the measurement period: 25 May 2021 to 21 June 2021 (one week after the last day of messaging). At time of launch, all Rhode Islanders over 16 had been eligible to get vaccinated since 19 April 2021, and free, walk-in availability was widespread. The study was deemed exempt by RIDOH’s institutional review board. The sample size was dictated by policy goals, as all eligible individuals received messages. A previous study2 with more conditions and a sample size similar to our first iteration detected meaningful effects.

We created eight messages (Extended Data Table 1, Supplementary Information section 1) on the basis of behavioural science research on COVID-19 health behaviours and other vaccination contexts. All included ownership language (‘a vaccine is waiting for you’)2,6, a sentiment also appearing in a standalone condition. Other conditions further emphasized safety, access, minimal likelihood of bad outcomes, reduced risk to one’s family, social norms or some combination. All included a link to a state-run page providing vaccination options.

Individuals were assigned to receive one of eight messages or no message (control group). We randomly divided the population into three consecutive iterations of 40,000, 39,709 or 78,394, and then into roughly equal groups per day within those weeks. Within these strata, individuals were assigned to receive one of eight messages or no message (control group).

To maximize overall vaccinations, in iterations 2 and 3 we used an adaptive design such that the likelihood of assignment to any given message was determined by message performance in the previous iteration, with an 𝜀-bounded Thompson sampler adjusting the probability of assignment to condition over time (Supplementary Information section 2).

This study is a block-randomized experiment. All analyses (pre-registration: https://osf.io/pkhae) use either the Cochran–Mantel–Haenszel (CMH) test for 9 (condition) × 2 (outcome) × 13 (day) strata tables or a block-specific weighting, which provides unbiased estimates of intent-to-treat effects and randomization-justified variance calculations.

No SMS message did substantially better or worse than the control whether vaccination rates were measured one week after the messages were sent or at the end of the study period. Figure 1 illustrates the small size of these differences: the largest positive difference was 0.002 for the ‘preventing bad outcomes’ condition (that is, 2% of control and 2.2% of ‘preventing bad outcomes’ were vaccinated). Furthermore, we see no evidence of differences in vaccination rates (however measured) between the control and an aggregated ‘any message’ condition (estimated difference in proportions vaccinated −0.001, 95% confidence interval (CI) −0.004 to 0.001, CMH test, P = 0.27), nor between the arms taken all together (CMH test for 9 × 2 × 13 table, P = 0.12). For demographics, see Extended Data Table 2; for additional analyses see Supplementary Information sections 3–6.

Fig. 1: Average treatment effects for the eight experimental conditions overall and proportions vaccinated by day.
figure 1

Top left, the differences in the proportion vaccinated by the end of the study between each message condition and the control or ‘no message’ condition (2% of the control condition was vaccinated within the study period). Top right, the differences in the proportion vaccinated within a week of message sending (1% of the control condition was vaccinated within a week of message sending). The total control condition participation was 11,327. The total size of each arm is shown on the right. All point estimates with 95% confidence intervals (CIs). No adjustment was made for multiple testing as no test cast doubt on the null of no difference. Bottom, proportions vaccinated by 22 June 2021 in each message by the date messages were sent. The grey vertical line shows the proportion vaccinated in the control condition. The 95% confidence intervals for small proportions come from the binomial ensemble method of ref. 17.

Full size image

We find no evidence that a strategy found effective early in the vaccine rollout6,7 increased COVID-19 vaccination among people who remained unvaccinated five or more weeks after becoming eligible. Public health officials—especially those avoiding or legally barred from mandates—may turn to this strategy to increase vaccination rates among the less enthusiastic but will probably see minimal impact. Dai et al. highlighted a promising, valuable and low-cost tool that can help to increase vaccinations; although our result does not contradict theirs, it does bound the reach of such approaches, a possibility one of their co-authors contemplated elsewhere10.

One limitation of our study is that the initial recipient list may contain some vaccinated people. Rhode Island residents could get tested at home but vaccinated out of state, and certain sites (for example, Veterans Affairs hospitals) do not need to report individual-level records to the state. Base rates may be inaccurate because of this and other sources of noise (Supplementary Information section 6), although this would not mask treatment effects, as message assignment was random. Another limitation is that race and ethnicity information is incomplete (Extended Data Table 2).

The study by Dai et al. differed from ours in several ways, including population age (mean age 70 versus 39), message source (recipients’ health network versus a state agency), sign-up ease (recipients being directed to a sign-up system versus a page providing vaccination options) and vaccination context (appointments were scarce in February 2021 but abundant by May 2021). Although these factors could account for the different outcomes, flu vaccine findings suggest otherwise: similar interventions have shown success among younger populations1, when issued by the state15, and using inconvenient media (mailed letters4), and flu vaccines are comparatively easy to procure. One feature that Dai et al. and many flu vaccine studies do share is that they were conducted early in their respective campaigns, whereas ours was not. Notably, a study of older adults found increased uptake of flu vaccines due to postcard messages in October but not November, December or January8. Taken together, this suggests that nudges help early in vaccination campaigns, but the efficacy decays. Another COVID-19 study recently made public provides further support16.

Although we cannot identify the mechanism(s) responsible for decaying efficacy of nudges, the possibilities include novelty effects early on, oversaturation effects later on, different types of hesitancy (logistical barriers versus objections to vaccines), and, especially for COVID-19, increasingly polarized discourse, divergent social norms and differential vaccine knowledge. Future work in public health communication should distinguish these mechanisms to better implement message campaigns. It may also be that short messages effectively encourage those somewhat inclined to vaccinate but cannot move those less inclined, regardless of timing, and with time, the former group shrinks. Despite our null result, nudges may serve foreseeable public health needs (for example, vaccinating children under 5 or promoting boosters) if timed correctly. Indeed, we know of no studies showing reduced vaccinations owing to message campaigns, so they carry little potential harm. However, their ability to move the more reluctant may be limited.

Reporting summary

Further information on experimental design is available in the Nature Research Reporting Summary linked to this paper.