arising from H. Dai et al. Nature https://doi.org/10.1038/s41586-021-03843-2 (2021)
Simple messages derived from behavioural science have increased the uptake of the seasonal flu vaccine1,2,3,4,5, and early studies from the coronavirus disease 2019 (COVID-19) vaccine rollout have found that this strategy works for recently eligible older adults6 and healthcare workers7. However, it is unknown whether messaging on its own will encourage vaccination against COVID-19 among reluctant populations. In a randomized controlled trial (RCT) five to eight weeks after all adults in the study population (n = 142,428) were eligible for vaccination, we find that the best-performing nudge in previous studies2,6 and seven additional messages—stressing vaccines’ safety, efficacy, minimization of bad outcomes, accessibility (free, no identification required), protection of recipients’ families or widespread adoption—had no detectable effect among people who had not been vaccinated according to state records. This suggests an important boundary condition for nudges that is consistent with a recent result from late in the flu season8. Public health authorities should consider simple messages to encourage vaccination at key inflection points (for example, rollout of paediatric COVID-19 vaccines and full Food and Drug Administration approval for adults), but may see diminishing returns if using them to encourage the more hesitant.
After a strong initial push, the rate of COVID-19 vaccinations declined in the USA. Efforts to encourage vaccination have run the gamut from free doughnuts and marijuana to million-dollar lotteries and rare experiences such as driving at a superspeedway. Recently, Dai et al.6 reported promising results from an RCT evaluating another tactic—sending people short messages informed by behavioural science. The appeal of this approach is clear: it is cheap and minimally invasive. It is also well supported by convergent evidence: email messages increased COVID-19 vaccination appointment sign-ups among healthcare workers7, and SMS1,2,3, mail4 and email5 messages have increased seasonal flu vaccinations. Moreover, it has garnered considerable media attention9, with pieces advocating it in The Washington Post, Fortune, The Guardian, U.S. News & World Report and this journal10. Policymakers also took note, as several states implemented SMS campaigns9.
The Dai et al. study was conducted early in the COVID-19 vaccine rollout with recently eligible older adults. Although the results show the potential of nudges, it is unknown whether short messages can change motivations in the population that did not get vaccinated immediately. Indeed, Dai et. al. distinguish burden reduction (helping people to follow through on pre-existing intentions) from demand creation (changing intentions), and numerous reviews find limited and mixed evidence on what drives demand11,12,13,14.
To test whether these findings generalize beyond the initial stages of COVID-19 vaccination, we evaluated the efficacy of text messages sent by the Rhode Island Department of Health (RIDOH) to increase uptake in May and June 2021. The messages included the best-performing ‘ownership’ language from Dai et al. and a related flu study2. This language was supplemented in most conditions with information about safety, efficacy or access, for example. This study offers a strong test of direct messaging because recipients were unvaccinated five to eight weeks after becoming eligible. It is also a realistic test of what a government can and, more importantly, cannot do (for example, craft messages containing false claims and send excessive communications).
RIDOH maintains separate databases of individuals who have been vaccinated and tested for COVID-19. Our study population is the difference of these lists (tested but not yet vaccinated) matched through a series of quasi-identifiers and excluding people under 18 when tested (final n = 142,428; see Extended Data Fig. 1 for randomization scheme). The primary outcome was vaccination by the end of the measurement period: 25 May 2021 to 21 June 2021 (one week after the last day of messaging). At time of launch, all Rhode Islanders over 16 had been eligible to get vaccinated since 19 April 2021, and free, walk-in availability was widespread. The study was deemed exempt by RIDOH’s institutional review board. The sample size was dictated by policy goals, as all eligible individuals received messages. A previous study2 with more conditions and a sample size similar to our first iteration detected meaningful effects.
We created eight messages (Extended Data Table 1, Supplementary Information section 1) on the basis of behavioural science research on COVID-19 health behaviours and other vaccination contexts. All included ownership language (‘a vaccine is waiting for you’)2,6, a sentiment also appearing in a standalone condition. Other conditions further emphasized safety, access, minimal likelihood of bad outcomes, reduced risk to one’s family, social norms or some combination. All included a link to a state-run page providing vaccination options.
Individuals were assigned to receive one of eight messages or no message (control group). We randomly divided the population into three consecutive iterations of 40,000, 39,709 or 78,394, and then into roughly equal groups per day within those weeks. Within these strata, individuals were assigned to receive one of eight messages or no message (control group).
To maximize overall vaccinations, in iterations 2 and 3 we used an adaptive design such that the likelihood of assignment to any given message was determined by message performance in the previous iteration, with an 𝜀-bounded Thompson sampler adjusting the probability of assignment to condition over time (Supplementary Information section 2).
This study is a block-randomized experiment. All analyses (pre-registration: https://osf.io/pkhae) use either the Cochran–Mantel–Haenszel (CMH) test for 9 (condition) × 2 (outcome) × 13 (day) strata tables or a block-specific weighting, which provides unbiased estimates of intent-to-treat effects and randomization-justified variance calculations.
No SMS message did substantially better or worse than the control whether vaccination rates were measured one week after the messages were sent or at the end of the study period. Figure 1 illustrates the small size of these differences: the largest positive difference was 0.002 for the ‘preventing bad outcomes’ condition (that is, 2% of control and 2.2% of ‘preventing bad outcomes’ were vaccinated). Furthermore, we see no evidence of differences in vaccination rates (however measured) between the control and an aggregated ‘any message’ condition (estimated difference in proportions vaccinated −0.001, 95% confidence interval (CI) −0.004 to 0.001, CMH test, P = 0.27), nor between the arms taken all together (CMH test for 9 × 2 × 13 table, P = 0.12). For demographics, see Extended Data Table 2; for additional analyses see Supplementary Information sections 3–6.
We find no evidence that a strategy found effective early in the vaccine rollout6,7 increased COVID-19 vaccination among people who remained unvaccinated five or more weeks after becoming eligible. Public health officials—especially those avoiding or legally barred from mandates—may turn to this strategy to increase vaccination rates among the less enthusiastic but will probably see minimal impact. Dai et al. highlighted a promising, valuable and low-cost tool that can help to increase vaccinations; although our result does not contradict theirs, it does bound the reach of such approaches, a possibility one of their co-authors contemplated elsewhere10.
One limitation of our study is that the initial recipient list may contain some vaccinated people. Rhode Island residents could get tested at home but vaccinated out of state, and certain sites (for example, Veterans Affairs hospitals) do not need to report individual-level records to the state. Base rates may be inaccurate because of this and other sources of noise (Supplementary Information section 6), although this would not mask treatment effects, as message assignment was random. Another limitation is that race and ethnicity information is incomplete (Extended Data Table 2).
The study by Dai et al. differed from ours in several ways, including population age (mean age 70 versus 39), message source (recipients’ health network versus a state agency), sign-up ease (recipients being directed to a sign-up system versus a page providing vaccination options) and vaccination context (appointments were scarce in February 2021 but abundant by May 2021). Although these factors could account for the different outcomes, flu vaccine findings suggest otherwise: similar interventions have shown success among younger populations1, when issued by the state15, and using inconvenient media (mailed letters4), and flu vaccines are comparatively easy to procure. One feature that Dai et al. and many flu vaccine studies do share is that they were conducted early in their respective campaigns, whereas ours was not. Notably, a study of older adults found increased uptake of flu vaccines due to postcard messages in October but not November, December or January8. Taken together, this suggests that nudges help early in vaccination campaigns, but the efficacy decays. Another COVID-19 study recently made public provides further support16.
Although we cannot identify the mechanism(s) responsible for decaying efficacy of nudges, the possibilities include novelty effects early on, oversaturation effects later on, different types of hesitancy (logistical barriers versus objections to vaccines), and, especially for COVID-19, increasingly polarized discourse, divergent social norms and differential vaccine knowledge. Future work in public health communication should distinguish these mechanisms to better implement message campaigns. It may also be that short messages effectively encourage those somewhat inclined to vaccinate but cannot move those less inclined, regardless of timing, and with time, the former group shrinks. Despite our null result, nudges may serve foreseeable public health needs (for example, vaccinating children under 5 or promoting boosters) if timed correctly. Indeed, we know of no studies showing reduced vaccinations owing to message campaigns, so they carry little potential harm. However, their ability to move the more reluctant may be limited.
Further information on experimental design is available in the Nature Research Reporting Summary linked to this paper.
The data analysed in this paper were provided by the Rhode Island Department of Health and contains protected health information. To protect privacy, we cannot publicly post individual-level data. Qualified researchers with a valuable research question and relevant approvals including ethical approval can request access to the de-identified data about this trial from the corresponding author. A formal contract will be signed and an independent data protection agency should oversee the sharing process to ensure the safety of the data. Lightly aggregated data that support most of the analyses in this paper can be found at https://github.com/thepolicylab/COVID-SMSExperiment. Some demographic analyses rely on publicly available data from the United States Census Bureau, the United States Department of Housing and Urban Development, the Rhode Island Geographical Information System and the Rhode Island Board of Elections. Copies of these data and, where appropriate, the code that gathered the data are available at https://github.com/thepolicylab/COVID-SMSExperiment.
The code to replicate the analyses and figures in the paper and the Extended Data is available at https://github.com/thepolicylab/COVID-SMSExperiment.
Lee, W.-N. et al. Large-scale influenza vaccination promotion on a mobile app platform: a randomized controlled trial. Vaccine 38, 3508–3514 (2020).
Milkman, K. L. et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment. Proc. Natl Acad. Sci. USA 118, e2101165118 (2021).
Regan, A. K., Bloomfield, L., Peters, I. & Effler, P. V. Randomized controlled trial of text message reminders for increasing influenza vaccination. Ann. Fam. Med. 15, 507–514 (2017).
Yokum, D., Lauffenburger, J. C., Ghazinouri, R. & Choudhry, N. K. Letters designed with behavioural science increase influenza vaccination in Medicare beneficiaries. Nat. Hum. Behav. 2, 743–749 (2018).
Baskin, E. Increasing influenza vaccination rates via low cost messaging interventions. PLoS ONE 13, e0192594 (2018).
Dai, H. et al. Behavioral nudges increase COVID-19 vaccinations. Nature 597, 404–409 (2021).
Santos, H. C., Goren, A., Chabris, C. F. & Meyer, M. N. Effect of targeted behavioral science messages on COVID-19 vaccination registration among employees of a large health system: a randomized trial. JAMA Netw. Open 4, e2118702 (2021).
Chen, N., Trump, K.-S., Hall S. & Le Q. The effect of postcard reminders on vaccinations among the elderly: a block-randomized experiment. Behav. Public Policy https://doi.org/10.1017/bpp.2020.34 (2020).
Mascarenhas, L. This simple text message can encourage people to get vaccinated, researchers say. CNN https://www.cnn.com/2021/06/29/health/text-messages-vaccination-wellness/index.html (2021).
Kozlov, M. A simple text has the power to increase COVID vaccinations. Nature https://doi.org/10.1038/d41586-021-02108-2 (2021).
Brewer, N. T., Chapman, G. B., Rothman, A. J., Leask, J. & Kempe, A. Increasing vaccination: putting psychological science into action. Psychol. Sci. Public Interest 18, 149–207 (2017).
Dubé, E., Gagnon, D. & MacDonald, N. E. Strategies intended to address vaccine hesitancy: review of published reviews. Vaccine 33, 4191–4203 (2015).
Jarrett, C., Wilson, R., O’Leary, M., Eckersberger, E. & Larson, H. J. Strategies for addressing vaccine hesitancy: a systematic review. Vaccine 33, 4180–4190 (2015).
Lawes-Wickwar, S. et al. A rapid systematic review of public responses to health messages encouraging vaccination against infectious diseases in a pandemic or epidemic. Vaccines 9, 72 (2021).
Esteban-Vasallo, M. et al. Effect of mobile phone text messaging for improving the uptake of influenza vaccination in patients with rare diseases. Vaccine 37, 5257–5264 (2019).
Chang, T. et al. Financial incentives and other nudges do not increase COVID-19 vaccinations among the vaccine hesitant. NBER https://doi.org/10.3386/w29403 (2021).
Park, H. & Leemis, L. M. Ensemble confidence intervals for binomial proportions. Stat. Med. 38, 3460–3475 (2019).
Milkman, K. L. et al. A mega-study of text-message nudges encouraging patients to get vaccinated at their pharmacy. SSRN https://doi.org/10.2139/ssrn.3780356 (2021).
Hamel, L. et al. KFF COVID-19 vaccine monitor: April 2021. KFF http://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-april-2021 (2021).
Callaghan, T. et al. Correlates and disparities of COVID-19 vaccine hesitancy. SSRN https://doi.org/10.2139/ssrn.3667971 (2020).
Gibson, D. et al. COVID-19 vaccine acceptability and inequity in the United States: results from a nationally representative survey. Preprint at MedRxiv https://doi.org/10.1101/2021.01.29.21250784 (2021).
Huynh, H. Most (but not all) anti-vaccination attitudes increased alongside the occurrence and persistence of COVID-19: a snapshot of the trends in the U.S. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/yzu9r (2020).
Kaplan, R. M. & Milstein, A. Influence of a COVID-19 vaccine’s effectiveness and safety profile on vaccination acceptance. Proc. Natl Acad. Sci. USA 118, e2021726118 (2021).
Reiter, P. L., Pennell, M. L. & Katz, M. L. Acceptability of a COVID-19 vaccine among adults in the United States: how many people would get vaccinated? Vaccine 38, 6500–6507 (2020).
Taylor, S. et al. A proactive approach for managing COVID-19: the importance of understanding the motivational roots of vaccination hesitancy for SARS-CoV2. Front. Psychol. 11, 575950 (2020).
Karafillakis, E. & Larson, H. J. The benefit of the doubt or doubts over benefits? A systematic literature review of perceived risks of vaccines in European populations. Vaccine 35, 4840–4850 (2017).
Yaqub, O. et al. Attitudes to vaccination: a critical review. Soc. Sci. Med. 112, 1–11 (2014).
Hallsworth, M., Mirpuri, S. & Toth, C. Four messages that can increase uptake of the COVID-19 vaccines. Behavioural Insights Team https://www.bi.team/blogs/four-messages-that-can-increase-uptake-of-the-covid-19-vaccines/ (2021).
Hamel, L. et al. KFF COVID-19 vaccine monitor: March 2021. KFF https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-march-2021/ (2021).
Prochaska, J. O. & Velicer, W. F. The transtheoretical model of health behavior change. Am. J. Health Promot. 12, 38–48 (1997).
Amin, A. B. et al. Association of moral values with vaccine hesitancy. Nat. Hum. Behav 1, 873–880 (2017).
de Figueiredo, A. et al. Mapping global trends in vaccine confidence and investigating barriers to vaccine uptake: a large-scale retrospective temporal modelling study. Lancet 396, 898–908 (2020).
Kerr, J. R. et al. Correlates of intended COVID-19 vaccine acceptance across time and countries: results from a series of cross-sectional surveys. BMJ Open 11, e048025 (2021).
Larson, H. J. et al. Measuring trust in vaccination: a systematic review. Hum. Vac. Immunother. 14, 1599–1609 (2017).
Linholdt, M. F. et al. Public acceptance of COVID-19 vaccines: cross-national evidence on levels and individual-level predictors using observational data. BMJ Open 11, e048172 (2021).
Mesch, G. S. & Schwirian, K. P. Social and political determinants of vaccine hesitancy: lessons learned from the H1N1 pandemic of 2009–2010. Am. J. Infect. Control 43, 1161–1165 (2015).
Banker, S. & Park, J. Evaluating prosocial COVID-19 messaging frames: evidence from a field study on Facebook. Judgm. Decis. Mak. 15, 1037–1043 (2020).
Rabb, N. et al. No evidence that collective-good appeals best promote COVID-related health behaviors. Proc. Natl Acad. Sci. USA 118, e2100662118 (2021).
Falco, P. & Zaccagni, S. Promoting social distancing in a pandemic: beyond good intentions. PLoS ONE 16, e0260457 (2021).
Bicchieri, C. et al. In science we (should) trust: expectations and compliance across nine countries during the COVID-19 pandemic. PLoS ONE 16, e0252892 (2021).
Goldberg, M. H. et al. Social norms motivate COVID-19 preventive behaviors. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/9whp4 (2020).
Lees, J. et al. Intentions to comply with COVID-19 preventive behaviors are associated with personal beliefs, independent of perceived social norms. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/97jry (2020).
Bruine de Bruin, W. et al. Reports of social circles’ and own vaccination behavior: a national longitudinal survey. Health Psychol. 38, 975–983 (2019).
Kumar, S. et al. The social ecological model as a framework for determinants of 2009 H1N1 influenza vaccine uptake in the United States. Health Educ. Behav. 39, 229–243 (2012).
Sheeran, P. et al. The impact of changing attitudes, norms, and self-efficacy on health-related intentions and behavior: a meta-analysis. Health Psychol. 35, 1178–1188 (2016).
Sinclair, S. & Agerström, J. Do social norms influence young people’s willingness to take the COVID-19 vaccine? Health Commun. https://doi.org/10.1080/10410236.2021.1937832 (2021).
Moehring, A. et al. Surfacing norms to increase vaccine acceptance. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/srv6t (2021).
Schmidtke, K. A. et al. Randomised controlled trial of a theory-based intervention to prompt front-line staff to take up the seasonal influenza vaccine. BMJ Qual. Saf. 29, 189–197 (2020).
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data figures and tables
RIDOH maintains separate databases of (a) individuals who have been vaccinated and (b) individuals who have been tested for COVID-19. Vaccination data comes from medical providers and pharmacies receiving vaccines supplied by the State of Rhode Island, who are required to participate in the Rhode Island Child and Adult Immunization Registry (RICAIR) through electronic data reporting. Immunization records can be accessed by an individual’s medical provider or by authorized RIDOH users conducting public health surveillance activities including linking vaccination records with the state’s COVID-19 testing or case databases to verify information collected during case investigation. COVID-19 testing data (b) is reported to the state through the National Electronic Disease Surveillance System (NEDSS). Our study population is the difference of lists (a) and (b); the resulting database contained 162,504 unique entries. The study ended one day early after RIDOH received complaints about excessive communication. It is unclear how many complaints were received and how many were specifically about this study; other concurrent outreach efforts included SMS messages about COVID-19 testing and phone calls to older adults encouraging vaccination. Nevertheless, leadership halted all such communications out of concern that people would block crucial emergency messages. The final N for the study is 142,428. A small subset of the initial population (N ≈ 800) had chosen Spanish as their preferred language on testing sign-up forms. While we had initially planned to send this group messages translated into Spanish, an unresolved encoding problem prevented Spanish characters from displaying properly on some cell phones. The project team decided to reintroduce these individuals into the general study population for Iteration 3.
About this article
Cite this article
Rabb, N., Swindal, M., Glick, D. et al. Evidence from a statewide vaccination RCT shows the limits of nudges. Nature 604, E1–E7 (2022). https://doi.org/10.1038/s41586-022-04526-2