## Introduction

Seasonal influenza is associated with approximately 290,000–640,000 deaths worldwide each season1, and impacts approximately 21 million people in the United States annually, resulting in significant public health and economic burdens2. Preventing viral illnesses such as influenza is truly a global concern, given the potential for transmission in a modern, global culture, with this global risk and impact having been emphasized by the ongoing COVID-19 pandemic. People with diabetes (PWD), face increased risks from influenza, including poor glycemic control, pneumonia, premature death, acute cardiovascular complications, and hospitalizations3,4,5 which may result in a significant burden to the personal costs of healthcare for PWD. Vaccination remains the most effective primary prevention method against influenza, with effectiveness ranging from 29 to 48%6,7. Vaccination for influenza in PWD is effective in reducing the risk of hospitalizations and mortality3,8, as well as the overall cost of hospitalizations3,7. Influenza vaccination has also been shown to be safe for PWD and does not impact an individual’s ability to engage in daily activities in the days following vaccination5,7,9. However, vaccination rates remain suboptimal, consistently falling under the 70% vaccination rate goal set by national guidelines for all individuals in the United States10. In 2015 in the United States, 61.6% of adults with diabetes received an influenza vaccine11. During 2016–17, national rates of influenza vaccination were approximately 40% in adults without any high risk conditions, and 59.7% for adults with a variety of high risk conditions (including diabetes)12.

Therefore there is a need for effective and scalable solutions to increase influenza vaccination rates in PWD. While a number of randomized controlled trials (RCTs) have assessed the effectiveness of interventions for increasing influenza vaccination rates, many have focused on other age groups or populations13. One prospective digital interventional study demonstrated the potential effectiveness of general messaging and incentives via a health-related smartphone application (app) to increase vaccine uptake in a general Canadian population14, suggesting this kind of intervention could be effective in PWD. A large RCT using digital messaging was also effective in increasing vaccination rates in the general population of adults in the United States15. The use of health information technology (e.g., searching the internet for health information, emailing providers) and even simple electronic reminders delivered via digital patient portals have resulted in increased influenza vaccination rates, suggesting the potential of simple digital solutions16,17. One of the primary reasons PWD report not getting vaccinated is a belief that they are not in a high-risk group, and providing education on the increased risk of negative health outcomes following influenza infection has shown promise in increasing vaccination rates in other populations18. Additional reasons include fear of adverse reactions, difficulties with accessing the vaccine (e.g., time, health center access), or other beliefs surrounding the influenza vaccine (e.g., not effective, transmits the flu)19. Therefore, digital messaging that counters this lack of knowledge and barriers to vaccination could be effective for increasing uptake.

The aim of this study was to evaluate the effectiveness of a digitally administered intervention to increase influenza vaccine rates for PWD using a decentralized, blinded RCT. The primary endpoint was to examine the difference in self-reported influenza vaccination rates in 2 groups: PWD who received a digital intervention (PWD-I) and PWD who received no intervention (PWD-C). The following exploratory associations were also examined: (a) the impact of engagement with interventions on the influenza vaccination; (b) the impact of the timing of the intervention messages during influenza season on influenza vaccination status; (c) the reported level of influence on getting the influenza vaccine by each intervention message type within the PWD-I group; (d) the level of engagement with each intervention message within the PWD-I group; and (e) the impact of a healthcare worker’s recommendation on getting the influenza vaccine.

## Results

### Participant description

The study was launched in September of 2018 and the last participant completed the final survey in April of 2019, with the intent to capture outcomes during the 2018–2019 influenza season in the United States. A CONSORT diagram is included (Fig. 1) as a description of the larger investigation, including information on the 3 cohorts. For the PWD cohort, a total of 31,404 individuals were randomized, resulting in 15,702 in the PWD-I group and 15,702 in the PWD-C group. The PWD-I group consisted of 5158 individuals who completed the mid-study or final survey reporting their influenza vaccination status, and 5271 in the PWD-C group who reported their influenza vaccination status over the same interval. Approximately one-third of participants who were enrolled and randomized reported on the final endpoint. Figure 2 shows the flow of participants through the study, timing of the surveys, and timing of the intervention messages.

Descriptive characteristics for the sample are presented in Table 1. Demographic variables of age, race/ethnicity, and sex were not different between the two groups, indicating that randomization was successful. The participants were predominantly female, White, and in the middle age range and had high levels of income and education. Hypertension, depression, and high cholesterol were the most commonly reported comorbidities, with a small proportion reporting coronary artery disease. Participants included individuals from 48 of the United States, as well as the District of Columbia, United States Armed Forces, and Guam. A total of 101 individuals reported discrepant answers at the 3-month and 6-month questionnaires with regard to vaccination status. The effect size remained unchanged and statistically significant after re-running the primary outcome without these individuals.

### Primary endpoint

With regards to the primary outcome, 3310 (64.2%) of 5158 participants from PWD-I reported flu vaccination, compared with 3220 (61.1%) of 5271 among the PWD-C participants, with an absolute intervention difference of 3.1%. The number needed to treat or message for this effect is 33 people to result in 1 additional influenza vaccination. After adjusting for age, sex, race, and comorbidities, the intervention group was more likely (RR 1.05, 95% CI [1.02, 1.08], p = 0·0013) to get flu vaccination than the control group.

### Exploratory outcomes

Greater rates of engagement measured by larger number of interventions completed was associated with an increased vaccination rate for PWD-I (Fig. 3). Participants in the PWD-I arm who completed at least one digital intervention were more likely to report influenza vaccination (PWD-I: 66.6%; PWD-C: 61.1%, OR 1.27 95% CI [1.17, 1.38], p < 0·0001) than PWD-C. Participants in PWD-I who responded to at least 3 messages had an even greater odds of reporting influenza vaccinations (PWD-I: 69.0%; PWD-C: 61.1%, OR 1.42, 95% CI [1.30, 1.56], p < 0·0001) than PWD-C. Of note, within PWD-I, participants who did not complete any messaging interventions were less likely to report influenza vaccination (OR 0.61, 95% CI [0.52, 0.72]) than those who completed interventions.

The impact of timing of the intervention messages during influenza season on influenza vaccination rates within the PWD-I group, compared to the passage of time in the PWD-C group, is shown in the Kaplan–Meir curves in Fig. 4. Participants were asked to score the level of influence each message had on their decision to vaccinate (Fig. 5). Participants within PWD-I did not report differences in the effect of the messaging itself on the decision to vaccinate. Participants reported a mean score ranging between 2.75 to 2.93 for each of the messages. Figure 6 presents the response rates (i.e., number of responses per 100 persons) for each of the 6 different interventions. Each intervention message had a pattern of similar rates of clicks and completions with a much smaller rate of dismissals. Participants were most likely to respond to the first two messages with an overall decline in response rates in later months. Advice given by a healthcare worker to vaccinate had a strong effect on the decision to vaccinate among both control and intervention groups (OR 12.3, 95% CI [10.2, 15]). Participants who reported specific counseling to get influenza vaccination were slightly older than those who did not (44.9 vs 46.7 yrs., OR 1.01, 95% CI [1.01, 1.02], p value <0.001), more likely to have a self-reported cholesterol disorder (33.3 vs 45.1%, OR 1.18 95% CI [1.05, 1.32], p value <0.01).

## Discussion

### Measures

Participants were sent an online baseline questionnaire, a mid-study assessment at three months, and a final assessment at six months. Due to the study design, participants could complete any or all assessments; completion of mid-study and final assessment was not predicated on completion of the baseline assessment. The primary endpoint of influenza vaccination status was collected in the three- and/or six-month questionnaires. Questions on demographics, influenza vaccination status, and healthcare worker recommendation were asked of all participants. Participants in the PWD-I cohort were additionally asked about their perceptions of the interventions, and their engagement with intervention tasks was assessed.

A demographic questionnaire was administered at baseline, three and six months to all participants. Age, sex, and race/ethnicity were collected in all three questionnaires. Participants were able to “select all” for race/ethnicity and comorbidities. Information on medical comorbidities, income, and education was collected only in the three- and six-month questionnaires. For assessing influenza vaccination status, all participants were asked, “Did you get vaccinated against the flu (sometimes called getting a flu shot) this season? This year’s flu season began in September 2018,” and responded with Yes/No. Self-reported influenza vaccination status has been shown to have good specificity and positive predictive value23,24. Influenza vaccination status was collected both at mid-study and end of study to minimize recall bias. Participants were also asked for the approximate date on which they received their influenza vaccine. All participants were asked in the final survey “Have any of your healthcare providers recommended that you get a flu shot?” and asked to indicate Yes/No. The PWD-I cohort was also asked about their perceptions of the interventions and how much each message-type influenced whether they got their flu shot, rating each message from 1—“not influential at all” to 5—“very influential.” For each intervention, engagement statistics were tracked for the PWD-I participants who (a) clicked to open the digital intervention, (b) continued to complete the call to action or (c) explicitly dismissed it.

### Intervention content

Each of the six monthly messages was structured in two parts: educational content, and a call to action for the participant to complete (Table 2). Education and recommendations provided in the intervention were based upon data from the United States Centers for Disease Control and Prevention (CDC), World Health Organization (WHO), International Diabetes Federation (IDF), and the American Diabetes Association (ADA), thus reflecting the current standard of care recommendations for PWD. Content was reviewed by an expert advisory board and focus group of PWD. Messages were communicated via the Achievement app using month-specific messages, for example occurring around World Diabetes Day in November and National Heart Month in February. Calls to action would award points to the participant for completing actions such as needing to find the nearest clinic offering flu shots (CDC Flu Finder Widget) or planning prompts32, which have been shown to be effective at increasing influenza vaccination33. Incentives (via Achievement points) were provided upon the completion of the call to action as described.

### Sample size calculations and analytical plan

For sample size determination, we estimated a 2.7% increase in vaccination rate between the PWD-I and PWD-C groups. This value was selected to be consistent with prior research and clinically meaningful14,15. Power analysis indicated the need for an analysis set of 4043 individuals in each arm of the study (total N = 8086) to achieve 80% power to detect a 2.7% increase in vaccination rate with a type I error rate of 0.0534. To account for potential non-response to study surveys, we tagged 31,404 PWD for study inclusion, with 15,702 randomized to each of the two arms (~25% assumed response rate).

Logistic regression models were used to compare self-reported vaccination rates in both study arms (PWD-I versus PWD-C) controlling for demographics (i.e., age, sex, and race/ethnicity) and comorbid conditions, to calculate the relative risk and absolute risk change using intent to treat analyses. Per protocol analyses were used on all other analyses. Kaplan–Meier plots were constructed to evaluate self-reported time-to-vaccination for the two arms. Logistic regression models were constructed to examine the effect of the number of completed interventions and recommendations given by health care workers on vaccination rates, controlling for demographics and comorbid conditions. Significance is reported at a type I error rate of 0.05, with a type II error rate of 0.20.

### Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.