Chatbots have become an increasingly popular tool in the field of health services and communications. Despite chatbots’ significance amid the COVID-19 pandemic, few studies have performed a rigorous evaluation of the effectiveness of chatbots in improving vaccine confidence and acceptance. In Thailand, Hong Kong, and Singapore, from February 11th to June 30th, 2022, we conducted multisite randomised controlled trials (RCT) on 2,045 adult guardians of children and seniors who were unvaccinated or had delayed vaccinations. After a week of using COVID-19 vaccine chatbots, the differences in vaccine confidence and acceptance were compared between the intervention and control groups. Compared to non-users, fewer chatbot users reported decreased confidence in vaccine effectiveness in the Thailand child group [Intervention: 4.3 % vs. Control: 17%, P = 0.023]. However, more chatbot users reported decreased vaccine acceptance [26% vs. 12%, P = 0.028] in Hong Kong child group and decreased vaccine confidence in safety [29% vs. 10%, P = 0.041] in Singapore child group. There was no statistically significant change in vaccine confidence or acceptance in the Hong Kong senior group. Employing the RE-AIM framework, process evaluation indicated strong acceptance and implementation support for vaccine chatbots from stakeholders, with high levels of sustainability and scalability. This multisite, parallel RCT study on vaccine chatbots found mixed success in improving vaccine confidence and acceptance among unvaccinated Asian subpopulations. Further studies that link chatbot usage and real-world vaccine uptake are needed to augment evidence for employing vaccine chatbots to advance vaccine confidence and acceptance.
As of January 2022, >500 million confirmed cases of COVID-19 have been reported worldwide1. COVID-19 vaccines have been proven to be effective in lowering hospitalisation and mortality rates2, and are considered the most efficient tools for reducing severe disease and economic burden3,4. However, by January 28th, 2022, only 17 countries worldwide had attained a vaccination coverage of 70 percent4.
Many factors contribute to low COVID-19 vaccination coverage, including vaccine supply and distribution, access to healthcare facilities, and vaccine hesitancy. In particular, vaccine hesitancy, named one of the ten biggest threats to global health by the World Health Organization (WHO)5, refers to the “motivational state of being conflicted about, or opposed to, getting vaccinated.”6 Vaccine hesitancy is a major obstacle to increasing vaccine uptake and returning to normalcy, especially among seniors and parents of children7,8,9,10. As of January 28th, 2022, immediately preceding the study period, the percentage of elderly people aged 60 and above who had received at least one dose of vaccine was 75.1% in Thailand and 45% in Hong Kong; vaccination coverage of at least one dose among children was 34.5% in Hong Kong and 44% in Singapore, while Thailand had not yet begun vaccinating children11,12,13,14.
Factors affecting COVID-19 vaccine hesitancy included perceptions of vaccine importance, efficacy and safety, concerns about side effects, vaccine accessibility, and personal or religious beliefs15,16,17,18. Major concerns from seniors regarded the risk of serious adverse events following immunisation, such as deaths and complications due to old age and medical history19. For example, in Thailand, vaccines were provided by both the government and the private sector; however, results from cross-sectional surveys indicated that vaccination uptake among Thai people, especially among seniors, was low compared to other Southeast Asian countries20. Further, as children became newly eligible for vaccinations, there were concerns over the necessity or safety of child vaccinations. A large-scale study showed that Thai parents’ willingness to get their children vaccinated was the lowest among lower- and middle-income countries, due to uncertainty about COVID-19 vaccines’ effectiveness and safety21. Likewise, Hong Kong parents had doubts about vaccine safety for their children, and Hong Kong caregivers’ willingness to have their children vaccinated was lower compared to other countries22. In Singapore, some parents voiced concerns about the risk of contracting COVID-19 despite taking their children to get vaccinated23. Studies showed that low vaccine uptake in the region could be linked to high complacency from relatively low numbers of daily COVID-19 cases and low confidence in vaccine safety, coupled with a lack of knowledge about differences between each vaccine type10,20,21,22,23,24.
Compared to previous infectious disease outbreaks, a significant challenge during the COVID-19 pandemic was the overabundance of online and offline information, which included deliberate attempts to undermine and jeopardize global pandemic response measures by disseminating misinformation25. In response, the need for timely vaccination communication called for more effective use of social media and digital technologies such as machine learning, artificial intelligence, and conversation technology26. Among different digital interventions, chatbots have become an increasingly popular tool in health communication and services due to their ubiquitous access points and potential for massive information dissemination. There are many possible benefits to chatbots in an outbreak setting, including the relatively limited resources required to employ them after development, offering accessibility to clear, accurate, and timely information as outbreaks evolve, and freeing up the time of healthcare workers’ to address more complex issues27. However, as the use of chatbots in the context of health communication, especially in vaccine communication, is a novel approach, rigorous evaluations of their impact and potential use cases are very limited28.
In this study, we test the effectiveness of COVID-19 chatbots on people who were unvaccinated or had delayed vaccinations until the government vaccine mandates in Thailand, Hong Kong, and Singapore. Among many factors affecting vaccine attitudes and behaviours, we focus on participants’ changes in levels of vaccine confidence and acceptance, before and after using the COVID-19 chatbots. Given differentiating findings per target populations, we encourage future studies to further substantiate the evidence of vaccine chatbots’ effectiveness in advancing vaccine confidence and acceptance.
Recruitment and retention
From February 11th, 2022 to June 30th, 2022, 2045 participants were enrolled and randomly assigned to the control and intervention groups. After excluding 1,280 participants who were lost to follow-up, responses from 748 participants were included in the final analysis. (Fig. 1 and Supplementary Fig. 8).
Across all five study groups in the three locations, the baseline characteristics between the control and intervention groups were mostly comparable (Supplementary Tables 1 and 2). We noted an overrepresentation of minority subpopulation (i.e., Filipinos and Indonesians) in the Hong Kong and Singapore groups.
Findings from the randomised controlled trial
In the Thailand child group, results from a one-way comparison using Fisher’s exact test showed that participants in the intervention group were less likely to decrease in confidence regarding vaccine effectiveness: “vaccines are effective” [Intervention: 4.3% vs. Control: 17%, P = 0.023], “vaccines are effective regardless of manufacturers” [14% vs. 55%, P < 0.001], and “vaccines are effective against all variants” [8.7% vs. 48%, P < 0.001] compared to the control group (Table 1). In the Hong Kong child group, we did not detect any statistically significant findings regarding vaccine confidence. However, participants in the intervention group were more likely to report decrease in vaccine acceptance (“has your child received/do you intend for your child to receive a COVID-19 vaccine?”) [26% vs. 12%, P = 0.028] (Table 1). In the Singapore child group, we found more participants in the intervention group with decreased vaccine confidence regarding safety “vaccines are safe” [29% vs. 10%, P = 0.041]. Regarding secondary outcomes, the Fisher’s exact test showed that fewer participants in the Thailand intervention group reported decreased perceived need of COVID-19 vaccines (“my child does not need to be vaccinated” [13% vs. 32%, P = 0.006]) and perceived risk of COVID-19 disease (“COVID-19 is a serious disease”) [7.2% vs. 25%, P = 0.008] compared to the control group, after the intervention (Supplementary Table 3). Similarly, fewer participants in the Singapore child group reported decreased perceived risk of infection (“My child might get COVID-19”) [8.9% vs. 30%, P = 0.011], compared to the control group participants (Supplementary Table 3).
In the regression model, chatbot users from the Thailand child group were more likely after the intervention to have increased confidence in vaccines’ importance [OR = 2.40 (95% CI: 1.34–4.32)], effectiveness in reducing severe conditions [OR = 2.07 (1.23–3.48)], effectiveness regardless of manufacturers [OR = 4.04 (2.87–5.69)], and effectiveness against all variants [OR = 3.21 (2.22–4.66)] compared to the control group; however, the intervention group was less likely to have improved vaccine acceptance after the intervention (“has your child received/do you intend for your child to receive a COVID-19 vaccine?”) [OR = 0.66 (0.45–0.96)] compared to the control group (Fig. 2 and Supplementary Table 9).
In the Thailand senior group, a comparison using Fisher’s exact test showed that more participants in the intervention group reported a decrease in vaccine confidence related to effectiveness in reducing severe conditions [12% vs. 21%, P = 0.024] and misinformation awareness related to COVID-19 vaccines’ completion of clinical trials [3.4% vs.18%, P = 0.026] than the control group (Table 2 and Supplementary Table 4). In the Hong Kong senior group, Fisher’s exact test showed no significant findings on either the primary or secondary outcomes (Supplementary Table 4). Similar results were obtained from sensitivity analyses excluding Filipino and Indonesian subgroups (Supplementary Tables 5–8 and Supplementary Figs. 2–3).
Chatbot users from the Thailand senior group were less likely to have increased confidence in vaccine safety [OR = 0.63 (0.41–0.96)] and effectiveness regardless of manufacturers [OR = 0.57 (0.36–0.88)] after chatbot use compared to the control group (Fig. 3 and Supplementary Table 9). In the Hong Kong senior group, the intervention group was more likely to show improvements after the intervention in vaccine acceptance [OR = 3.26 (2.53–4.21)], and confidence in vaccine importance [OR = 1.87 (1.12–3.12)] and effectiveness [OR = 1.72 (1.11–2.64)] compared to the control group (Fig. 5 and Supplementary Table 10).
Factors of vaccine confidence and acceptance
Chatbots were found to be significantly more effective at improving vaccine confidence and acceptance among people who are minorities (i.e., non-Thai in Thailand and non-Chinese in Hong Kong and Singapore) and those who had lower education levels (i.e., below college level). Specifically, in the child groups, being a minority was associated with higher odds of an improved belief in vaccine effectiveness [OR: Thailand = 12.91 (3.56–46.78); Hong Kong = 2.73 (1.11–6.7)], effectiveness regardless of manufacturers [OR: Thailand = 2.99 (1.28–6.97)], and safety [OR: Hong Kong = 5.68 (2.01–16.1); Singapore = 10.46 (1.14–95.79)] (Figs. 2, 4, 6 and Supplementary Tables 9–11).
In the Hong Kong child group, parents with a college education or above were less likely to improve perceptions of vaccine importance [OR = 0.26 (0.14–0.48)], perceptions of vaccine effectiveness in reducing severe conditions [OR = 0.34 (0.18–0.64)], as well as vaccine acceptance [OR = 0.33 (0.22–0.48)], compared to parents with lower education levels in the intervention group (Fig. 4 and Supplementary Table 10). Likewise, in the Hong Kong senior group, respondents with a college or above education level showed lower odds of experiencing improved perceptions of vaccine importance [OR = 0.31 (0.18–0.55)], safety [OR = 0.18 (0.11–0.29)], and effectiveness [OR = 0.41 (0.26–0.67)] (Fig. 5 and Supplementary Table 10).
Respondents who were better at identifying COVID-19 vaccine misinformation in the baseline survey were more likely to have improved vaccine confidence and acceptance. In the Hong Kong child group, parents who correctly identified the misinformation: “COVID-19 vaccines cause infection” in the pre-intervention questionnaire were more likely to have improved beliefs in vaccine importance [OR = 2.24 (1.39–3.62)], safety [OR = 6.09 (3.51–10.55)], effectiveness [OR = 1.67 (1.09–2.56)], and acceptance [OR = 1.77 (1.32–2.36)] (Fig. 4, Supplementary Table 10). In the Singapore child group, parents who correctly identified the misinformation: “COVID-19 vaccines cause genetic change” were more likely to have improved beliefs in vaccine importance [OR = 7.28 (2.35–22.54)] and effectiveness [OR = 5.24 (1.81–15.16)] (Fig. 6 and Supplementary Table 11).
Respondents with higher risk perceptions were less likely to improve in their vaccine confidence and acceptance compared to those with lower risk perceptions (Figs. 2–6 and Supplementary Tables 9–11), meaning that perceived risks might have been the reason for their hesitation but chatbot use was not enough to sway their opinions or reduce their concerns about the vaccine. In the Thailand child group, high risk perception of the severity of COVID-19 disease was negatively associated with increases in beliefs of importance [OR = 0.62 (0.45–0.86)], vaccine safety [OR = 0.7 (0.53–0.92)], effectiveness in reducing severe conditions [OR = 0.71 (0.54–0.93)], effectiveness regardless of manufacturers [OR = 0.65 (0.55–0.78)], and effectiveness regardless of all variants [OR = 0.67 (0.55–0.82)] (Fig. 2 and Supplementary Table 9); in the Singapore child group, higher risk perception of COVID-19 infection was negatively associated with vaccine acceptance [OR = 0.42 (0.26–0.68)] (Fig. 6 and Supplementary Table 11).
Re-aim framework: evaluation of the intervention
Results from RE-AIM evaluations on the chatbot intervention are presented in the Supplementary Table 13. In brief, Hong Kong and Singapore’s D24H chatbot reached a total of 185 targeted users during the study. Thai ChatSure, which was further along in development compared with the D24H chatbot and had already been publicly accessible, reached 40,613 users since its launch in June 2021 and 1495 users during the study period. Both chatbots were supported by existing studies that demonstrated chatbots’ potential for combating infodemic via prompt information dissemination29 and improving vaccination intention through enhanced knowledge and self-efficacy30. Our formative data on chatbot users’ adaptions to the intervention suggested considerable interest in the public toward vaccine-related chatbots, as corroborated by the proportion of chatbot users who agreed or strongly agreed with the question: “I intend to use the chatbot again” [Thailand = 87%; Hong Kong = 73%; Singapore = 82%]. The Thai Ministry of Health brought ChatSure to the public specifically to combat COVID-19 vaccine-related misinformation; the D24H chatbot is currently being expanded to cover other vaccines, such as the HPV vaccine, so it may serve as a scalable intervention for existing vaccination campaigns to enhance online engagement with the goal of increasing vaccine confidence.
This study reports the impact of COVID-19 chatbots on vaccine confidence and acceptance of individuals who are unvaccinated or have delayed vaccinations in Thailand, Hong Kong, and Singapore. Most notably, in the Thai child group, we saw greater improvements in the chatbot users’ beliefs regarding vaccine effectiveness and debunking misinformation about COVID-19 vaccines and infertility. However, we also observed differentiating results based on the target population and region including backfire effects of chatbot use on both vaccine acceptance and confidence. The impact of chatbots was more effective in improving vaccine confidence and acceptance among unsure minorities and people with lower level of education, but less so for the highly educated population.
This study is a multisite, parallel RCT on multilingual vaccine chatbots. It was conducted in three Asian regions, one being upper-middle-income and two being high-income. As this study was conducted during the aggressive implementation of containment interventions such as social distancing rules and mandatory vaccine pass schemes by the governments in our study sites, we employed the RCT design to evaluate the impact of the chatbot intervention. Chatbot development and evaluation were constantly updated and tailored to changing local epidemic situations and vaccine policies and programmes (e.g., approval of the 5–11 age group vaccinations)22 to disseminate accurate information. Chatbots accommodated the most used languages (i.e., Thai in Thailand; English, Simplified, and Traditional Chinese in Hong Kong and Singapore) and one of the most widely used communication platforms in their respective regions (i.e., WhatsApp Messenger in Hong Kong and Singapore), and were developed by leading public health research organisations or government agencies in the study sites. The questionnaires were standardised across countries and contexts to compare outcome variables of interest. The chatbots’ high practicality, flexibility (i.e., the ability to adapt to different settings, such as HPV vaccination campaigns), and scalability demonstrated promising evidence for future research and applications.
Nevertheless, our study has several limitations. First, our sample sizes across all three regions were smaller than our target sample sizes. A combination of heightened risk perception owing to increased daily case counts and public health measures, such as school-based vaccine rollout and vaccine mandates, led to surges in vaccination uptake in our study locations, leaving only a small population who remained unvaccinated during our study period. Although we have found some statistically significant findings, our small sample sizes might lack sufficient power to detect the effect of the intervention in its entirety, e.g., dose-dependent effect (Supplementary Table 14). In order to recruit more participants, we relaxed our study criteria to include those who delayed their vaccination until the implementation of governmental vaccine mandates. Another possible reason for our small sample size is the high proportions of participants lost to follow-up, potentially due to our chatbot’s design31. For instance, our chatbot was not able to recognize users’ emotions and tailor phrase responses to questions. In addition, since participants recruited by Premise were more familiar with surveys related to market research rather than vaccines, their indifference to domains of chatbot contents might have led to user dissatisfaction and consequently a high drop-out rate. Second, our sample population was not representative of the populations in respective regions. To address this limitation, we applied population weightings in regression models based on respective regions’ census data to adjust for potential biases. Third, the chatbot employed in Hong Kong and Singapore only had COVID-19 vaccine-related content and was unable to answer general COVID-19 questions (i.e., COVID-19 home care instructions, daily COVID-19 cases). As a result, participants might have engaged less with the chatbot and rated the chatbots as less helpful than they would have otherwise. Fourth, our study might have social desirability bias since outcomes are self-reported amid active governmental encouragement and mandates on vaccination during the Omicron outbreak. Fifth, our study design incorporated responses of guardians to gauge vaccine confidence and acceptance of unvaccinated seniors due to lacking eligible senior participants in the existing panel. It is inferred that changes in the guardians’ vaccine confidence and acceptance will affect those of unvaccinated seniors, as family support is one of the most influential factors in senior vaccine hesitancy in this region32,33,34. However, this proxy approach might not directly reflect the changes in unvaccinated seniors’ vaccine confidence and acceptance. Further studies could advance the generalizability of chatbot interventions to target seniors, instead of their guardians, directly, and also investigate whether improve confidence in vaccine effectiveness could be translated into vaccination actions. Finally, our study focused on vaccine confidence and acceptance among numerous other factors that could drive vaccine hesitancy and deter children and seniors’ COVID-19 vaccine uptake. Studies suggest that differences in sociodemographic structures, health literacy, prevalence of chronic diseases, distributions of vaccine supplies, convenience to vaccinations, trust towards healthcare systems, governments, and vaccine developers impact vaccination coverages35,36,37,38,39,40. While vaccine confidence and acceptance are important attributes of vaccine uptake, our findings are not to be interpreted as the sole indicator of vaccination behaviours.
The first major finding of this study is that there was an increase in vaccine acceptance and confidence among some chatbot users across different study groups, adding to a growing body of evidence that well-designed and implemented chatbots can have a positive influence on health behaviours19,27,41,42,43,44,45,46,47. While there has been growing interest in chatbots across a range of public health areas19,27,41,42,43,48, very few studies have previously investigated the effectiveness of chatbots in promoting vaccine acceptance using RCTs43,49,50,51. For COVID-19 vaccination, our study lends weight to previous findings that interactive conversations between chatbots and users can contribute to increased vaccine confidence, as seen in the Thailand child group27,52. Additionally, since our chatbots were hosted on two major mobile messenger apps, WhatsApp and Facebook Messenger, our study also adds weight to previous evidence supporting the potential of mobile messenger applications in delivering vaccination knowledge, debunking vaccine-related misinformation, and providing vaccination guidance to unvaccinated individuals53,54. Our data showed that a majority of chatbot users were less likely to decrease in vaccine confidence and acceptance, compared with non-users; however, our study also found some apparent evidence of ‘backfire’ effects, specifically among some chatbot using parents in Hong Kong where we observed decreased vaccine acceptance with chatbot use. ‘Backfire effects’ are a controversial topic within the literature on digital health interventions—some research suggests that, in certain circumstances, pro-vaccine messaging delivered through social media can be counterproductive29. This might occur if, for example, messaging runs counter to the values of the target group or conflicts with individuals’ personal experiences—in our case, we noted chatbots worked better for improving vaccine confidence and uptake among minority subpopulations and users with lower level of education, but were less effective in swaying the mainstream population with higher education levels. Some studies have also raised concerns that repeating misinformation in order to correct or debunk it can have the perverse effect of increasing people’s familiarity with the misinformation, or may even spread misinformation to new audiences who had never been exposed to it before55. However, other studies have failed to replicate these findings56. In the case of our study, it is unclear why certain groups should have seen adverse outcomes on certain variables. Further in-depth investigations are needed. Conceivably, there may have been specific safety concerns or misinformation narratives that some had been less aware of prior to the study, and the process of engaging with the chatbot may have increased their familiarity with these topics or narratives.
Future chatbots may improve on their vaccine promotion and communication strategies as well as their message delivery, such as by using an emotion-based approach that can convey reassurances to chatbot users to ameliorate their doubts and fears28,57. An anthropomorphic chatbot that can share anecdotes may also have a positive impact; Loft et al. found that personal stories that go beyond facts and traditional sources of authority can be more persuasive in online communications campaigns30. Further advancement in AI technology, Natural Language Processing, and machine learning is immediately needed as the current chatbot operation relies heavily on human analysis to ensure response accuracy, especially in free text conversations. Further, chatbots should be supervised by trusted experts to ensure not only information accuracy, but data security and ethics compliance. Nevertheless, chatbots can be a useful component of a multi-pronged approach to health service delivery and communication, for example in combination with a webinar series or website with interactive features29,58,59. A more standardized assessment should be conducted to better analyse and improve chatbot’s effectiveness in handling users’ questions and influencing behaviours.
These study target populations who are unvaccinated or have delayed vaccination to identify viable strategies that could be applied in ongoing endeavours towards vaccine hesitancy alleviation22,23,60,61,62. The findings of this study present that, while chatbots are a potentially beneficial intervention that could provide insights to policymakers on the nature of vaccine concerns and inform strategies that can better address vaccine hesitancy, evidence on the effectiveness of chatbots on vaccine confidence and uptake is not conclusive across different populations and requires further assessment. We suggest interventions be interpreted and modified to address idiosyncratic local contexts in order to reach optimal results. Concurrently, it is important that future investigations on chatbot interventions to enhance vaccine confidence include underrepresented minority groups, such as migrant workers, to broaden the applicability and scalability of chatbots.
Ethics and informed consent
This study was approved by the Institutional Review Board of The University of Hong Kong/ Hospital Authority Hong Kong West Cluster (UW 21–659), National University of Singapore, Saw Swee Hock School of Public Health Departmental Ethics Review Committee (SSHSPH-158), and the Ethics Committee of the Institute for the Development of Human Research Protections of the Ministry of Public Health, Thailand (IHRP 1122–2654). The study protocol was registered at ClinicalTrials.gov (ID: NCT05424952) and made publicly available on July 22, 2022. All study participants electronically signed the consent form.
We conducted a multisite, parallel, randomised controlled trial (RCT) designed to evaluate the effectiveness of COVID-19 chatbots for improving COVID-19 vaccine confidence and acceptance in three Asian locations concurrently: Thailand, Hong Kong, and Singapore. Randomisation and group allocation were performed by a participant recruitment and market research company, Premise. We ran a pilot study from January 17th to February 11th and validated the survey tools using confirmatory factor analysis (Supplementary Method 1)63,64,65. From February 11th to June 30th, 2022, eligible users in the online panel within Premise’s mobile application were randomly invited to join the control or intervention group in our study with an allocation ratio of 1:1. The study was double-blinded—both participants and outcomes assessors were concealed from the intervention assignment. After signing the consent form, participants were asked to complete the pre- and post-intervention questionnaires and to use the chatbot if assigned to the intervention group. The formative, impact, and process assessment of the chatbot intervention was conducted following the RE-AIM framework66,67, and was quantitatively assessed based on its reach, efficacy, adoption, implementation, and maintenance.
Our study population included guardians of those who were unvaccinated or delayed their COVID-19 vaccinations until the government vaccine mandates (Supplementary Method 2 and Supplementary Fig. 1). Children and seniors had the lowest vaccination coverages in all study regions despite their COVID-19 disease vulnerability. Since guardians can make direct or indirect vaccination decision on behalf of children and seniors, we tested the effectiveness of chatbot in increasing guardians’ vaccine confidence and acceptance for their dependent family members. Taking an average of estimates from similar studies conducted in Japan and France53,68, we estimated an effect size of 15% and determined a sample size of 250 for each of the control and intervention group using power analysis. However, our sample sizes did not reach the target due to the surge in vaccine uptake at the time of participant recruitment following the emergence of Omicron and subsequent governmental vaccine mandates, school-based vaccine rollouts and increased risk perceptions of COVID-19 (Supplementary Fig. 1). In Thailand, the eligibility criteria included (1) adults with unvaccinated parents/grandparents aged 60 years or above, or (2) parents of unvaccinated children aged 5–11 years. The eligibility criteria in Hong Kong were slightly relaxed to include (1) adults whose parents/grandparents were 60 years old or above and had not been vaccinated against COVID-19 before the announcement of the mandatory Vaccine Pass on January 4, 202269, or (2) parents of unvaccinated children under 18 years old. In Singapore, the eligibility criteria included parents of unvaccinated children aged 5–11 years (Supplementary Method 2). In all three locations, participants were recruited by Premise, a participant recruitment and market research company70, via random sampling using existing online panels.
In Thailand, we adopted and updated ChatSure (Supplementary Method 3), an extant COVID-19 chatbot on Facebook Messenger, developed by the Ministry of Public Health, the Thai Health Promotion Foundation, Facebook Thailand, Hbot, the International Health Policy Program, and the National Vaccine Institute71. For Hong Kong and Singapore, we designed D24H vaccine chatbot and implemented it on the WhatsApp platform (Supplementary Method 4). Both free text mode and FAQ-style browsing mode were employed in the chatbots to improve vaccine confidence and acceptance by promptly providing accurate and consolidated vaccine-related information. Questions and comments from the participants are presented in supplementary information as word clouds (Supplementary Figs. 4–7).
The COVID-19 vaccine chatbot content covered seven major categories of commonly asked questions: (1) Importance/Necessity, (2) Safety, (3) Effectiveness, (4) How to get vaccinated, (5) Tips before vaccination, (6) Tips after vaccination, and (7) Debunking COVID-19 vaccine-related misinformation. The information and dialogues were tailored and timely modified to the local contexts and regulations. Participants were allowed to enquire using English or the mainstream local languages (i.e., Thai in Thailand; Traditional Chinese, Simplified Chinese and English in Hong Kong; and Simplified Chinese and English in Singapore).
For both the intervention and control groups, pre-intervention surveys were sent to participants to determine their demographics. All participants were then asked to answer questions regarding COVID-19 vaccine confidence, including perceived importance, effectiveness, and safety, vaccine acceptance, and COVID-19 vaccine-related misinformation. Study instruments are available in Supplementary Methods 5–7.
Next, participants assigned to the intervention group were directed to use the chatbot for up to 1 week through an invite-only link after completing the pre-intervention questionnaire. Participants in the intervention group were prompted by the chatbot to exchange at least ten messages regarding COVID-19 vaccines with the chatbot and submit screenshots of chatbot dialogues for validation before receiving a small reward for participation (<USD$15.00; Supplementary Table 12). Participants assigned to the control group were not given access to COVID-19 vaccine-related chatbots. After the intervention period, participants from both groups were asked to complete the same questionnaire about COVID-19 vaccines; chatbot users were asked to evaluate their usage experience.
The primary outcomes were changes in the levels of COVID-19 vaccine confidence and acceptance from the pre- and post-intervention questionnaires. Confidence was measured using the Vaccine Confidence Index TM (VCI)72, which has been used in the context of COVID-19 vaccinations73,74 and included perceptions of the importance, effectiveness, and safety of COVID-19 vaccines. VCI was recorded on a 5-point Likert scale (ranging from 5-strongly agree, 4-agree, 3-neither agree nor disagree, 2-disagree, to 1-strongly disagree) and the pre- and post-intervention differences were categorized into a 3-point scale of “improved”, “no change”, and “declined”. For example, if a participant answered “Agree” in the pre-intervention questionnaire and “Strongly Agree” in the post-intervention questionnaire, the participant would score 1-point. Positive, zero, or negative differences were categorized into “improved”, “no change”, or “decreased” outcomes, respectively. Similarly, vaccine acceptance was measured by the change in participants’ expressed willingness to vaccinate their senior parents/grandparents or children from the pre- and post-intervention questionnaires; the difference in pre- and post-intervention responses was categorized into the same 3-point scale. Secondary outcomes included differences in perceived risks and benefits of COVID-19 vaccines and participants’ awareness and knowledge of COVID-19 vaccine-related misinformation before and after the intervention. For the chatbot evaluation, outcomes were described based on the RE-AIM framework criteria (Supplementary Table 13), and conversation contents of D24H chatbot and ChatSure in local languages were presented in word clouds (Supplementary Figs. 4–7).
All participants who completed the assigned questionnaires and the intervention were analysed per protocol. The changes in outcome variables were measured on a 3-point scale (“improved”, “no change”, and “decreased”) and compared by both Fisher’s exact test and a proportional odds logistic regression model between the control and intervention groups to examine the effect of chatbot use on primary outcomes variables. We further employed proportional odds logistic regressions to investigate factors of primary outcome measures—vaccine confidence and acceptance where all participants’ data were weighted with sex and ethnicity using the latest local census data48,75,76.
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
Anonymized data and code used can be found at: https://github.com/lkwok/VCF_chatbot.
All codes used for the analyses and visualizations can be found at: https://github.com/lkwok/VCF_chatbot.
WHO. WHO Coronavirus (COVID-19) Dashboard. https://covid19.who.int/ (2022).
Worldometers. COVID-19 Coronavirus Pandemic. https://www.worldometers.info/coronavirus/ (2022).
Polack, F. P. et al. Safety and efficacy of the BNT162b2 mRNA covid-19 vaccine. N. Engl. J. Med. 383, 2603–2615 (2020).
Rubin, R. COVID-19 vaccines vs variants—determining how much immunity is enough. JAMA 325, 1241–1243 (2021).
WHO. Ten Threats to Global Health in 2019. https://www.who.int/news-room/spotlight/ten-threats-to-global-health-in-2019 (2020).
World Health Organization. L’Organisation mondiale de la Santé. Wkly Epidemiol. Rec. 95, 417–440 (2020).
Ma, A. & Parry, J. When Hong Kong’s “dynamic zero” covid-19 strategy met omicron, low vaccination rates sent deaths soaring. BMJ 377, o980 (2022).
Wang, G. et al. Determinants of COVID-19 vaccination status and hesitancy among older adults in China. Nat. Med. https://doi.org/10.1038/s41591-023-02241-7 (2023).
Griva, K. et al. Evaluating rates and determinants of COVID-19 vaccine hesitancy for adults and children in the Singapore population: strengthening our community’s resilience against threats from emerging infections (SOCRATEs) cohort. Vaccines (Basel) https://doi.org/10.3390/vaccines9121415 (2021).
Kitro, A. et al. Acceptance, attitude, and factors affecting the intention to accept COVID-19 vaccine among Thai people and expatriates living in Thailand. Vaccine 39, 7554–7561 (2021).
Singapore Ministry of Health. A Singapore Government Agency Website. https://www.moh.gov.sg/ (2022).
Thailand Ministry of Public Health. Covid-19 Vaccination Dashboard Thailand. https://dashboard-vaccine.moph.go.th/ (2022).
Singapore Ministry of Health. Vaccination Statistics. https://www.moh.gov.sg/covid-19/vaccination/statistics (2022).
Census and Statistics Department. Table 1B: Population by Sex and Age. https://www.censtatd.gov.hk/en/web_table.html?id=1B (2022).
Soares, P. et al. Factors associated with COVID-19 vaccine hesitancy. Vaccines https://doi.org/10.3390/vaccines9030300 (2021).
Sallam, M. COVID-19 vaccine hesitancy worldwide: a concise systematic review of vaccine acceptance rates. Vaccines (Basel) https://doi.org/10.3390/vaccines9020160 (2021).
Wang, Q., Yang, L., Jin, H. & Lin, L. Vaccination against COVID-19: a systematic review and meta-analysis of acceptability and its predictors. Prev. Med. 150, 106694 (2021).
Harapan, H. et al. Vaccine hesitancy among communities in ten countries in Asia, Africa, and South America during the COVID-19 pandemic. Pathog. Glob. Health 116, 236–243 (2022).
Amiri, P. & Karahanna, E. Chatbot use cases in the Covid-19 public health response. J. Am. Med. Inform. Assoc. 29, 1000–1010 (2022).
Mueangpoon, K. et al. Self-reported COVID-19 vaccine hesitancy and willingness to pay: a cross-sectional survey in Thailand. Vaccines (Basel) https://doi.org/10.3390/vaccines10040627 (2022).
Pengid, S. et al. Using Andersen’s model of health care utilization to assess factors associated with COVID-19 testing among adults in nine low-and middle-income countries: an online survey. BMC Health Serv. Res. 22, 265 (2022).
Lau, E. Y. H., Li, J. B. & Chan, D. K. C. Intention to vaccinate young children against COVID-19: a large-scale survey of Hong Kong parents. Hum. Vaccin Immunother. https://doi.org/10.1080/21645515.2022.2065838 (2022).
Zhong, Y. et al. Childhood vaccinations: Hidden impact of COVID-19 on children in Singapore. Vaccine 39, 780–785 (2021).
WHO. 10 Global Health Issues to Track in 2021. https://www.who.int/news-room/spotlight/10-global-health-issues-to-track-in-2021 (2020).
WHO. Managing the COVID-19 Infodemic: Promoting Healthy Behaviours and Mitigating the Harm from Misinformation and Disinformation. https://www.who.int/news/item/23-09-2020-managing-the-covid-19-infodemic-promoting-healthy-behaviours-and-mitigating-the-harm-from-misinformation-and-disinformation (2020).
Gesualdo, F., Bucci, L. M., Rizzo, C. & Tozzi, A. E. Digital tools, multidisciplinarity and innovation for communicating vaccine safety in the COVID-19 era. Hum. Vaccines Immunother. 18, 1865048 (2022).
Altay, S., Hacquin, A.-S., Chevallier, C. & Mercier, H. Information delivered by a chatbot has a positive impact on COVID-19 vaccines attitudes and intentions. J. Exp. Psychol.: Appl. https://doi.org/10.1037/xap0000400 (2021).
Caldarini, G., Jaf, S. & McGarry, K. A literature survey of recent advances in chatbots. Information 13, 41 (2022).
Attwell, K. & Freeman, M. I Immunise: an evaluation of a values-based campaign to change attitudes and beliefs. Vaccine 33, 6235–6240 (2015).
Loft, L. H., Pedersen, E. A., Jacobsen, S. U., Søborg, B. & Bigaard, J. Using Facebook to increase coverage of HPV vaccination among Danish girls: an assessment of a Danish social media campaign. Vaccine 38, 4901–4908 (2020).
Følstad A, Skjuve M. Chatbots for customer service: user experience and motivation. In Proc. 1st International Conference on Conversational User Interfaces (ACM, Dublin, Ireland, 2019).
Yuan, J. et al. Why do Chinese older adults in Hong Kong delay or refuse COVID-19 vaccination? A qualitative study based on Grounded Theory. J. Gerontol. B Psychol. Sci. Soc. Sci. https://doi.org/10.1093/geronb/gbac184 (2022).
Zhang, D. et al. Vaccine resistance and hesitancy among older adults who live alone or only with an older partner in community in the early stage of the fifth wave of COVID-19 in Hong Kong. Vaccines 10, 1118 (2022).
Tan, M., Straughan, P. T. & Cheong, G. Information trust and COVID-19 vaccine hesitancy amongst middle-aged and older adults in Singapore: a latent class analysis approach. Soc. Sci. Med. 296, 114767 (2022).
Wu, J. et al. COVID-19 vaccine hesitancy among Chinese population: a large-scale national study. Front. Immunol. https://doi.org/10.3389/fimmu.2021.781161 (2021).
Thanapluetiwong, S., Chansirikarnjana, S., Sriwannopas, O., Assavapokee, T. & Ittasakul, P. Factors associated with COVID-19 vaccine hesitancy in Thai seniors. Patient Prefer Adherence 15, 2389–2403 (2021).
Wang, C. et al. Vaccination willingness, vaccine hesitancy, and estimated coverage at the first round of COVID-19 vaccination in China: a national cross-sectional study. Vaccine 39, 2833–2842 (2021).
Long, S. et al. Changes of factors associated with vaccine hesitancy in Chinese residents: a qualitative study. Front. Pub. Health 10, 929407 (2022).
Kong, Y., Jiang, H., Liu, Z., Guo, Y. & Hu, D. The uptake and vaccination willingness of COVID-19 vaccine among Chinese residents: web-based online cross-sectional study. Vaccines 10, 90 (2022).
Li, T., Qiu, X., Gong, X., Zhan, R. & Zheng, X. The cross-sectional survey on COVID-19 vaccine hesitancy and it predictors among Chinese parents of 3–17 years aged children in Shenzhen City. Ann. Agric. Environ. Med. 29, 120–125 (2022).
Crutzen, R., Peters, G.-J. Y., Portugal, S. D., Fisser, E. M. & Grolleman, J. J. An artificially intelligent chat agent that answers adolescents’ questions related to sex, drugs, and alcohol: an exploratory study. J. Adolesc. Health 48, 514–519 (2011).
Moghadasi MN, Zhuang Y, & Gellban H. (eds) Robo: A Counselor Chatbot for Opioid Addicted Patients (ACM, 2020).
Skorburg, J. A. & Yam, J. Is there an App for that?: Ethical issues in the digital mental health response to COVID-19. AJOB Neurosci. 13, 177–190 (2021).
Fadhil, A. & Gabrielli, S. (eds) Addressing challenges in promoting healthy lifestyles: the al-chatbot approach. In Proc. 11th EAI International Conference on Pervasive Computing Technologies for Healthcare (ACM, 2017).
Dulin, P., Mertz, R., Edwards, A. & King, D. Contrasting a mobile App with a conversational Chatbot for reducing alcohol consumption: randomized controlled pilot trial. JMIR Form. Res. 6, e33037 (2022).
Poland, G. A., Jacobson, R. M. & Ovsyannikova, I. G. Trends affecting the future of vaccine development and delivery: The role of demographics, regulatory science, the anti-vaccine movement, and vaccinomics. Vaccine 27, 3240–3244 (2009).
Gabarron E, Larbi D, Denecke K, Årsand E. What do we know about the use of Chatbots for public health? Stud. Health Technol. Inform. 270, 796–800 (2020).
Statistics Singapore. Resident Population by Ethnic Group, Age Group and Sex Dashboard. https://www.singstat.gov.sg/find-data/search-by-theme/population/population-and-population-structure/visualising-data/resident-population-by-ethnic-group-age-group-and-sex-dashboard (2021).
Lee, W.-N. et al. Large-scale influenza vaccination promotion on a mobile app platform: a randomized controlled trial. Vaccine 38, 3508–3514 (2020).
van Baal, S., Le, S., Fatehi, F., García, A. & Hohwy, J. Effecting behaviour change using an artificial intelligence chatbot: a pilot randomised controlled study. PsyArXiv https://doi.org/10.31234/osf.io/2xuat (2022).
Lau, A. Y. et al. Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial. J. Am. Med. Inform. Assoc. 19, 719–727 (2012).
Hong, Y.-J., Piao, M., Kim, J. & Lee, J.-H. Development and evaluation of a child vaccination chatbot real-time consultation messenger service during the COVID-19 pandemic. Appl. Sci. https://doi.org/10.3390/app112412142 (2021).
Kobayashi, T. et al. 439. Corowa-kun: Impact of a COVID-19 vaccine information Chatbot on vaccine hesitancy, Japan 2021. Open Forum Infect. Dis. 8, S321–S322 (2021).
Pandit, J. A., Radin, J. M., Quer, G. & Topol, E. J. Smartphone apps in the COVID-19 pandemic. Nat. Biotechnol. https://doi.org/10.1038/s41587-022-01350-x (2022).
Pluviano, S., Watt, C. & Della Sala, S. Misinformation lingers in memory: Failure of three pro-vaccination strategies. PLoS One 12, e0181640 (2017).
Ecker, U. K. H., Lewandowsky, S. & Chadwick, M. Can corrections spread misinformation to new audiences? Testing for the elusive familiarity backfire effect. Cogn. Res. Princ. Implic. 5, 41 (2020).
Følstad, A. et al. Future directions for chatbot research: an interdisciplinary research agenda. Computing 103, 2915–2942 (2021).
Korda, H. & Itani, Z. Harnessing social media for health promotion and behavior change. Health Promot. Pract. 14, 15–23 (2013).
Pereira da Veiga, C. R., Semprebon, E., da Silva, J. L., Lins Ferreira, V. & Pereira da Veiga, C. Facebook HPV vaccine campaign: insights from Brazil. Hum. Vaccine Immunother. 16, 1824–1834 (2020).
Bono, S. A. et al. Adults’ acceptance of COVID-19 vaccine for children in selected lower- and middle-income countries. Vaccines (Basel) https://doi.org/10.3390/vaccines10010011 (2021).
Horiuchi, S. et al. Factors of parental COVID-19 vaccine hesitancy: a cross sectional study in Japan. PLoS One 16, e0261121 (2021).
Wan, X. et al. Willingness and influential factors of parents of 3–6-year-old children to vaccinate their children with the COVID-19 vaccine in China. Hum. Vaccines Immunother. 17, 3969–3974 (2021).
McDonald, R. P. & Ho, M. H. Principles and practice in reporting structural equation analyses. Psychol. Methods 7, 64–82 (2002).
Sun, J. Assessing goodness of fit in confirmatory factor analysis. Meas. Eval. Couns. Dev. 37, 240–256 (2005).
Rosseel, Y. lavaan: An R package for structural equation modeling. J. Stat. Softw. 48, 1–36 (2012).
Glasgow, R. E., Vogt, T. M. & Boles, S. M. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am. J. Public Health 89, 1322–1327 (1999).
Gaglio, B., Shoup, J. A. & Glasgow, R. E. The RE-AIM framework: a systematic review of use over time. Ame. J. Pub. Health 103, e38–e46 (2013).
Tran, V.-T., Sidorkiewicz, S., Péan, C. & Ravaud, P. Impact of an interactive web tool on patients’ intention to receive COVID-19 vaccination: a before-and-after impact study among patients with chronic conditions in France. BMC Med. Inform. Decis. Mak. 21, 228 (2021).
Education Bureau. Arrangements of Resumption of Face-to-Face Classes after Schools’ Easter Holidays. https://www.edb.gov.hk/attachment/en/sch-admin/admin/about-sch/diseases-prevention/edb_20220411_eng.pdf (2022).
Premise. Data for Every Decision™. https://www.premise.com (2022).
Bangkok Post. Ministry of Public Health and Thai Health Promotion Foundation Partner with Facebook. https://www.bangkokpost.com/thailand/pr/2130215/ (2021).
de Figueiredo, A., Simas, C., Karafillakis, E., Paterson, P. & Larson, H. J. Mapping global trends in vaccine confidence and investigating barriers to vaccine uptake: a large-scale retrospective temporal modelling study. The Lancet 396, 898–908 (2020).
Yadete, T. et al. Assessing acceptability of COVID-19 vaccine booster dose among adult americans: a cross-sectional study. Vaccines (Basel) https://doi.org/10.3390/vaccines9121424 (2021).
Achrekar, G. C. et al. Assessing COVID-19 booster hesitancy and its correlates: an early evidence from India. Vaccines 10, 1048 (2022).
The Government of the Hong Kong Special Administration Region. COVID-19 Vaccination Programme. https://www.covidvaccine.gov.hk/en/ (2021).
Government of Singapore. Singapore Census of Population 2020. https://www.singstat.gov.sg/publications/reference/cop2020/cop2020-sr1 (2021).
This work was supported by the Vaccine Confidence Fund (#VCF-020) and AIR@InnoHK administered by Innovation and Technology Commission. The Health Intervention and Technology Assessment Program (HITAP) was supported by a grant from the Access and Delivery Partnership (ADP), hosted by the United Nations Development Programme (UNDP) and funded by the Government of Japan. We thank Supatnuj Sorndamrih, Wattaporn Thanomsing, Nicha Krishnamra and Sindh R from the Thai Health Promotion Foundation (ThaiHealth) for their support on using the chatbot in Thailand; Dr. Yot Teerawattananon, Assoc. Prof. Wanrudee Isaranuwatchai, Dr. Pritaporn Kingkaew and Ms. Benjarin Santatiwongchai from the Health Intervention and Technology Assessment Program (HITAP) for their inputs on the study; Ruobing Mei from the Laboratory of Data Discovery for Health (D24H) for her input on the study.
There are no competing interests for any author. JAE’s institutional affiliation is provided for identification purpose only and does not constitute institutional endorsement. Any views and opinions expressed are personal and belong solely to the individual and do not represent any people, institutions, or organizations that the individual may be associated with in a personal or professional capacity unless explicitly stated.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Lee, K.Y., Dabak, S.V., Kong, V.H. et al. Effectiveness of chatbots on COVID vaccine confidence and acceptance in Thailand, Hong Kong, and Singapore. npj Digit. Med. 6, 96 (2023). https://doi.org/10.1038/s41746-023-00843-6