Introduction

Emerging technology, near-ubiquitous access to the internet, and ease of electronic communication makes it possible to contact and recruit participants over the internet, consent and collect data without in-person visits, and repurpose new sensor devices and smartphone technology for longitudinal research data collection. This so-called “eCohort” approach can be an extremely efficient epidemiologic approach that is attractive in an era of shrinking funds for traditional studies1. Even the well-endowed Precision Medicine Initiative will employ internet- and mobile phone application- (app-) based recruitment to recruit over a third of the planned 1 million person cohort2.

This approach, however, may yield substantial volunteer bias. Technology use is not uniform in the US3, and reliance on response to electronically-delivered invitations for study participation is likely to select for particular individuals with favorable impressions of the research establishment, strong altruistic motivation, and time to complete research activities. Prior internet-based surveys, for example, have reported over-representation of women, married, and well-educated individuals4. No prior analyses have reported on internet-based recruitment into a US-based eCohort in comparison with the US population.

The Health eHeart Study is a large eCohort study focused on cardiovascular health. Health eHeart invites any adult age 18 years or older with an email address to participate, recruits primarily over the internet via electronically-delivered invitations, collects surveys and patient-reported outcomes, and supports connection of a wide variety of consumer electronic devices and apps to the study so that the mHealth data they collect can be donated and delivered to the Health eHeart Study database. We compared participants in the Health eHeart Study to participants in the National Health And Nutrition Examination Survey (NHANES), which was designed to be representative of the US population, for the purpose of informing inferences made using Health eHeart Study analyses and for targeting recruitment to balance our study sample.

Methods

Health eHeart Study Sample

The Health eHeart Study is a cardiovascular focused eCohort, with enrollment, consent and participant occurring entirely using the internet. We analyzed cross-sectional baseline examination data and follow-up data from Bluetooth-enabled blood pressure measurement devices obtained between March 8, 2013 (enrollment initiation) and March 24, 2016 from consecutive participants enrolled in the Health eHeart Study. Participation in the Health eHeart Study is open to any person (world-wide) with a self-reported date of birth indicating age ≥18 years and an email address. Recruitment into the study occurred via several news media stories, social media and word-of-mouth in addition to being actively sought via email campaigns sent to persons associated with the American Heart Association (primarily via emails sent to participants in their Go Red for Women campaign5), to adult patients at the University of California, San Francisco (UCSF) Medical Center (primarily via unsolicited email invitation), through various other specific referral sources (we track referral source by provided a special URL to referring partners), and from unspecified sources (through our general URL).

After online registration (name, date of birth, email and password) and consent, participants were prompted to complete a series of online questionnaires pertaining to basic socio-demographics, family history, medical history, activity and well-being, habits and lifestyle, mental health, food and nutrition, and use of internet or social media. Participants were also invited to “connect” devices and apps (that they already own) from Fitbit, iHealth, Withings, Qardio, Alivecor, Azumio, Ginger.io and Google Fit and donate their data to the study. We limited our primary analysis to participants age ≥20 years (for comparability with NHANES) and with complete information and without “unknown” or “refused” responses on all baseline core survey instruments and survey items. For our secondary analysis, we additionally limited the sample to such participants who also contributed at least one blood pressure measurement via Bluetooth-enabled blood pressure measurement devices (iHealth, Withings and Qardio were all supported).

NHANES Sample

We used NHANES 2013–2014 to represent the US population and compare against participants in the Health eHeart Study. NHANES is a program of the National Center for Health Statistics (NCHS) that aims to investigate the health and nutritional status of the US population. Since 1999, the survey has been released every 2 years in a continuous fashion. These cross-sectional data are representative of the non-institutionalized US population. Every year, approximately 5,000 individuals of all ages are interviewed in their homes and complete the health examination component of the survey. NHANES follows a complex, multistage sampling procedure where the primary sampling units are counties or small groups of contiguous counties, within which city blocks are selected. Within these blocks, households are then randomly selected, and then individuals are drawn at random6. All NHANES protocols were approved by the NCHS Research Ethics Review Board7. In 2013–2014, 14,332 persons were selected for NHANES from 30 different study locations. Of those selected, 10,175 completed the interview. NHANES provides study weights that account for both non-response and deliberate oversampling of particular segments of the population.

Because various components of NHANES are only delivered to adults ≥20 years, we limited our analyses to these participants, leading to a sample size of 5,769. In order to maintain strict representativeness of the NHANES study sample ≥20 years and allow for direct comparisons with Health eHeart, we performed multiple imputation using chained equations to estimate missing and “unknown”/non-response values of all variables of interest (n = 13 variables) for all participants (n = 1,162 participants with at least one missing value)8, 9. We used 10-fold multiple imputation to generate imputed datasets, each with complete data on all 5,769 NHANES participants included in our sample. This 10-fold imputed dataset was used for all subsequent analyses.

Informed consent was obtained from all participants in both Health eHeart and NHANES. Our analysis of the Health eHeart Study data is covered by the UCSF Institutional Review Board (IRB); our analysis of the de-identified NHANES data is exempt from IRB Review. Methods were performed in accordance with the relevant guidelines and regulations.

Statistical Method

We first used descriptive statistics to compare the demographic characteristics, medical conditions, and lifestyle factors of the Health eHeart sample by recruitment source, using ANOVA and chi-square tests for between-source differences. Then, to identify factors independently associated with participation in Health eHeart, we used a case-control approach, using pooled data for the combined NHANES and Health eHeart samples to estimate logistic regression models for the “outcome” of inclusion in the Health eHeart Study sample. We first fit single-predictor models for age, sex, race, income, marriage status, educational level, hypertension, hyperlipidemia, diabetes, stroke, coronary heart disease, heart failure, heart attack, general health, smoking and sleeping duration, and then fit a final multivariable model for Health eHeart participation that included this entire set of predictors. Results are summarized as odd ratios (ORs) and 95% confidence intervals (CIs). We accounted for the complex stratified survey design of NHANES using the sampling weights, pseudo-strata, and primary sampling unit (PSU) variables provided by NHANES, with weights normalized to sum to the NHANES sample size. In the pooled analyses, Health eHeart participants were each given unit weight, and randomly assigned to two PSUs with a distinct pseudo-stratum. Multiple imputation of the NHANES data was implemented using the mi package in Stata Version 14.0, and the case-control models were estimated using the Stata svy package for complex survey data, which accommodates multiply-imputed data. Two-sided P values less than 0.05 were considered to be statistically significant.

Results

At the time of our data lock, 42,828 participants had registered for the Health eHeart Study by providing their name, email and date of birth. Of those, 33,236 (78% of registered participants) signed the online consent, 28,420 completed at least one survey, (86% of consented participants), and 12,280 were participants age ≥20 years with complete core baseline survey data and without “unknown” or “refused” responses to any survey item (Fig. 1). These participants constitute our primary analysis sample. Of these, 251 contributed at least one blood pressure measurement via Bluetooth-enabled blood pressure measurement device; these participants constitute our secondary analysis sample (Fig. 1). As described in our Methods, all NHANES participants age ≥20 years were included after multiple imputation successfully imputed missing/unknown/refused items for the 1,162 participants missing at least one required data element.

Figure 1
figure 1

Flowchart of self-selection process in Health eHeart Study as of 24/03/2016. *Health eHeart Study sample used in Tables 1 and 2. #Health eHeart Study sample subset used in Table 3.

Full size image

Baseline characteristics of Health eHeart Study participants differed by referral source (Table 1). For example, only 3% of participants referred by American Heart Association sources were male (consistent with the primary focus on the Go Red for Women program), compared with 37%-44% from other sources (p < 0.001). We also detected differences by recruitment source in age (more elderly participants from UCSF), race/ethnicity (more Black, non-Hispanic participants from AHA), income and education (higher in both from UCSF), general health (highest among participants from unspecified referral source), and sleep duration (lowest duration from AHA referrals, Table 1, all p-values < 0.001).

Table 1 Baseline characteristics of Health eHeart Study population with complete survey variables (N = 12280).
Full size table

Compared with all adults in the US, as represented by NHANES participants (applying sample weights), Health eHeart Study participants were more likely to be middle-aged: more likely to be female; less likely to be Black, Hispanic, or Asian versus White/non-Hispanic; more likely to be highly educated; more likely to have cardiovascular disease and risk factors but less likely to have diabetes; more likely to be in excellent general health; less likely to be current smokers; and more likely to report low sleep duration (Table 2). Associations with higher income and marital status did not persist in adjusted models. The higher prevalence of female participants in Health eHeart persisted even after excluding participants referred from the Go Red for Women program (ORadj = 1.6; 95% CI: 1.5–1.7). When we limited both the Health eHeart Study and NHANES population to participants with coronary heart disease (Health eHeart Study n = 1297; NHANES n = 293), characteristics of the sample were different (e.g., higher prevalence of cardiovascular risk factors), but predictors of participation in the Health eHeart Study were quite similar (Supplemental Table 1).

Table 2 Predictors of volunteering for the Health eHeart Study with reference to the National Health and Nutrition Examination Survey.
Full size table

Only a small subset of Health eHeart Study participants (n = 251, 2%) used a Bluetooth-enabled blood pressure measurement device, connected their device account to their Health eHeart Study account, and donated at least one blood pressure measurement to the study (median number of measurements per participant = 30; interquartile range 9–82). These highly self-selected participants showed mostly similar patterns of characteristics when compared with NHANES as the full Health eHeart sample, with some striking contrasts (Table 3). Instead of a large female preponderance in the full Health eHeart sample (73%, Table 2), Health eHeart participants contributing device-measured blood pressure values were less likely to be female than the US population (35%, Table 3). Persons with hypertension and coronary heart disease were even more heavily over-represented in this subset. Also, in this subsample in which moderately expensive purchases were required (blood pressure cuff and smartphone), higher income persisted as a strong predictor even after adjustment for education and other factors.

Table 3 Predictors of being in Health eHeart Study using sample with app based blood pressure against National Health and Nutrition Examination Survey.
Full size table

Discussion

The Health eHeart Study used efficient electronic methods for recruitment and took advantage of partner organizations willing to refer patients to our study website. This resulted in extremely efficient recruitment into the study. The sample of recruited individuals, however, differs from the US population in a variety of ways. Not only does the study over-represent persons with cardiovascular diseases and risk factors (as expected based on the study focus), but it also appears to over-represent females and non-Hispanic Whites, higher educational level, persons with more prevalent medical conditions but better self-reported general health, and fewer current smokers than would be expected if participation were proportional from all segments of the US population. Patterns were different (e.g., reversal of the female predominance) in the highly selected subset of the Health eHeart Study who contributed blood pressure measurements from a Bluetooth-enabled device.

Internet- and technology-enabled epidemiology can have major advantages in terms of efficiency. Consistent with the Health eHeart Study recruitment experience, one Danish internet-based study estimated more than 50% savings in their recruitment compared with a conventional approach ($160 vs. $322 per subject)10, and an internet-based clinical trial similarly reported that their web-based methods cost about half that of a hospital based approach11. Web-based questionnaires generally reduce cost substantially12, as do studies that invite participation by e-mail13. Aside from cost, web-based surveys can be more efficient in terms of response speed from respondents14, easier to adjust and modify by the research team15, quicker and less error-prone to process since data are entered electronically and coded automatically16, and easier to complete for disabled participants17.

Our results, in terms of which characteristics predicted participation, were similar in some ways, but different in others when compared with prior studies. As with Health eHeart, women and those with higher socioeconomic status appear to be consistently more likely to participate in epidemiologic studies18, especially in eCohorts14, 19, 20. For example, the NutriNet-Santé study in France found a much higher percentage of women compared with the corresponding national figures (78.0% vs 52.4%); and both the NutriNet-Santé study and the Australian Longitudinal Study on Women’s Health found higher participation rates in persons with higher educational levels. In contrast to the NutriNet-Santé study, however, which found higher proportions of married or partnered participants compared to their national data (70.8% vs. 62.0%), the unadjusted association we found in Health eHeart (69% married vs. 62% in NHANES) was not significant after adjusting for other selection factors. Also in contrast with Health eHeart, the Australian Longitudinal Study reported a higher percentage of study participants who rated their health in the online survey as fair or poor, and a higher percentage of study participants who were current smokers compared to their Census data. Their study, however, was limited to a very narrow demographic band (women age 18–23) so may not be comparable. We did not find another study describing self-selected participation in a study requiring use of sensor technology such as our analysis of participants in the Bluetooth-connected blood pressure cuff subsample.

Several factors likely contribute to the differences we observed between the Health eHeart Study and NHANES. First of all, NHANES makes special efforts to recruit underrepresented minorities. In fact, such individuals are oversampled in NHANES (though sample weights correct this factor so results are generalizable to the US population). No such efforts are made in the Health eHeart Study. Second, the Health eHeart Study’s focus naturally attracts participants at risk for heart disease, so the overrepresentation of people with cardiovascular diseases, such as coronary heart disease, stroke and heart failure, is to be expected. However, when we subset both samples to only participants with coronary heart disease, general selection patterns (e.g., for sex, race/ethnicity, education level and smoking) were consistent with those we found in the full Health eHeart sample. Clearly, the “digital divide” may explain differences in participation by education, and particularly also by income for the subset of Health eHeart using a Bluetooth-enabled blood pressure measurement device. As the digital divide diminishes21 and technology diffuses through all segments of society, this participation selection factor may ameliorate to some degree.

The Health eHeart Study is large and nationally-scoped and includes participants who complete extensive online surveys and device-associated data collection; and the NHANES study provides a near-ideal way to compare to the US population. However, our analysis has some limitations. Unlike NHANES, the Health eHeart Study does not limit participation to US residents. In contrast to Health eHeart, bias from self-selected non-participation in NHANES is minimized by post-stratification re-weighting based on the known demographic characteristics of the target sample; however, missing values arising from so-called item non-response in NHANES may not be missing at random (even conditional on other factors included in our imputation model), such that multiple imputation may be flawed. Finally, while both Health eHeart and NHANES collect many additional measurements, we were only able to evaluate measurements that were identically collected in both studies (or nearly so), preventing us from assessing the representativeness of Health eHeart on other potentially important dimensions.

Our results have some clear implications. First, given that Health eHeart recruitment is ongoing, this analysis provides guidance for how the study team can refocus recruitment efforts to target thus-far under-represented subgroups of the US population. It also represents a roadmap for prospective targeting efforts that can be used by the Precision Medicine Initiative as it begins internet-based direct volunteer recruitment later this year. While some self-selection characteristics may be expected from prior work on participation in research (e.g., under-representation of racial/ethnic minorities22), our findings regarding the technology product-dependent subsample (e.g., reversal of the sex ratio) are more surprising and potentially important to account for.

The other clear implication relates to inference: it is clear that simple descriptive analyses of the self-selected Health eHeart Study (e.g., % technology use) will often not yield results that are representative of the US population, either on average or within strata defined by other covariates (e.g., gender). However, it is important to note that estimates of average adjusted associations are likely robust to over- or under- (mis-) sampling even on the variables included in the association, provided that the mis-sampling occurs independently for each variable, and that the association is not modified by factors associated with self-selection. For example, we might obtain valid adjusted estimates of the marginal association of technology use with gender, despite oversampling of technology users and of women in the Health eHeart Study, provided that the oversampling on each factor is independent, and that the effect of technology use on gender does not vary, for example, by education. Note, even in the presence of effect modification, estimates within strata of the effect modifier should remain valid (e.g., there is internal validity). Furthermore, the effects of these various aspects of selection bias may potentially be minimized by re-weighting the Health eHeart sample (similar to the post-stratification weighting performed by NHANES), based on an extension of the multivariable logistic model developed here, with the result that all included covariates have weighted distributions very close to those in NHANES.

In conclusion, the Health eHeart Study demonstrates efficient internet-based recruitment, and allows remote data collection from online surveys and sensor/device technology. While it also clearly demonstrates that participants who volunteer for the study are different on average than the US population, this does not rule out its potential for providing valid estimates of adjusted associations. Whether this limitation can be overcome by future internet-based studies such as the planned Precision Medicine Initiative Cohort remains to be seen and will likely require more deliberate sampling, more costly targeted recruitment efforts, and application of post-recruitment standardization methods that correct for unrepresentative volunteer participation.