Introduction

An increasing number of studies have demonstrated that internet-delivered, or online, health interventions for depression and anxiety are both efficacious,1, 2, 3, 4, 5 and can be delivered to a population on a large scale.1, 6, 7, 8, 9, 10 These interventions have the potential to overcome traditional access barriers as they can be available anytime to individuals at low cost, and without waiting lists that are common for traditional face-to-face interventions. The relative user anonymity of these interventions may also appeal to people who may not otherwise access help and thus may increase help seeking.11

The value of online interventions or internet interventions has been recognized at policy level internationally. For instance, guidelines from the United Kingdom’s National Health Service (NHS),12, 13 the Scottish Intercollegiate Guidelines Network (SIGN)14 and the Department of Veteran’s Affairs in the United States15 have endorsed online interventions as part of stepped care management of depression. It seems likely that other countries will follow in the near future as further studies become available.

However, a limitation of randomized controlled trials upon which evidence is based is the potential for sampling bias to arise from the inclusion of only a minority of the intended population participating. This bias may systematically limit the generalizability of the trial results and translational into clinical use. This raises uncertainty regarding the impact of the intervention on the more diverse or complex population seen in clinical practice.

The two main sources of sampling bias are the selection biases arising from rigorously defined inclusion and exclusion criteria and those from self-selection into the study. In terms of the former, Wisniewski et al.16 found that only 22.2% of people likely to be prescribed antidepressants would have met the inclusion criteria for an antidepressant phase III trial. More recently, Van der Lem et al.17 found that of patients deemed to have a major depressive disorder, suitable for antidepressant treatment in clinical practice, only 17–25% would have met inclusion criteria for efficacy trials. When explored further, those who did meet trial inclusion criteria were more likely to be younger, more educated, employed and earning a higher income, married and of Caucasian descent. Their illness was also likely to be less severe with a shorter average duration, fewer anxiety or atypical symptoms and a lower number of prior suicide attempts,16 indicating that selection may bias participants to those more likely to recover.

Participation may be prone to self-selection bias that can threaten both the internal18 and external validity of the study.19 Self-selection can generate a difficult to mitigate bias, as there is often little or no information upon those not volunteering to participate in the study. In observational studies, previous work has indicated that those who choose not to participate in research surveys are more likely to have poorer psychological20, 21 and physical health20, 22, 23, 24, 25, 26 than those who participate. Conversely, those who do participate are more likely to be younger,20, 24, 27 more educated,22, 24, 26 female,22, 23, 27 of higher socioeconomic status26, 27 and married.23, 26, 28, 29 Such demographic factors significantly affect depression treatment outcome: lower levels of baseline dysfunction and illness duration,30, 31, 32 being younger,32 having higher levels of education33, 34 and income,33, 34, 35 not living alone,30, 31, 34 having a ‘good’ employment status30, 34 and having higher expectations of improvement32 are all associated with better outcomes across treatment modalities.32

The aim of this study was to (1) identify the factors associated with self-selection of eligible trial participants recruited from a large community health cohort study into a randomized controlled trial of an online depression treatment trial and (2) evaluate whether these factors were associated with outcome and thus potentially a source of sampling bias.

Materials and methods

Cardiovascular Risk E-couch Depression Outcome (CREDO) is a randomized, double-blind, parallel, attention-controlled, internet-delivered trial targeting depressive symptoms in those with cardiovascular disease or risk factors. The trial protocol been described in detail elsewhere.36

Participants

Participants were recruited from the 45 and Up Study,37 a longitudinal study of health and ageing, which has been shown to be reasonably representative of the state of New South Wales, Australia, on a number of key indices.38 Potential participants from the 45 and Up Study population pool were selected for invitation to be assessed for trial eligibility if they were aged between 45 and 75 years, had provided a valid email address, had self-reported significant risk factors for, or a history of, cardiovascular disease and had screened positive for at least ‘moderate’ psychological distress on the Kessler 10 scale (K10)39, 40 during the 45 and Up Study baseline data collection. These 7999 potential participants identified were approached via email. If the individuals consented to be assessed for trial inclusion they were directed to the CREDO trial website where a more rigorous evaluation took place to ascertain full trial inclusion and exclusion criteria including the presence of current depressive symptomology. Reminder emails were sent to participants who did not respond to the initial emailed invitation. Where participants failed to open either of these email invitations, as indicated by an electronic notification, a written invitation to participate was posted to them. Those whose emails ‘bounced back’ indicating no current email address were excluded from this study as they were ineligible to participate in the trial. From the initial and reminder emailed invitations to participate, 2914 (41.1%) emails were unopened and had not bounced back. Postal addresses were obtained for these potential participants and written invitations for participation were posted. This process identified 7086 participants who received a verified invitation to participate. The flow diagram for participant selection and recruitment can be viewed in Figure 1. Of the 7086 potential participants, 1885 (26.6%) provided consent to be assessed for trial inclusion and exclusion criteria.

Figure 1
figure 1

Flow diagram demonstrating participant recruitment for the Cardiovascular Risk E-couch Depression Outcome (CREDO) trial.

Full size image

We compared those who consented with nonconsenters (n=5201) on a range of a priori demographic, health and behavioural indices based on the predictors found in the self-selection bias literature and variables thought to be predictive of help-seeking behaviours. These data were obtained from the 45 and Up Study baseline health survey from which these participants were selected.

Potential sampling bias measures in health survey

Data analyses were completed using PASW computer software package version 18.0 (SPSS Inc, Chicago, IL, USA). Several variables were recoded for ease of analysis. For all potential participants (n=7086), ‘Country of birth’ was recoded into ‘born in Australia’ versus ‘other’. Similarly, ‘Language spoken at home’ was recoded into ‘English’ versus ‘Other’. Level of education was recoded into a two-option variable of ‘School only’ (those who had completed ‘No school certificate or other qualification’, ‘School or intermediate certificate’ or ‘Higher school or leaving certificate’) versus any ‘Further education’. Marital status was combined to create a three-option variable of ‘currently in relationship’ (containing de facto or married), ‘previously in relationship’ (divorced, separated, or widowed) or single. Employment status was recoded to a variable with three options consisting of working (full-time or part-time paid employment), not working (disabled, retired or unemployed) or other (student or in unpaid work). Income, initially recorded in bands, was dichotomized at a median split of $70 000 per year. A further variable was created indicating whether or not the participant engaged in health protective behaviours, defined as self-reported screening for either prostate or breast cancer (yes/no).

The distributions of continuous variables were then checked graphically and for skewness, with those not normally distributed being converted to dichotomized categorical variables using a median split. These consisted of: perceived availability of social support, number of social contacts per week and number of social telephone calls per week.

Variables that were not recoded were: age, body mass index, total physical functioning score (SF-36),41 psychological distress (K10),39 number of times engaged in moderate and vigorous physical activity per week, having private health insurance (yes/no), requiring help for a disability (yes/no), being a carer for a sick or disabled person (yes/no), having a self-reported history of heart disease (yes/no), has had treatment for heart disease in the past month (yes/no), has a family history of heart disease (yes/no), ever being diagnosed with depression (yes/no), ever having been treated for depression in the past month (yes/no), ever having been diagnose with anxiety (yes/no), being treated for anxiety in the past month (yes/no), having a family history for depression (yes/no), is a current smoker (yes/no) and self-rated health (with 1 being poor and 5 being excellent). Any variable with >10% of the total data set missing was excluded from further analysis.

The univariate association of each variable with consent status was evaluated using independent sample t-test, Mann–Whitney U-test and χ2 test as appropriate. A multivariable binary logistic regression analysis was then completed using the enter method to assess the ability of these data to predict consent to participate in the research. Variables were included in the regression model on an empirical basis: those that had a statistical difference between the two groups of P=<0.10. Autocorrelations, considered to be a correlation of r>0.30, were assessed before modelling, and variables identified a priori as being more relevant (for example, depression versus anxiety) were entered into the model.

Trial outcome measure and sampling bias

The primary outcome measure of the study was depressive symptoms as assessed by the Primary Health Questionnaire (PHQ-9), a widely used 9-item, self-report tool designed for community samples. Items are scored on a 0–3 scale and are provided with a summary score ranging from 0 to 27. The PHQ-9 has shown to have sufficient sensitivity and specificity for major depressive disorder,42, 43 and an indicator of minimal clinically important change for individuals.44

In order to determine if any factors associated with consent to participation biased outcome, logistic regression models were completed using the data from the 487 participants who met inclusion and exclusion criteria and finally entered the trial and had outcome data available postintervention. The regression used the enter method to determine the association between the variables that were found to be associated with consenting to participate and the outcome—a clinically significant change in depressive symptoms as determined by a reduction in the PHQ score of ≥5. Sequential interactions between treatment arm and each of these variables were then examined.

Ethical approval

Written informed consent was obtained from all the participants and the ethics approval for the 45 and Up Study was provided by the University of New South Wales Human Research Ethics Committee. Ethics approval for the CREDO trial was obtained from the University of Sydney Human Research Ethics Committee.

Results

Factors associated with consent to be assessed for trial inclusion: univariate analyses

Compared with nonconsenters (n=5201), those who consented (n=1885) to be assessed for trial inclusion were significantly more likely to be younger, female, born in Australia and speak English at home (Tables 1 and 2). They were significantly more socioeconomically advantaged on a number of indices, being more likely to have completed some form of education after school and to have private health insurance (which is not compulsory in Australia but encouraged through taxation breaks). They also lived in more socioeconomically advantaged areas as indicated by the SEIFA (Socio-Economics Indexes for Areas) Index of relative socioeconomic advantage, which was derived from the participant’s postcode45 at initial data collection. They were less likely than nonconsenters to be a carer for someone with an illness or disability, and were more commonly engaging in ‘moderate’ physical activity per week. Consenters were also more likely to have been previously diagnosed with depression or anxiety, and to have been treated for either of these in the month before completing the 45 and Up Study baseline survey. There were however, no significant differences between consenters and nonconsenters in any self-reported cardiovascular risk factors: having received treatment for heart attack/angina, other heart disease, hypertension or high blood cholesterol in the past month; taking medications for heart disease, hypertension or high blood cholesterol in the past month; previous doctor’s diagnosis of heart disease, stroke or hypertension; previous doctor’s diagnosis of diabetes and report taking glucose-lowering therapy in the past month; two or more of the following risk factors: current smoker, obese, aged ≥65 years, family history of heart disease or stroke in two or more first-degree relatives.

Table 1 Association of categorical sociodemographic, clinical and behavioural variables with consent to participate in the CREDO trial
Full size table
Table 2 Association of continuous sociodemographic, clinical and behavioural variables with consent to participate in the CREDO trial
Full size table

There were no difference in marital status (married, single or separated) or any other measure of social support, overall physical functioning (SF-36 score), needing assistance for a disability (yes/no) or current level of psychological distress (K10 score).

Results: autocorrelations between significant variables related to consenting to be assessed for trial inclusion

Significant and expected autocorrelations were found between several variables. Where significant correlations were found, the variable that was considered a priori and/or more likely to induce self-selection bias based on the literature was retained for the final analyses. In the final multivariable model, those variables that were retained for further analysis consisted of language spoken at home (over country of birth), having a private health insurance (over having no private health insurance) and having ever been diagnosed with depression (over a family history of depression or anxiety or a personal history of anxiety).

Factors associated with consenting to assessment for trial inclusion: multivariate analysis

A total of 6396 cases were analysed in the final model (Table 3). A significance level of P<0.01 was selected in order to reduce the probability of Type I errors in the statistical analyses because of the large N of the study. Of the 13 variables included in the fully entered model, only 4 made such statistically significant contributions to the final model predicting consent status. These were female sex (odds ratio (OR) 1.11, 95% confidence interval (CI) 1.05–1.19), speaking English at home (OR 1.48, 95% CI 1.15–1.90), having completed further education (OR 1.67, 95% CI 1.46–1.92) and having a prior diagnosis of depression (OR 1.37, 95% CI 1.22–1.55).

Table 3 Multivariate logistic regression examining the factors that were associated with consenting to participate in the CREDO trial
Full size table

The fully entered model accounted for only 3.5% of the variance in consent status. The C-statistic for the model, which is not dependent upon the frequency of the outcome, was only a little greater than chance at 0.6.

A backwards binary logistic regression was modelled using the likelihood ratio method to test the sensitivity of the above model’s results and produced a similar result accounting for 3.7% of the variance in consent status with a C-statistic of 0.598.

Sampling bias with respect to depression outcome among trial completers

A binary logistic regression was completed to determine if the variables found to predict participation influenced the outcome of the trial in the 487 trial participants who had outcome data. None of the four variables included in the model were found to be significantly related to the outcome, a clinically significant improvement in depressive scores, over the trial (Table 4). There was also no interaction of these factors with outcome by treatment arm.

Table 4 Association of consent-related factors with clinically significant improvement in depression (reduction of ≥5 on the PHQ-9 postintervention)
Full size table

Discussion

Sampling, and specifically a self-selection, bias in the consent to take part in trials may limit the translation of trial results into real-world effects. This is the first study to have been able to evaluate the presence and impact of such a bias. Of the 35 potential sociodemographic, health and behavioural factors assessed, only 4 were independently associated with an increased likelihood of consenting to participate in an internet-delivered randomized controlled trial treating depression in people aged ≥45 years who had high risk of cardiovascular disease: female sex, speaking English, having engaged in further education after school and prior depression diagnosis were consistently associated in all models. The important issue is whether these factors that may influence consent bias the results of the trials by differentially influencing outcome. None of the factors identified in this study did so, either overall or were associated with the proportional effect of the intervention in the active arm.

These four factors identified are partly consistent with previous literature from observational studies that have commonly found that those who are more educated,22, 24 speak English, as opposed to another language in an English-predominant country,46, 47, 48 and are female22, 23, 27 are more likely to participate in research. However, unlike previous research, marital status23, 28, 29 and socioeconomic status27 were not related to consent to participate in this study. Also, contradictory to other findings,20, 21 greater psychological distress at baseline in the health survey did not reduce participation. In fact, a prior diagnosis of depression was associated with a higher likelihood of consenting.

Higher levels of education and speaking English in Australia may lead to individuals doing better in interventional studies because of their socioeconomic advantages,16 through lower rates of attrition from trials49 and/or being more likely to engage in help seeking outside of interventional studies.50 Any undetected effect of this may be that the trial results could overestimate the real-world effect in less educated or minority groups who do not speak English. Women are generally more likely to engage in both help seeking,51, 52, 53, 54 and research studies, as confirmed here. Most55, 56, 57 but not all50 research also tends to indicate that women respond more favourably to treatment for depression in trials. Similarly, in naturalistic studies, women were more likely to achieve better treatment outcomes than men regardless of therapy prescribed.58 Thus, the increased likelihood of women participating in this study may have led to potential overestimation of the benefits that men may receive from the online intervention.

Those consenting to be assessed for trial participation were more likely to have had a prior diagnosis of depression, but did not display a greater level of baseline psychological distress. This may indicate that consenters were more likely either to have a recurrent illness or one with other factors that had warranted prior medical help seeking. Although this may reflect the greater salience of the proposed intervention to this group, recurrent and or more severe episodes have been associated with a poorer outcome.59 However, in our study we found that there was no evidence that the participants with prior depression had poorer outcomes. As shown by Rogge et al.,60 those who think they may be most able to benefit may self-select into the study.

Limitations

A limitation of this study is that it recruited from a population who had already agreed to participate in a longitudinal study of health. As such, they may be more likely to engage in health care, research and/or interventions. The base sample had a high number of unopened, but valid, emails likely to represent less frequent users of the internet, particularly given that a significant number of this group was contactable via postal letter. It is probable that, despite a working email address, these people may be more difficult to engage in an online intervention, potentially needing more technical assistance and frequent reminders to access the programme. This could limit the cost efficiencies of such interventions in the wider population but does not necessarily indicate a bias.

The participation rate (27%) within the CREDO trial appears to be similar, if not better, than that expected from community-recruited internet trials.27, 61, 62 Response rates are likely to be different from trials recruiting directly from clinical centres or intervention waitlists, as community-based studies do not recruit participants actively engaging in help seeking. Therefore, our potential participants may not have previously recognized a need for intervention, or may be less motivated to engage in treatment. Given this, participants who consent to participate may be more motivated or may participate because of interest in research, rather than wanting help per se. However, there was no suggestion that consenters differed on other help-seeking variables such as previous cancer screening. The variables available and their coding were dependent upon the 45 and Up Study baseline data. The use of such categorical variables may have reduced the sensitivity of our analyses and we missed subtle effects. Furthermore, we could not assess potentially more pertinent variables such as psychological attitudes or values about research63, 64 that might have better determined consent status and potentially bias results. It is also important to note that the study population is an older group of individuals who, although self-identified as internet users, are clearly a generation who is not brought up with such technology. Therefore, findings might be limited in their ability to generalize to a younger population.

Finally, this study’s participants had consented to participate in an unguided internet-based intervention for depression. People have different motivations for participating in and persisting with internet interventions65, 66, 67 and the perception of a relationship is often important.66 Given that this was an unguided internet intervention where there was little contact with the research team, those people who believe the relationship to be important may not consent. Therefore, generalizing these results to guided interventions should be done so with caution and further research needs to be completed to determine the difference in consenters and nonconsenters in both guided and unguided interventions.

Future direction

Future research would benefit from prospectively exploring the reasons why people choose to participate in interventional trials and how, and whether, trial participants differ from those accessing open-access interventions. Understanding why people choose to participate in trials might help to increase recruitment and improve the translation of trial results within a health-care model. A difficult, but more fruitful, task will be understanding why nonparticipants did not choose to participate despite experiencing a level of distress. Such research may be best implemented using interviews of people who declined to participate, thereby providing a more in-depth understanding about these barriers and how nonconsenters may differ from those who consent. However, recruitment of this group is likely to be difficult and it may take some time to build up an adequate picture of these barriers to participation.

In summary, although there were some individual factors associated with consent status, the ability of these factors to predict consent was poor, accounting for <3.5% of the variance. Furthermore, and importantly for translating research into real-world outcomes, we were unable to show that any of the self-selection sampling differences between consenters and nonconsenters biased the trial results. This supports the evidence base for the more routine and wider translation of ehealth interventions into effective clinical practice and public health applications.