How to increase response rates to postal questionnaires

Abstract

Data sources

Fourteen electronic databases were searched, namely Cochrane Controlled Trials Register, CINAHL (Cumulative Index to Nursing & Allied Health Literature), ERIC (Education Resources Information Center) PsycLit, Dissertation Abstracts, Medline, Embase, Social Science Citation Index, Science Citation Index, Social Psychological Educational Criminological Trials Register, EconLit, Sociological Abstracts, Index to Scientific and Technical Proceedings, and National Research Register. Literature reviews and meta-analyses were inspected for eligible trials and two journals (Public Opinion Quarterly and American Journal of Epidemiology) were searched by hand.

Study selection

Studies included were randomised controlled trials of methods to increase response rates to postal questionnaires.

Data extraction and synthesis

Data describing trial participants, intervention, number of individuals randomised to intervention and comparison groups, and allocation concealment was extracted. For each strategy, pooled odds ratios and 95% confidence intervals in a random-effects model were estimated. Evidence for selection bias was assessed using Egger's weighted regression method, Begg's rank correlation test and funnel plots. Heterogeneity in trial odds ratios was assessed using a chi-square test at a 5% significance level and the degree of inconsistency between trial results was quantified using the I² statistic (The I² statistic measures the proportion of the variation across studies that is because of heterogeneity. It is calculated by dividing the heterogeneity chi-square minus the number of degrees of freedom by the heterogeneity chi-square, and then multiplying by 100.).

Results

From 372 eligible trials that were included, 98 different ways of increasing response rates to postal questionnaires were evaluated. For 62 of these strategies, the combined trials included over 1000 participants. There was substantial heterogeneity in the trial results from half of the strategies. Strategies found to response are shown in table 1, while three strategies were found to decrease response, questionnaire included questions of a sensitive nature (odds ratio 0.94, 95% CI 0.88–1.00; heterogeneity P =0.51; I² 0%), questionnaires began with the most general questions (odds ratio 0.80, 95% CI 0.67–0.96; heterogeneity Not calculated), Participants were offered the opportunity to opt out of the study (odds ratio 0.76, 95% CI 0.65–0.89; heterogeneity P =0.46, I2 0%).

Table 1 Strategies to increase questionnaire response

Conclusions

Health researchers using postal questionnaires can increase response rates using the strategies shown to be effective in this systematic review.

Commentary

Most of us will at some time have received a postal questionnaire, and many will have used one to collect data, because they are widely used for research purposes. In large, geographically dispersed populations they are often the only viable option. If people do not respond (nonresponders) the effective sample size is reduced and can introduce bias. This review looks at strategies to increase the response rates to postal questionnaires.

The ubiquity of the questionnaire and the large number of databases searched mean that several thousand potentially relevant reports were identified (26 937). Because of the large number, the accuracy and reliability of the screening was assessed: pairs of reviewers missed 4% of potentially relevant records (range, 0–6%), compared with individual reviewers who missed 22% (range, 3–55%). This supports the use of at least two reviewers for the initial screening of studies.

The comprehensive search strategy also means that the likelihood of including all the relevant trials in this review is high. There is evidence for this in the fact that the most comprehensive of the other reviews identified¹ only included 115 trials: 372 were selected for this review. Including all the relevant trials reduces random error in the meta-analysis and may also reduce bias.

Although the review clearly identifies a number of strategies that increase response rates, the authors raise a number of issues in their discussion which need to be taken into consideration. It is recognised that inadequate allocation concealment (a technique used to prevent selection bias by concealing the allocation sequence from the people assigning participants to intervention groups, until the moment of assignment), can bias the results of clinical trials² and information on allocation concealment was unavailable from most of the included studies.

The authors also found considerable heterogeneity between the trial results and consequently the pooled odds ratios presented should be treated with a degree of caution. As the authors point out in their discussion, however, even though there was statistical heterogeneity, the direction of the effects was similar. Although there is a lack of certainty about the size of the effect, therefore, we can say that there was a positive effect on response rates.

In summary, this review gives the clearest indication to date of a range of strategies to increase the response rate in questionnaires. Some of these will require additional resources (material or administration time) but some others can be implemented at little or no extra cost. The review provides a great deal of information that people considering using a questionnaire should read before embarking on one.

Practice point

A range of strategies to increase the response postal questionnaires have been quantified which should be taken into account when planning to use one.