Abstract
Systematic reviews that include nonrandomized studies (NRS) face a number of logistical challenges. However, the greatest threat to the validity of such reviews arises from the differing susceptibility of randomized controlled trials (RCTs) and NRS to selection bias. Groups compared in NRS are unlikely to be balanced because of the reasons leading study participants to adopt different health behaviours or to be treated differentially. Researchers can try to minimize the susceptibility of NRS to selection bias both at the design stage, for example, by matching participants on key prognostic factors, and during data analysis, for example, by regression modelling. However, because of logistical difficulties in matching, imperfect knowledge about the relationships between prognostic factors and between prognostic factors and outcome, and measurement limitations, it is inevitable that estimates of effect size derived from NRS will be confounded to some extent. Researchers, reviewers and users of evidence alike need to be aware of the consequences of residual confounding. In poor quality RCTs, selection bias tends to favour the new treatment being evaluated. Selection bias need not necessarily lead to systematic bias in favour of one treatment but, even if it acts in an unpredictable way, it will still give rise to additional, nonstatistical uncertainty bias around the estimate of effect size. Systematic reviews of NRS studies run the risk of compounding these biases. Nutritional choices and uptake of health education about nutrition are very likely to be associated with potential confounding factors. Therefore, pooled estimates of the effects of nutritional exposures and their confidence intervals are likely to be misleading; reviewers need to take into account both systematic and uncertainty bias.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Alderson P, Green S & Higgins JPT (2004) In eds, JPT Higgins, S Green, Cochrane Library, Issue 1. Chichester, UK: John Wiley & Sons Ltd.
Barrett-Connor E (2004): Commentary: observation versus intervention—what's different? Int. J. Epidemiol. 33, 457–459.
Beral V, Banks E & Reeves G (2002): Evidence from randomised trials on the long-term effects of hormone replacement therapy. Lancet 360, 942–944.
Beral V, Banks E, Reeves G & Appleby P (1999): Use of HRT and the subsequent risk of cancer. J. Epidemiol. Biostat. 4, 191–210.
Black NA (1996): Why we need observational studies to evaluate the effectiveness of health care. BMJ 312, 1215–1218.
Caraballoso M, Sacristan M, Serra C & Bonfill X (2003): Drugs for preventing lung cancer in healthy people. The Cochrane Database of Systematic Reviews Issue 2, Art No.: CD002141.
D'Agostino Jr RB (1998): Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat. Med. 17, 2265–2281.
Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M & Altman DG, International Stroke Trial Collaborative Group, European Carotid Surgery Trial Collaborative Group (2003): Evaluating non-randomised intervention studies. Health Technol. Assess. 7 (27), iii–173.
Egger M, Smith GD & Schneider M (2001): Systematic reviews of observational studies (chapter 12). In eds, M Egger, GD Smith & DG Altman, Systematic Reviews in Health Care: Meta-Analysis in Context, pp 211–227. London: BMJ Books.
Greenland S (2004): Interval estimation by simulation as an alternative to and extension of confidence intervals. Int. J. Epidemiol. 33, 1389–1397.
Heart Protection Study Collaborative Group (2002): MRC/BHF Heart Protection Study of antioxidant vitamin supplementation in 20 536 high-risk individuals: a randomised placebo-controlled trial. Lancet 360, 23–33.
Hill AB (1965): The environment and disease: association or causation? Proc. R. Soc. Med. 58, 295–300.
Jha P, Flather M, Lonn E, Farkouh M & Yusuf S (1995): The antioxidant vitamins and cardiovascular disease. A critical review of epidemiologic and clinical trial data. Ann. Intern. Med. 123, 860–872.
Juni P, Altman DG & Egger M (2001): Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 323, 42–46.
Kuller LH (2004): Commentary: hazards of studying women: the oestrogen oestrogen/progesterone dilemma. Int. J. Epidemiol. 33, 459–460.
MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT & Black AM (2000): A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technol. Assess. 4 (34), 1–154.
Meade TW & Vickers MR (1999): HRT and cardiovascular disease. J. Epidemiol. Biostat. 4, 165–190.
Miettinen OS (1983): The need for randomization in the study of intended effects. Stat. Med. 2, 267–271.
Petitti D (2004): Commentary: hormone replacement therapy and coronary heart disease: four lessons. Int. J. Epidemiol. 33, 461–463.
Siegfried N, Muller M, Volmink J, Deeks J, Egger M, Low N, Weiss H, Walker S & Williamson P (2003): Male circumcision for prevention of heterosexual acquisition of HIV in men. The Cochrane Database of Systematic Reviews, Issue 3, Art. No.: CD003362.
Smith GD (2004): Classics in epidemiology: should they get it right? Int. J. Epidemiol. 33, 441–442.
Stampfer M (2004): Commentary: hormones and heart disease: do trials and observational studies address different questions? Int. J. Epidemiol. 33, 454–455.
Stampfer MJ & Colditz GA (1991): Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. Prev. Med. 20, 47–63.
Taggart DP, D'Amico R & Altman DG (2001): Effect of arterial revascularisation on survival: a systematic review of studies comparing bilateral and single internal mammary arteries. Lancet 358, 870–875.
Vandenbroucke JP (2004): Commentary: the HRT story: vindication of old epidemiological theory. Int. J. Epidemiol. 33, 456–457.
Acknowledgements
Dr Reeves is very grateful to colleagues in the Non-Randomized Studies Methods Group of Cochrane Collaboration for discussions, which underpin this paper, especially Jon Deeks and Ole Olsen.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Reeves, B., van Binsbergen, J. & van Weel, C. Systematic reviews incorporating evidence from nonrandomized study designs: reasons for caution when estimating health effects. Eur J Clin Nutr 59 (Suppl 1), S155–S161 (2005). https://doi.org/10.1038/sj.ejcn.1602190
Published:
Issue Date:
DOI: https://doi.org/10.1038/sj.ejcn.1602190
Keywords
This article is cited by
-
Evaluation of drug sensitivity, immunological characteristics, and prognosis in melanoma patients using an endoplasmic reticulum stress-associated signature based on bioinformatics and pan-cancer analysis
Journal of Molecular Medicine (2023)
-
Relationship between eating behaviors and physical activity of preschoolers and their peers: a systematic review
International Journal of Behavioral Nutrition and Physical Activity (2016)
-
Correlates, determinants, and effectiveness of childcare educators’ practices and behaviours on preschoolers’ physical activity and eating behaviours: a systematic review protocol
Systematic Reviews (2015)
-
Disagreement in primary study selection between systematic reviews on negative pressure wound therapy
BMC Medical Research Methodology (2008)
-
Limited evidence for effects of diet for type 2 diabetes from systematic reviews
European Journal of Clinical Nutrition (2007)