Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Risk of bias: why measure it, and how?

A Correction to this article was published on 19 January 2022

This article has been updated

Selection bias

High quality RCTs will randomize patients, and as important, conceal that randomization. Why? To limit selection bias. Selection bias is best described as a fundamental difference between the patients included in the treatment arms of a study due to the way in which patients were allocated to the treatment groups [1, 4]. To assess selection bias, one must consider both the random sequence generation and allocation concealment methods of the RCT [3].

Sequence generation refers to the method in which patients were randomized to the treatment groups. A truly random sequence for treatment allocation means that the baseline characteristics of both groups will be inherently balanced, but a bias in this allocation can result in systematic differences between the comparison groups [1, 3]. Some possible risks of bias due to sequence generation may exist due to non-, quasi-randomized methods of allocation. These methods may allow clinicians to choose which treatment patients’ will get within the study based on their expertize and prior experiences (i.e., A non-random factor). In addition, allocation concealment refers to the methods used to prevent anyone from being able to predict or deduce patient allocations [3]. Proper allocation concealment can prevent anyone in the research team from determining or predicting which patient’s have gotten which treatment within the trial. In summary, sequence generation refers to how patients are allocated to the comparison groups, and allocation concealment refers to how that allocation is kept secret from all relevant parties.

Performance bias

What about factors that can influence how a patient of clinician performs during the course of an RCT? Performance bias may be present if there are differences between the study groups as a result of systematic differences in performance outside of the study treatment received [1]. Risks of performance bias can result due to the masking (or blinding) methods of participants and personnel [1]. If masking is appropriately implemented, one can be assured that there was no additional and undue influence on the outcome of patients that occurred other than the assigned intervention [3, 5]. Many vital outcomes in ophthalmology, such as assessments of visual acuity, could be skewed if the patient or assessor are aware of the treatment allocation. When assessing performance bias, it is important to consider if the lack of masking could reasonably impact the outcomes being assessed [1].

Detection bias

The prior biases have focussed on the methods for randomizing and masking patients and clinicians, but what about biases in the way outcomes are measured? Detection bias can be described as the possibility for differences between the comparison groups with regard to how the outcomes are measured or assessed [1]. Detection bias also focuses on the concept of masking; however, it is the outcome assessor that should be masked in order to mitigate detection bias [3]. Masking of outcome assessors ensures that the methods in which an outcome is measured does not differ between patients allocated to the comparison groups—meaning that the outcome measurement is consistent for all participants in the study [1, 3].

Attrition bias

After patients have been included in an RCT, there is always a potential for them to withdrawal from the study before completing their follow-ups. Attrition bias can occur as a result of systematic cause of patient withdrawals in a study that disproportionately affect a certain subset of patients [1]. If a cause for withdrawal is present—or more predominant—in the comparison groups, the withdrawal imbalance could impact the results and conclusions drawn from the study [1, 3]. If a specific group of patients were more likely to withdrawal from the study within one of the comparison groups, the imbalance would have clear implications on results [1, 6].

Reporting bias

The final form of bias that any clinician should consider when reading an RCT is reporting bias. Reporting bias may occur when there are concerns with regard to the outcomes reported within the results of a study [1]. Selective outcome reporting is the primary concern in this form of bias, which refers to the reporting of some, but not all, measured outcomes within a study’s results [1, 3]. This commonly manifests as a study reporting on significant outcome findings, while omitting outcome findings that are not significant [1, 7]. Although this can be difficult to detect, it highlights the importance of a pre-defined study protocol that identifies all outcomes that will be assessed. You, as a reader, should actively seek the confirmation of that important step.

How should we interpret risk of bias assessments?

The next time you read an RCT, consider these risks of bias before making changes to your clinical practice. RCTs that are deemed to have a high risk of bias should be interpreted cautiously, as biases directly impacts the validity of the findings [1]. Empirical investigations have shown studies with high risk of bias may lead to an exaggeration of treatment effects within trials when compared to studies with a low risk of bias [8, 9]. It is common to assess the risks of bias in a study based solely on the reporting in the study manuscript, but poor reporting is not the same as biased conduct [3]. This is an important distinction to make with regard to risk of bias assessment that requires thoughtful consideration of the potential validity implications of study design decisions. You can refer to the Cochrane Handbook for Systematic Reviews of Interventions, “Chapter 8: Risk of Bias in Randomized Trials” for a comprehensive guide to risk of bias assessment for RCTs [1].

Summary

What is risk of bias

  • Clinicians read and interpret randomized controlled trials (RCTs) on a regular basis to inform their practice—but how can they be certain that the RCT is accurate and reliable? Not all RCTs are the same, and thus careful consideration needs to be taken when determining if RCT results are worthy of changing the way you manage future patients. The validity of an RCT can best be evaluated by understanding the possible risks of bias for that particular study.

  • Bias exists when a component of the design or execution of a study has systematic impacts on the results of the study that deviate from the truth. When such a bias exists, a study could result in over- or underestimation of the truth, compromising the validity of the study findings or results—even if all other facets of the study were appropriate [1,2,3].

  • Imagine, for example, providing navigation using a compass that was not accurately pointing “North”, but instead had a bias of pointing toward “North East”. Even if you provided thorough navigation steps to a fellow traveller, the end result will not be accurate due to the bias that existed from the inaccuracy of the compass. Similarly, an otherwise robust study that has some form of bias may provide results to clinicians and patients that are not accurate, despite the comprehensiveness of the investigation. With this in mind, it is important to understand the types of bias that may exist within RCTs, how to detect these potential biases, and how to interpret the results of a study in the context of such possible biases.

What types of bias exist, and how can we assess them

  • There are five main forms of bias that are important to consider for clinical trials: Selection Bias, Performance Bias, Detection Bias, Attrition Bias, and Reporting Bias (Table 1) [1, 3]. The Cochrane Risk of Bias assessment tool is the gold standard bias assessment tool for RCTs, as it assesses the risk of each of these forms of bias [3]. Below, we provide a summary of each form of bias and discuss how to minimize the risk of each bias during the design, conduct, analysis and reporting of trials.

    Table 1 Types of Bias Summary.

Change history

References

  1. The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions, Version 6.2. Chapter 8: Risk of Bias in Randomized Trials. Published 2021. https://training.cochrane.org/handbook/current.

  2. Tripepi G, Jager KJ, Dekker FW, Zoccali C. Selection bias and information bias in clinical research. Nephron Clin Pr. 2010;115:c94–99. https://doi.org/10.1159/000312871.

    Article  Google Scholar 

  3. Higgins JPT, Altman DG, Gøtzsche PC, The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343. https://doi.org/10.1136/bmj.d5928.

  4. Berger VW. Quantifying the magnitude of baseline covariate imbalances resulting from selection bias in randomized clinical trials. Biom J Biom Z. 2005;47:119–27. https://doi.org/10.1002/bimj.200410106. discussion 128–39.

    Article  Google Scholar 

  5. Boutron I, Estellat C, Guittet L, Dechartres A, Sackett DL, Hróbjartsson A, et al. Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: a systematic review. PLoS Med. 2006;3:e425. https://doi.org/10.1371/journal.pmed.0030425.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Fergusson D, Aaron SD, Guyatt G, Hébert P. Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis. BMJ. 2002;325:652–4. https://doi.org/10.1136/bmj.325.7365.652.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. https://doi.org/10.1136/bmj.c365.

    Article  PubMed  Google Scholar 

  8. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336:601–5. https://doi.org/10.1136/bmj.39465.451748.AD.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408–12. https://doi.org/10.1001/jama.273.5.408.

    CAS  Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Consortia

Contributions

MP was responsible for conception of idea, writing of paper and review of paper. VC was responsible for conception of idea, writing of paper and review of paper. MB was responsible for conception of idea, writing of paper and review of paper. PK was responsible for critical review and feedback on paper. LT was responsible for critical review and feedback on paper.

Corresponding author

Correspondence to Varun Chaudhary.

Ethics declarations

Competing interests

MP: Nothing to disclose. LT: Nothing to disclose. VC: Advisory Board Member: Alcon, Roche, Bayer, Novartis; Grants: Bayer, Novartis—unrelated to this study. PK: Consultant; Novartis, Bayer, Regeneron, Kanghong, Allergan, RegenxBio—unrelated to this study. MB: Research funds; Pendopharm, Bioventus, Acumed—unrelated to this study.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Members of the Retina Evidence Trials InterNational Alliance (R.E.T.I.N.A.) Study Group are listed below Author contributions.

The original version of this article was revised: In this article the middle initial in author name Sophie J. Bakri was missing.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Phillips, M.R., Kaiser, P., Thabane, L. et al. Risk of bias: why measure it, and how?. Eye 36, 346–348 (2022). https://doi.org/10.1038/s41433-021-01759-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41433-021-01759-9

Further reading

Search

Quick links