Authors of systematic reviews of treatment effectiveness often go to considerable effort to identify and then include every trial that has ever been conducted on the topic of interest, regardless of its susceptibility to bias. Bias is a concern because it often (but not always) favours the intervention [1]. Most authors of systematic reviews flag the potential for bias in included trials through the use of tools such as the Cochrane Risk of Bias tool [2] or the PEDro tool [3]. Nonetheless, often the results of trials, which are highly vulnerable to bias, are included in the final meta-analyses and conclusion. Without sufficient consideration of the potential for bias, this can overstate treatment effectiveness, thereby distorting the truth.

Some authors believe that trials susceptible to bias should be removed from systematic reviews altogether (or at least removed from the primary analysis) [1, 4]. This is often done on the basis of some arbitrary minimal score on a risk of bias tool. Alternatively, authors exclude trials that fail to adhere to one or more critical aspects of study design [1]. For example, trials are excluded if they do not blind assessors.

The Injuries Group of the Cochrane Collaboration (the group responsible for most Cochrane Systematic Reviews involving people with spinal cord injuries) have taken an alternate and very bold approach to the problem. They restrict their systematic reviews on treatment effectiveness to trials that have been prospectively registered (unless the trial was published before 2010) [5]. They argue that trial registration is a good proxy for trial quality and indicates lesser vulnerability to bias because it ensures that researchers are upfront about their intentions and cannot easily manipulate outcomes, protocols, analyses or anything else to ensure a “positive” trial result when submitting a manuscript. The reader will note that this stance fails to acknowledge that a poorly designed trial can still be registered. Nonetheless, trial registration does allow authors of systematic reviews to identify any anomalies between the final paper and the initial registration.

Interestingly, the Injuries Group also insist that trial authors provide their original (“raw”) data for checking if there are any unexplained anomalies identified by the systematic reviewers [5]. If the authors are unwilling or unable to do this, the trial is not included in the systematic review. These two policies of the Cochrane Injuries Group have been met with heated debate both within the Cochrane Collaboration and without, for obvious reasons, and we are not suggesting that Spinal Cord adopts either (although we are insisting that trials be registered [6] and we are encouraging authors to share their data [7, 8]).

There are dangers with any approach to excluding trials from systematic reviews (or meta-analyses) because there is no clear-cut distinction between high- and low-quality trials, and we can never know for sure whether a trial is biased: we can only know that it is susceptible to bias. In addition, adhering to all critical aspects of design is not always feasible, making some trials inherently more vulnerable to bias. For example, it is not always possible to blind participants or assessors in rehabilitation research that utilises outcomes that rely on self-report, because the unblinded participant is also the assessor. The use of any criteria to exclude trials from systematic reviews (or analyses) will always be a simplistic approach to a complex issue [9].

We do not have a solution other than encouraging researchers to be more aware of all the possible sources of bias, and to minimise biases when devising their trials, through good design. We also urge systematic reviewers to be more cognisant of the risks associated with meticulously including every trial ever published on a topic, in the belief that this approach gets us closer to the truth than does simply ignoring trials at high risk of bias. The end result of the former approach is invariably that treatments are recommended based on findings that may be highly biased. (A secondary consequence is that trials that are highly vulnerable to bias and the journals that publish them receive citations, rewarding poor science). Of course, if we exclude trials that are highly vulnerable to bias, we will have fewer trials in our systematic reviews. But perhaps this approach will get us closer to the truth about the effectiveness of an intervention and will give us more confidence in the conclusions of our systematic reviews. When there is insufficient evidence for the effectiveness of a treatment, we can openly resort to making recommendations for clinical practice based on consensus opinion or clinical expertise, of course being very much aware of the limitations of doing so.

We welcome more discussion and letters to the Editor on this contentious issue.