Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The role of the funnel plot in detecting publication and related biases in meta-analysis

What is publication bias and why is it important?

The late and distinguished Alvan Feinstein expressed concern about the proliferation of positive associations with the ‘menace[s] of everyday life’ in risk factor epidemiology.1 Publication bias is a long-standing problem in clinical research, in which treatments that demonstrate a significant health benefit are more likely to be published than those that do not.2,3 The problem was highlighted in the 1950s when a survey of empirical studies published in leading psychology journals discovered an astonishing 97% reported statistically significant results.4 Meta-analyses which include only published studies are therefore likely to be biased.

The impact of publication bias on the integrity of medical research and the findings of meta-analyses have been widely discussed and evaluated.2, 3, 5, 6, 7, 8 In an early example (1986), Simes compared the benefit of treating advanced ovarian cancer using an alkylating agent with combination chemotherapy. He conducted a meta-analysis of published trials only and a meta-analysis of all registered trials.2 The pooled median survival ratio for the published trials was 1.16 (95% CI 1.06 to 1.27), which suggested improved survival using combination therapy, while the pooled ratio for the registered trials was 1.06 (95% CI 0.97-1.15), a smaller and less optimistic improvement that was not statistically significant. The difference between the two pooled estimates was attributed to publication bias.2

Publication bias in medical research continues to be a problem today, as shown by a recent meta-analysis which compared the effectiveness of reboxetine with selective serotonin reuptake inhibitors (SSRI) or placebo for treating major depression.8 The investigators found that 74% of data were not published and ‘published data overestimated the benefit of reboxetine versus placebo by up to 115% and reboxetine versus SSRIs by up to 23%, and also underestimated harm’.8 Previous findings of reboxetine's effectiveness were reversed.

In dentistry, Scholey and Harrison demonstrated that more than 50% of studies presented at international conferences in dental health were unpublished five years after the conference, suggesting that a meta-analysis in a dental health-related area could be biased if a search for unpublished studies is not included.9 A review of five orthodontic journals shows that 88% of studies demonstrated statistically significant results.10 A similar study of journals in maxillofacial surgery came up with a figure of 77%.11 Like other health care domains, publication bias is clearly a problem in dental research.9, 10, 11, 12

The funnel plot as a tool to detect publication bias

The funnel plot is a commonly used graphical device to detect publication bias in systematic reviews. Originally advocated by Light and Pillemer13, it is a plot of a study-specific effect estimate against an estimate of the study's precision. Precision may be assessed in various ways, through a function of the standard error for an effect measure or simply the sample size in each study of a systematic review.13, 14, 15, 16, 17 Effect estimates include relative risk, risk ratio, odds ratio, absolute risk and logarithmic transformations of these measures.

In the absence of publication bias, the points will be symmetrically distributed around the true effect in the shape of an inverted funnel (Figure 1).13, 14, 15, 16, 17 As the sample size of studies in an unbiased meta-analysis increases, the effect estimates become more precise. The scatter of effect estimates around the true effect would therefore be expected to become wider at the bottom end of the plot, where the smaller studies are located. The scatter becomes progressively narrower as the studies increase in size going up the plot. In principle, the pooled effect estimate should reflect the true effect. In the presence of publication bias the shape of the funnel plot will be asymmetric, with negative effect estimates from smaller studies missing from the plot and the pooled effect estimate diverging from the true effect.18 (Figure 2).

Figure 1
figure 1

Symmetrical funnel plot

Figure 2
figure 2

Asymmetric funnel plot

However, the empirical evidence strongly suggests that visual inspection of funnel plots alone is not a reliable way to assess the shape of the funnel plot.19

Statistical tests to assess shape of funnel plot

Various statistical tests have thus been developed to more objectively evaluate the shape of the funnel plot and test for the presence of publication bias. Regression models are the most commonly used,20 and test for an association between the study's effect estimate and its precision, as measured for example by the inverse of the standard error.21 Publication bias may be present if the fitted regression model suggests that the less precise or smaller studies have bigger effect estimates than the more precise or larger studies.21 A weakness of these tests is that they have low statistical power and sometimes fail to detect publication bias when it actually exists. The Cochrane Handbook for Systematic Reviews of Interventions, available online at http://www.cochrane-handbook.org/, provides a list of different regression models.20

The results of these regression models can vary, depending on the choice of effect measure and the choice of precision measure used to construct the funnel plot.15 This issue has been discussed in various contexts for many years15, 16, 19, 22, 23, 24, 25, 26 and is beyond the scope of the present Toolbox article. The reader is advised to refer to the Cochrane Collaboration's latest recommendations, which undergo regular review and periodic changes.20

Publication bias as part of a more general phenomenon

It should be noted that publication bias is part of a more general type of bias:

Reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results. Statistically significant, ‘positive’ results that indicate that an intervention works are more likely to be published, more likely to be published rapidly, more likely to be published in English, more likely to be published more than once, more likely to be published in high impact journals and, related to the last point, more likely to be cited by others.20

The Cochrane Handbook provides a detailed discussion and typology of reporting biases.20 Reporting bias (including publication bias), true heterogeneity and chance, could all account for an asymmetric funnel plot.15, 20, 27 True heterogeneity refers to genuine variation in effect size according to the size of the study. This could happen if the characteristics of patients in small trials are different from those in large trials, eg small trials may enrol higher risk patients than large trials and effect size may depend on the underlying patient risk.27, 28, 29 Finally, play of chance could also result in an asymmetric plot.

Implications for dental health research

Like other health care domains, publication bias is a serious problem in dentistry.9, 10, 11, 12 Dental health researchers might quite understandably think of the funnel plot and associated statistical tests as tools to detect only publication bias. A funnel plot can be asymmetric for a number of reasons and statistical tests to assess reporting bias have their limitations. In the absence of universal access to trial data in an ideal world, these tests serve as an aid to interpretation.30,31 Dental health researchers and practitioners need to assess funnel plots with the above caveats in mind when reading systematic reviews and to seek advice from experienced systematic reviewers in using funnel plots and associated tests when conducting their own reviews.

References

  1. Feinstein AR . Scientific standards in epidemiologic studies of the menace of daily life. Science 1988; 242: 1257–1263.

    Article  Google Scholar 

  2. Simes RJ . Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986; 4: 1529–1541.

    Article  Google Scholar 

  3. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR . Publication bias in clinical research. Lancet 1991; 337(8746): 867–872.

    Article  Google Scholar 

  4. Sterling TD . Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. J Am Stat Assoc 1959; 54: 30–34.

    Google Scholar 

  5. Kennedy D . The old file-drawer problem. Science 2004; 305: 451.

    Article  Google Scholar 

  6. Liu JL, Altman DG . Conduct and reporting of clinical research. Science 2005; 308: 201–202.

    Article  Google Scholar 

  7. Rosenthal R . The file drawer problem and tolerance for null results. Psychol Bull 1979; 86: 638–641.

    Article  Google Scholar 

  8. Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010; 341: c4737.

    Article  Google Scholar 

  9. Scholey JM, Harrison JE . Delay and failure to publish dental research. Evid Based Dent 2005; 6: 58–61.

    Article  Google Scholar 

  10. Koletsi D, Karagianni A, Pandis N, Makou M, Polychronopoulou A, Eliades T . Are studies reporting significant results more likely to be published? Am J Orthod Dentofacial Orthop 2009; 136: 632.

    Article  Google Scholar 

  11. Pitak-Arnnop P, Sader R, Rapidis AD, et al. Publication bias in oral and maxillofacial surgery journals: an observation on published controlled trials. J Craniomaxillofac Surg 2010; 38: 4–10.

    Article  Google Scholar 

  12. Scholey JM, Harrison JE . Publication bias: raising awareness of a potential problem in dental research. Br Dent J 2003; 194: 235–237.

    Article  Google Scholar 

  13. Light R, Pillemer DB . Summing up: the science of reviewing research. Cambridge: Harvard University Press, 1984.

    Google Scholar 

  14. Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F . Methods for meta-analysis in medical research. Chichester: John Wiley & Sons, 2000.

    Google Scholar 

  15. Tang JL, Liu JL . Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol 2000; 53: 477–484.

    Article  Google Scholar 

  16. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I . The case of the misleading funnel plot. BMJ 2006; 333: 597–600.

    Article  Google Scholar 

  17. Petitti DB . Meta-analysis, decision analysis, and cost effectiveness analysis: methods for quantitative synthesis in medicine. 2nd ed. New York: Oxford University Press, 2000.

    Google Scholar 

  18. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L . Comparison of two methods to detect publication bias in meta-analysis. JAMA 2006; 295: 676–680.

    Article  Google Scholar 

  19. Terrin N, Schmid CH, Lau J . In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005; 58: 894–901.

    Article  Google Scholar 

  20. Higgins JPT, Green S . Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Updated March 2011] (http://www.cochrane-handbook.org/). 2011.

  21. Moreno SG, Sutton AJ, Ades AE, et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol 2009; 9: 2 doi: 10.1186/1471-2288-9-2.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Pfeiffer T, Bertram L, Ioannidis JP . Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 2011; 6: e18362.

    Article  Google Scholar 

  23. Sterne JA, Egger M . Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001; 54: 1046–1055.

    Article  Google Scholar 

  24. Gigerenzer G, Wegwarth O, Feufel M . Misleading communication of risk. BMJ 2010; 341: 791–792.

    Article  Google Scholar 

  25. Rucker G, Schwarzer G, Carpenter JR, Binder H, Schumacher M . Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics 2011; 12: 122–142.

    Article  Google Scholar 

  26. Ioannidis JP . Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract 2008; 14: 951–957.

    Article  Google Scholar 

  27. Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Publishing Group, 2001.

    Book  Google Scholar 

  28. Altman DG . Practical Statistics for Medical Research. London: Chapman and Hall, 1991.

    Google Scholar 

  29. Smith GD, Song F, Sheldon TA . Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ 1993; 306: 1367–1373.

    Article  Google Scholar 

  30. Steinbrook R, Kassirer JP . Free the data. BMJ 2010; 341: 811–812.

    Article  Google Scholar 

  31. Freemantle N . Commentary: Journals must facilitate the dissemination and scrutiny of clinical research. BMJ 2010; 341: 812–813.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Liu, J. The role of the funnel plot in detecting publication and related biases in meta-analysis. Evid Based Dent 12, 121–122 (2011). https://doi.org/10.1038/sj.ebd.6400831

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/sj.ebd.6400831

Further reading

Search

Quick links