Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Original Article
  • Published:

The application of quantitative methods for identifying and exploring the presence of bias in systematic reviews: PDE-5 inhibitors for erectile dysfunction

International Journal of Impotence Research volume 20, pages 264–277 (2008)Cite this article

Abstract

A systematic review of PDE-5 inhibitors for erectile dysfunction was performed to evaluate the utility of quantitative methods for identifying and exploring the influence of bias and study quality on pooled outcomes from meta-analyses. We included 123 randomized controlled trials (RCTs). Methodological quality was poorly reported. All three drugs appeared highly effective. Indirect adjusted analyses showed no differences between the three drugs. Funnel plots and statistical tests showed no evidence of small-study effects for sildenafil whereas there was evidence of such bias for tadalafil and vardenafil. Adjustment for missing studies using trim and fill techniques did not alter the pooled estimates substantially. The exclusion of previous sildenafil nonresponders was associated with larger treatment effects for tadalafil. This investigation was hampered by poor reporting of methodological quality, a low number of studies, heterogeneity and large effect sizes. Despite such limitations, a comprehensive assessment of biases should be a routine in systematic reviews.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1

Similar content being viewed by others

References

  1. Fink HA, Mac DR, Rutks IR, Nelson DB, Wilt TJ . Sildenafil for male erectile dysfunction: a systematic review and meta-analysis [see comment]. [Review] [37 refs]. Arch Intern Med 2002; 162: 1349–1360.

    Article  CAS  PubMed  Google Scholar 

  2. Markou S, Perimenis P, Gyftopoulos K, Athanasopoulos A, Barbalias G . Vardenafil (Levitra) for erectile dysfunction: a systematic review and meta-analysis of clinical trial reports. [Review] [34 refs]. Int J Impot Res 2004; 16: 470–478.

    Article  CAS  PubMed  Google Scholar 

  3. Carson CC, Rajfer J, Eardley I, Carrier S, Denne JS, Walker DJ et al. The efficacy and safety of tadalafil: an update. BJU Int 2004; 93: 1276–1281.

    Article  CAS  PubMed  Google Scholar 

  4. Berner MM, Kriston L, Harms A . Efficacy of PDE-5-inhibitors for erectile dysfunction. A comparative meta-analysis of fixed-dose regimen randomized controlled trials administering the International Index of Erectile Function in broad-spectrum populations. Int J Impot Res 2006; 18: 229–235.

    Article  CAS  PubMed  Google Scholar 

  5. Moore RA, Derry S, McQuay HJ . Indirect comparison of interventions using published randomised trials: systematic review of PDE-5 inhibitors for erectile dysfunction. BMC Urol 2005; 5: 18.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Schulz KF, Chalmers I, Hayes RJ, Altman DG . Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–412.

    Article  CAS  PubMed  Google Scholar 

  7. Moher D, Pham B, Jones A, Cook DJ, Jadad AR . Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609–613.

    Article  CAS  PubMed  Google Scholar 

  8. Dickersin K, Min YI, Meinert CL . Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA 1992; 267: 374–378.

    Article  CAS  PubMed  Google Scholar 

  9. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR . Publication bias in clinical research. Lancet 1991; 337: 867–872.

    Article  CAS  PubMed  Google Scholar 

  10. Broderick GA, Donatucci CF, Hatzichristou D, Torres LO, Valiquette L, Zhao YL et al. Efficacy of tadalafil in men with erectile dysfunction naive to phosphodiesterase 5 inhibitor therapy compared with prior responders to sildenafil citrate. J Sex Med 2006; 3: 668–675.

    Article  CAS  PubMed  Google Scholar 

  11. Gehr BT, Weiss C, Porzsolt F . The fading of reported effectiveness. A meta-analysis of randomised controlled trials. BMC Med Res Methodol 2006; 6: 25.

    Article  PubMed  PubMed Central  Google Scholar 

  12. US Food and Drug Administration, http://www.fda.gov/cder/. Accessed: September 4, 2007.

  13. Robinson KA, Dickersin K . Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol 2002; 31: 150–153.

    Article  PubMed  Google Scholar 

  14. Altman DG . Practical Statistics for Medical Research, 2nd edn. Chapman & Hall: London, 1997.

    Google Scholar 

  15. Bucher HC, Guyatt GH, Griffith LE, Walter SD . The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997; 50: 683–691.

    Article  CAS  PubMed  Google Scholar 

  16. Song F, Altman DG, Glenny AM, Deeks JJ . Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ 2003; 326: 472.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Deeks JJ, Altman DG, Bradburn J . Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. In: Egger M, Davey SG, Altman DG (eds). Systematic Reviews in Health Care. Meta-Analysis in Context, 3rd edn. BMJ: London, 2003, pp 285–312.

    Google Scholar 

  18. Higgins JP, Thompson SG . Quantifying heterogeneity in a meta-analysis. Stat Med 2002; 21: 1539–1558.

    Article  PubMed  Google Scholar 

  19. Higgins JP, Thompson SG, Deeks JJ, Altman DG . Measuring inconsistency in meta-analyses. BMJ 2003; 327: 557–560.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Sterne JA, Gavaghan D, Egger M . Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000; 53: 1119–1129.

    Article  CAS  PubMed  Google Scholar 

  21. Sterne JA, Egger M . Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001; 54: 1046–1055.

    Article  CAS  PubMed  Google Scholar 

  22. Begg CB, Mazumdar M . Operating characteristics of a rank correlation test for publication bias. Biometrics 1994; 50: 1088–1101.

    Article  CAS  PubMed  Google Scholar 

  23. Egger M, Davey SG, Schneider M, Minder C . Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–634.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Duval S, Tweedie R . Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000; 56: 455–463.

    Article  CAS  PubMed  Google Scholar 

  25. Duval SJ, Tweedie RL . A non-parametric ‘trim and fill’ method of assessing publication bias in meta-analysis. J Am Stat Assoc 2000; 95: 89–98.

    Google Scholar 

  26. Begg CB, Cho MK, Eastwood S, Horton R, Moher D, Olkin I et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996; 276: 637–639.

    Article  CAS  PubMed  Google Scholar 

  27. Lau J, Ionnidis JPA, Terrin N, Schmidt CH, Olkin I . The case of the misleading funnel plot. BMJ 2006; 333: 597–600.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Sterne JA, Egger M . Re: High false positive rate for trim and fill method. Website only: http://www.bmj.com/cgi/eletters/320/7249/1574 2000.

  29. Sutton AJ, Duval SJ, Tweedie RL . Re: Re: High false positive rate for trim and fill method. Website only: http://www.bmj.com/cgi/eletters/320/7249/1574 2000.

  30. Lewis R, Bennett CJ, Borkon WD, Boykin WH, Althof SE, Stecher VJ et al. Patient and partner satisfaction with Viagra (sildenafil citrate) treatment as determined by the Erectile Dysfunction Inventory of Treatment Satisfaction Questionnaire. Urology 2001; 57: 960–965.

    Article  CAS  PubMed  Google Scholar 

  31. Brock G, Nehra A, Lipshultz LI, Karlin GS, Gleave M, Seger M et al. Safety and efficacy of vardenafil for the treatment of men with erectile dysfunction after radical retropubic prostatectomy. J Urol 2003; 170 (4 Part 1): 1278–1283.

    Article  CAS  PubMed  Google Scholar 

  32. Carson C, Shabsigh R, Segal S, Murphy A, Fredlund P, Kuepfer C et al. Efficacy, safety, and treatment satisfaction of tadalafil versus placebo in patients with erectile dysfunction evaluated at tertiary-care academic centers. Urology 2005; 65: 353–359.

    Article  PubMed  Google Scholar 

  33. Gravel J, Opatmy L, Shapiro S . The intention-to-treat approach in randomized controlled trials: are authors saying what they do and doing what they say? Clin Trials 2007; 4: 350–356.

    Article  PubMed  Google Scholar 

  34. Hollis S, Campbell F . What is meant by intention to treat analysis? Survey of published randomized controlled trials. BMJ 1999; 319: 670–674.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B . Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 2003; 326: 1171–1173.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This paper is based on original research conducted by Kleijnen Systematic Reviews. The paper, its content, financial support and editorial control were at all times under the control of Kleijnen Systematic Reviews and its lead authors. However, to ensure full transparency we would like to declare that we have been involved in a systematic review of PDE-5 inhibitors that was supported and funded by Pfizer Ltd.

Author information

Authors and Affiliations

  1. BeSyRe Bekkering Systematic Reviews, Geel, Belgium

    G E Bekkering

  2. Department of General Practice, Catholic University of Leuven, Leuven, Belgium

    G E Bekkering

  3. Giza, Egypt

    A M Abou-Setta

  4. Kleijnen Systematic Reviews Ltd, York, UK

    J Kleijnen

Authors
  1. G E Bekkering
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A M Abou-Setta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J Kleijnen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to G E Bekkering.

Additional information

Supplementary Information accompanies the paper on International Journal of Impotence Research website (http://www.nature.com/ijir).

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bekkering, G., Abou-Setta, A. & Kleijnen, J. The application of quantitative methods for identifying and exploring the presence of bias in systematic reviews: PDE-5 inhibitors for erectile dysfunction. Int J Impot Res 20, 264–277 (2008). https://doi.org/10.1038/sj.ijir.3901626

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/sj.ijir.3901626

Keywords

This article is cited by

Search

Advanced search

Quick links