Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review
  • Published:

Are there roles for observational database studies and structured quantification of expert opinion to answer therapy controversies in transplants?

A Corrigendum to this article was published on 13 May 2009

Abstract

Approaches to determine whether one transplant-related therapy is better than another include: (1) using experimental data, such as those from randomized controlled trials (RCTs); (2) using observational data, such as those from observational databases (ODBs) and (3) using conclusions from the structured quantification of expert opinion based on a consideration of evidence from RCTs, ODBs and other sources. Large RCTs are widely and appropriately regarded as the gold standard of clinical investigation. However, data from large RCTs are rarely available for transplant-related therapy questions. We discuss some of the limitations of RCTs in the transplant setting often including small size and short follow-up. These limitations are only partly solved by meta-analyses of RCTs. Data from high-quality ODBs are not only often useful in this setting but also have limitations. Biases may be difficult or impossible to identify and/or adjust for. However, ODBs have large numbers of diverse subjects receiving diverse therapies and analyses that often give answers more useful to clinicians than RCTs. Side-by-side comparisons suggest analyses from high-quality ODBs often give similar conclusions to meta-analyses of high-quality RCTs. Meta-analyses combining data from RCTs and ODBs are sometimes appropriate. Quantitation of expert opinion, when of high quality, is also useful: experts rarely disagree under precisely defined circumstances and their consensus conclusions are often concordant with results of high-quality RCTs and ODBs. We suggest increased use of ODBs and expert opinion as reliable and effective ways to determine relative efficacies of new therapies in transplant settings.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Figure 1
Figure 2

Similar content being viewed by others

References

  1. Herland K, Akselsen JP, Skjønsberg OH, Bjermer L . How representative are clinical study patients with asthma or COPD for a larger ‘real life’ population of patients with obstructive lung disease? Respir Med 2005; 99: 11–19.

    Article  PubMed  Google Scholar 

  2. Rothwell PM . Can overall results of clinical trials be applied to all patients? Lancet 1995; 345: 1616–1619.

    Article  CAS  PubMed  Google Scholar 

  3. Mant D . Can randomized trials inform clinical decisions about individual patients? Lancet 1999; 353: 743–746.

    Article  CAS  PubMed  Google Scholar 

  4. Lambert PC, Sutton AJ, Abrams KR, Jones DR . A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol 2002; 55: 86–94.

    Article  CAS  PubMed  Google Scholar 

  5. Horwitz RI . Complexity and contradiction in clinical trial research. Am J Med 1987; 82: 498–510.

    Article  CAS  PubMed  Google Scholar 

  6. Zittoun RA, Mandelli F, Willemze R, de Witte T, Labar B, Resegotti L et al. Autologous or allogeneic bone marrow transplantation compared with intensive chemotherapy in acute myelogenous leukemia. European Organization for Research and Treatment of Cancer (EORTC) and the Gruppo Italiano Malattie Ematologiche Maligne dell’Adulto (GIMEMA) Leukemia Cooperative Groups. N Engl J Med 1995; 332: 217–223.

    Article  CAS  PubMed  Google Scholar 

  7. Cassileth PA, Harrington DP, Appelbaum FR, Lazarus HM, Rowe JM, Paietta E et al. Chemotherapy compared with autologous or allogeneic bone marrow transplantation in the management of acute myeloid leukemia in first remission. N Engl J Med 1998; 339: 1649–1656.

    Article  CAS  PubMed  Google Scholar 

  8. Ioannidis JP . Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005; 294: 218–228.

    Article  CAS  PubMed  Google Scholar 

  9. Kunz R, Oxman AD . The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998; 317: 1185–1190.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Logan BR, Leifer ES, Bredeson C, Horowitz MM, Ewell M, Carter S et al. Use of biological assignment in hematopoietic stem cell transplantation clinical trials. Clin Trials 2008; 5: 607–616.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Manjoo F . True Enough: Learning to Live in a Post-Fact Society. John Wiley and Sons: Hoboken, NJ, 2008.

    Google Scholar 

  12. Lord C, Ross L, Lepper M . Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence. J Pers Soc Psychol 1979; 37: 2098–2109.

    Article  Google Scholar 

  13. Bacon F . Novum organum. 1620.

  14. Westen D, Blagov PS, Harenski K, Kilts C, Hamann S . Neural bases of motivated reasoning: an FMRI study of emotional constraints on partisan political judgment in the 2004 US Presidential election. J Cogn Neurosci 2006; 18: 1947–1958.

    Article  PubMed  Google Scholar 

  15. Thaler RH, Sunstein CR . Nudge. Improving Decisions about Health, Wealth, and Happiness. Yale University Press: New Haven, CT, 2008.

    Google Scholar 

  16. Wansink B . Mindless Eating: Why We Eat More Than We Think. Bantam Books: New York, NY, 2006.

    Google Scholar 

  17. Ariely D . Predictably Irrational: the Hidden Forces that Shape our Decisions. HarperCollins: New York, NY, 2008.

    Google Scholar 

  18. Preventive Services Task Force. Guide to Clinical Preventive Services: Report of the US. Preventive Services Task Force, 2nd edn. Williams & Wilkins: Baltimore, 1996.

  19. Patsopoulos NA, Analatos AA, Ioannidis JP . Relative citation impact of various study designs in the health sciences. JAMA 2005; 293: 2362–2366.

    Article  CAS  PubMed  Google Scholar 

  20. Harbour R, Miller J . A new system for grading recommendations in evidence-based guidelines. Br Med J 2001; 323: 334–336.

    Article  CAS  Google Scholar 

  21. Kyzas PA, Denaxa-Kyza D, Ioannidis JP . Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer 2007; 43: 2559–2579.

    Article  PubMed  Google Scholar 

  22. Palma S, Delgado-Rodriguez M . Assessment of publication bias in meta-analyses of cardiovascular diseases. J Epidemiol Community Health 2005; 59: 864–869.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Sutton AJ, Cooper NJ, Abrams KR, Lambert PC, Jones DR . Bayesian approach to evaluating net clinical benefit allowed for parameter uncertainty. J Clin Epidemiol 2005; 58: 26–40.

    Article  PubMed  Google Scholar 

  24. Jørgensen AW, Hilden J, Gøtzsche PC . Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 2006; 333: 782.

    Article  PubMed  PubMed Central  Google Scholar 

  25. LeLorier J, Gre’goire G, Benhaddad A, Lapierre J, Derderian F . Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997; 337: 536–542.

    Article  CAS  PubMed  Google Scholar 

  26. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG et al. Comparison of evidence of treatment effects in randomized and non-randomized studies. JAMA 2001; 286: 821–830.

    Article  CAS  PubMed  Google Scholar 

  27. Shrier I . Cochrane reviews: new blocks on the kids. Br J Sports Med 2003; 37: 473–474.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Shrier I, Boivin J-F, Steele RJ, Platt RW, Furlan A, Kakuma R et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemol 2007; 166: 1203–1209.

    Article  Google Scholar 

  29. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ et al. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. Ann Intern Med 2007; 147: w163–w194.

    Article  PubMed  Google Scholar 

  30. Concato J, Shah N, Horwitz RI . Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342: 1887–1892.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Benson K, Hartz AJ . A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000; 342: 1878–1886.

    Article  CAS  PubMed  Google Scholar 

  32. Britton A, McPherson K, McKee M, Sanderson C, Black N, Bain C . Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998; 2: i–iv.

    Article  CAS  PubMed  Google Scholar 

  33. Guyatt GH, DiCenso A, Farewell V, Willan A, Griffith L . Randomized trials versus observational studies in adolescent pregnancy prevention. J Clin Epidemiol 2000; 53: 167–174.

    Article  CAS  PubMed  Google Scholar 

  34. MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AM . A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technol Assess 2000; 4: 1–154.

    Article  CAS  PubMed  Google Scholar 

  35. Powles R, Mehta J, Kulkarni S, Treleaven J, Millar B, Marsden J et al. Allogeneic blood and bone marrow stem-cell transplantation in hematological malignant diseases: a randomized trial. Lancet 2000; 355: 1231–1237.

    Article  CAS  PubMed  Google Scholar 

  36. Bensinger WI, Martin PJ, Storer B, Clift R, Forman SJ, Negrin R et al. Transplantation of bone marrow as compared with peripheral blood cells from HLA-identical relatives in patients with hematological cancers. N Engl J Med 2001; 344: 175–181.

    Article  CAS  PubMed  Google Scholar 

  37. Schmitz N, Beksac M, Hasenclever D, Bacigalupo A, Ruutu T, Nagler A et al. Transplantation of mobilized peripheral blood cells to HLA-identical siblings with standard risk leukemia. Blood 2002; 100: 761–767.

    Article  CAS  PubMed  Google Scholar 

  38. Champlin RE, Schmitz N, Horowitz MM, Chapuis B, Chopra R, Cornelissen JJ et al. Blood stem cells compared with bone marrow as a source of hematopoietic cells for allogeneic transplantation. Blood 2000; 95: 3702–3709.

    CAS  PubMed  Google Scholar 

  39. Schmitz N, Eapen M, Horowitz MM, Zhang MJ, Klein JP, Rizzo JD et al. Long-term outcome of patients given transplants of mobilized blood or bone marrow: a report from the International Bone Marrow Transplant Registry and the European Group for Blood and Marrow Transplantation. Blood 2006; 108: 4288–4290.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Eapen M, Logan BR, Confer DL, Haagenson M, Wagner JE, Weisdorf DJ et al. Peripheral blood grafts from unrelated donors are associated with increased acute and chronic graft-versus-host disease without improved survival. Biol Blood Marrow Transplant 2007; 13: 1461–1468.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Eapen M, Horowitz MM, Klein JP, Champlin RE, Loberiza Jr FR, Ringdén O et al. Higher mortality after allogeneic peripheral-blood transplantation compared with bone marrow in children and adolescents: the Histocompatibility and Alternate Stem Cell Source Working Committee of the International Bone Marrow Transplant Registry. J Clin Oncol 2004; 22: 4872–4880.

    Article  PubMed  Google Scholar 

  42. Schrezenmeier H, Passweg JR, Marsh JC, Bacigalupo A, Bredeson CN, Bullorsky E et al. Worse outcome and more chronic GvHD with peripheral blood progenitor cells than bone marrow in HLA-matched sibling donor transplants for young patients with severe acquired aplastic anemia. Blood 2007; 110: 1397–1400.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Smith GC, Pell JP . Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ 2003; 327: 1459–1461.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Berry DA, Broadwater G, Klein JP, Aisner J, Bitran J, Costanza M et al. High-dose versus standard chemotherapy in metastatic breast cancer: comparison of cancer and leukemia group B trials with data from the autologous blood and marrow transplant registry. J Clin Oncol 2002; 20: 743–750.

    Article  CAS  PubMed  Google Scholar 

  45. Lazarus HM, Stiff PJ, Carreras J, Logan BR, Akard L, Bolwell BJ et al. Utility of single versus tandem autotransplants for advanced testes/germ cancer. Biol Blood Marrow Transplant 2007; 13: 944–952.

    Article  Google Scholar 

  46. Surowiecki J . The Wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Random House: New York, NY, 2004.

    Google Scholar 

  47. Herrin J, Etchason JA, Kahan JP, Brook RH, Ballard DJ . Effect of panel composition on physician ratings of appropriateness of abdominal aortic aneurysm surgery: elucidating differences between multispecialty panel results and specialty society recommendations. Health Policy 1997; 42: 67–81.

    Article  CAS  PubMed  Google Scholar 

  48. Krupski TL, Kwan L, Afifi AA, Litwin MS . Geographic and socioeconomic variation in the treatment of prostate cancer. J Clin Oncol 2005; 23: 7881–7888.

    Article  PubMed  Google Scholar 

  49. Chassin MR, Kosecoff J, Park RE, Winslow CM, Kahn KL, Merrick NJ et al. Does inappropriate use explain geographic variations in the use of health care services? A study of three procedures. JAMA 1987; 258: 2533–2537.

    Article  CAS  PubMed  Google Scholar 

  50. Goodman C, Baratz SR (eds). Improving Consensus Development for Health Technology Assessment: an International Perspective. Council on Health Care Technology. Inst. Med. Natl Academy Press: Washington, DC, 1990.

    Google Scholar 

  51. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE et al. A method for the detailed assessment of the appropriateness of medical technologies. Intl J Tech Assess Health Care 1986; 2: 53–63.

    Article  CAS  Google Scholar 

  52. Leape LL, Hilborne LH, Park RE, Bernstein SJ, Kamberg CJ, Sherwood M et al. The appropriateness of coronary artery bypass surgery in New York State. JAMA 1993; 269: 753–760.

    Article  CAS  PubMed  Google Scholar 

  53. Bernstein SJ, Hilborne LH, Leape LL, Fiske ME, Park RE, Kamberg CJ et al. The appropriateness of coronary artery bypass surgery in New York State. JAMA 1993; 269: 766–769.

    Article  CAS  PubMed  Google Scholar 

  54. Bernstein SJ, McGlynn EA, Siu AL, Roth CP, Sherwood MJ, Keesey JW et al. The appropriateness of hysterectomy; a comparison of seven health plans. JAMA 1993; 269: 2398–2402.

    Article  CAS  PubMed  Google Scholar 

  55. Bengtson A, Herliz J, Karlsson T, Brandrup-Wognsen G, Hjalmarson A . The appropriateness of performing coronary angioplasty and coronary revascularization in a Swedish population. JAMA 1994; 271: 1260–1265.

    Article  CAS  PubMed  Google Scholar 

  56. Gray D, Hampton JR, Bernstein SJ, Kosecoff J, Brook RH . Audit of coronary angiography and bypass surgery. Lancet 1990; 335: 1317–1320.

    Article  CAS  PubMed  Google Scholar 

  57. Pearson SD, Margolis CZ, Davis S, Schreier LK, Sokol HN, Gottlieb LK . Is consensus reproducible? A study of an algorithmic guidelines development process. Med Care 1995; 33: 643–660.

    Article  CAS  PubMed  Google Scholar 

  58. Shekelle PG, Chassin MR, Park RE . Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care 1998; 14: 707–727.

    Article  CAS  PubMed  Google Scholar 

  59. Kravitz RL, Laouri M, Kahan JP, Guzy P, Sherman T, Hilborne L et al. Validity of criteria used for detecting underuse of coronary revascularization. JAMA 1995; 274: 632–638.

    Article  CAS  PubMed  Google Scholar 

  60. Hahn T, Wolff SN, Czuczman M, Fisher RI, Lazarus HM, McCarthy PL et al. American Society for Blood and Marrow Transplantation (ASBMT). The role of cytotoxic therapy with hematopoietic stem cell transplantation in the treatment of diffuse large cell B-cell non-Hodgkin's lymphoma. Biol Blood Marrow Transplant 2003; 7: 308–331.

    Article  Google Scholar 

  61. Hahn T, Wingard JR, Anderson KC, Bensinger WI, Berenson JR, Brozeit G et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of multiple myeloma: an evidence-based review. Biol Blood Marrow Transplant 2003; 9: 4–37.

    Article  PubMed  Google Scholar 

  62. Hahn T, Wall D, Camitta B, Davies S, Dillon H, Gaynon P et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute lymphoblastic leukemia in adults: an evidence-based review. Biol Blood Marrow Transplant 2006; 12: 1–30.

    Article  CAS  PubMed  Google Scholar 

  63. Hahn T, Wall D, Camitta B, Davies S, Dillon H, Gaynon P et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute lymphoblastic leukemia in children: an evidence-based review. Biol Blood Marrow Transplant 2005; 11: 823–861.

    Article  PubMed  Google Scholar 

  64. Oliansky DM, Appelbaum F, Cassileth PA, Keating A, Kerr J, Nieto Y et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute myelogenous leukemia in adults: an evidence-based review. Biol Blood Marrow Transplant 2008; 14: 137–180.

    Article  PubMed  Google Scholar 

  65. Oliansky DM, Rizzo JD, Aplan PD, Arceci RJ, Leone L, Ravindranath Y et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute myeloid leukemia in children: an evidence-based review. Biol Blood Marrow Transplant 2007; 13: 1–25.

    Article  PubMed  Google Scholar 

  66. Gale RP, Park RE, Dubois RW, Bitran JD, Buzdar A, Hortobagyi G et al. Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in women with breast cancer. Clin Transplant 2000; 14: 32–41.

    Article  CAS  PubMed  Google Scholar 

  67. Gale RP, Park RE, Dubois RW, Herzig GP, Hocking WG, Horowitz MM et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in chronic myelogenous leukemia in chronic phase. Leuk Res 1998; 23: 817–826.

    Article  Google Scholar 

  68. Gale RP, Park RE, Dubois RW, Herzig GP, Hocking WG, Horowitz MM et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute myelogenous leukemia in 1st remission. Leuk Res 1999; 23: 709–718.

    Article  CAS  PubMed  Google Scholar 

  69. Gale RP, Park RE, Dubois RW, Anderson KC, Audeh WM, Bergsagel D et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow autotransplants in newly diagnosed multiple myeloma. Leuk Lymphoma 1999; 33: 511–519.

    Article  CAS  PubMed  Google Scholar 

  70. Gale RP, Park RE, Dubois RW, Bierman PJ, Fisher RI, Loh KK et al. Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in diffuse large-cell lymphoma. Leuk Lymphoma 1998; 32: 139–149.

    Article  CAS  PubMed  Google Scholar 

  71. Gale RP, Park RE, Dubois RW, Herzig GP, Hocking WG, Horowitz MM et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute lymphoblastic leukemia in first remission. Leuk Res 1998; 22: 973–981.

    Article  CAS  PubMed  Google Scholar 

  72. Nathan PC, Sung L, Crump M, Beyene J . Consolidation therapy with autologous bone marrow transplantation in adults with acute myeloid leukemia: a meta-analysis. J Natl Cancer Inst 2004; 96: 38–45.

    Article  CAS  PubMed  Google Scholar 

  73. Levi I, Grotto I, Yerushalmi R, Ben-Bassat I, Shpilberg O . Meta-analysis of autologous bone marrow transplantation versus chemotherapy in adult patients with acute myeloid leukemia in first remission. Leuk Res 2004; 28: 605–612.

    Article  CAS  PubMed  Google Scholar 

  74. Breems DA, Boogaerts MA, Dekker AW, Van Putten WL, Sonneveld P, Huijgens PC et al. Autologous bone marrow transplantation as consolidation therapy in the treatment of adult patients under 60 years with acute myeloid leukaemia in first complete remission: a prospective randomized Dutch–Belgian Haemato-Oncology Co-operative Group (HOVON) and Swiss Group for Clinical Cancer Research (SAKK) trial. Br J Haematol 2005; 128: 59–65.

    Article  PubMed  Google Scholar 

  75. Dolnick E . Fish or foul?. New York Times, 2 September 2008. Opinions.

  76. Karanicolas PJ, Montori VM, Devereaux PJ, Schuenemann H, Guyatt GH . A new ‘mechanic-practical’ framework for designing and interpreting randomized trials. J Clin Epidemiol; e-pub ahead of print, 9 May 2008.

Download references

Acknowledgements

We are grateful to our many colleagues at more than 150 transplant centers worldwide who, over the past 35 years, have challenged us to tackle these issues. We are especially indebted to our severest critics (you know who you are) who forced us to think harder about the use of ODBs to answer important clinical questions. Dr R Ed Park taught one of us (RPG) quite a lot about structured quantification of expert opinion. We shamelessly pirated the brilliant parody of Smith and Pell43 to illustrate some points. The late Drs Mortimer Bortin, Albert Rimm and their colleagues pioneered the concept of using ODBs to address issues in transplants; we dedicate this article to Mort. Mary M Horowitz, and John Klein continued and expanded Mort's legacy. Sabine Jacob kindly struggled through endless iterations of the typescript.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to R P Gale.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gale, R., Eapen, M., Logan, B. et al. Are there roles for observational database studies and structured quantification of expert opinion to answer therapy controversies in transplants?. Bone Marrow Transplant 43, 435–446 (2009). https://doi.org/10.1038/bmt.2008.447

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/bmt.2008.447

Keywords

This article is cited by

Search

Quick links