Approaches to determine whether one transplant-related therapy is better than another include: (1) using experimental data, such as those from randomized controlled trials (RCTs); (2) using observational data, such as those from observational databases (ODBs) and (3) using conclusions from the structured quantification of expert opinion based on a consideration of evidence from RCTs, ODBs and other sources. Large RCTs are widely and appropriately regarded as the gold standard of clinical investigation. However, data from large RCTs are rarely available for transplant-related therapy questions. We discuss some of the limitations of RCTs in the transplant setting often including small size and short follow-up. These limitations are only partly solved by meta-analyses of RCTs. Data from high-quality ODBs are not only often useful in this setting but also have limitations. Biases may be difficult or impossible to identify and/or adjust for. However, ODBs have large numbers of diverse subjects receiving diverse therapies and analyses that often give answers more useful to clinicians than RCTs. Side-by-side comparisons suggest analyses from high-quality ODBs often give similar conclusions to meta-analyses of high-quality RCTs. Meta-analyses combining data from RCTs and ODBs are sometimes appropriate. Quantitation of expert opinion, when of high quality, is also useful: experts rarely disagree under precisely defined circumstances and their consensus conclusions are often concordant with results of high-quality RCTs and ODBs. We suggest increased use of ODBs and expert opinion as reliable and effective ways to determine relative efficacies of new therapies in transplant settings.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Standardizing haematopoietic cell transplants in China
Journal of Hematology & Oncology Open Access 02 March 2018
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Rent or buy this article
Get just this article for as long as you need it
Prices may be subject to local taxes which are calculated during checkout
Herland K, Akselsen JP, Skjønsberg OH, Bjermer L . How representative are clinical study patients with asthma or COPD for a larger ‘real life’ population of patients with obstructive lung disease? Respir Med 2005; 99: 11–19.
Rothwell PM . Can overall results of clinical trials be applied to all patients? Lancet 1995; 345: 1616–1619.
Mant D . Can randomized trials inform clinical decisions about individual patients? Lancet 1999; 353: 743–746.
Lambert PC, Sutton AJ, Abrams KR, Jones DR . A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol 2002; 55: 86–94.
Horwitz RI . Complexity and contradiction in clinical trial research. Am J Med 1987; 82: 498–510.
Zittoun RA, Mandelli F, Willemze R, de Witte T, Labar B, Resegotti L et al. Autologous or allogeneic bone marrow transplantation compared with intensive chemotherapy in acute myelogenous leukemia. European Organization for Research and Treatment of Cancer (EORTC) and the Gruppo Italiano Malattie Ematologiche Maligne dell’Adulto (GIMEMA) Leukemia Cooperative Groups. N Engl J Med 1995; 332: 217–223.
Cassileth PA, Harrington DP, Appelbaum FR, Lazarus HM, Rowe JM, Paietta E et al. Chemotherapy compared with autologous or allogeneic bone marrow transplantation in the management of acute myeloid leukemia in first remission. N Engl J Med 1998; 339: 1649–1656.
Ioannidis JP . Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005; 294: 218–228.
Kunz R, Oxman AD . The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998; 317: 1185–1190.
Logan BR, Leifer ES, Bredeson C, Horowitz MM, Ewell M, Carter S et al. Use of biological assignment in hematopoietic stem cell transplantation clinical trials. Clin Trials 2008; 5: 607–616.
Manjoo F . True Enough: Learning to Live in a Post-Fact Society. John Wiley and Sons: Hoboken, NJ, 2008.
Lord C, Ross L, Lepper M . Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence. J Pers Soc Psychol 1979; 37: 2098–2109.
Bacon F . Novum organum. 1620.
Westen D, Blagov PS, Harenski K, Kilts C, Hamann S . Neural bases of motivated reasoning: an FMRI study of emotional constraints on partisan political judgment in the 2004 US Presidential election. J Cogn Neurosci 2006; 18: 1947–1958.
Thaler RH, Sunstein CR . Nudge. Improving Decisions about Health, Wealth, and Happiness. Yale University Press: New Haven, CT, 2008.
Wansink B . Mindless Eating: Why We Eat More Than We Think. Bantam Books: New York, NY, 2006.
Ariely D . Predictably Irrational: the Hidden Forces that Shape our Decisions. HarperCollins: New York, NY, 2008.
Preventive Services Task Force. Guide to Clinical Preventive Services: Report of the US. Preventive Services Task Force, 2nd edn. Williams & Wilkins: Baltimore, 1996.
Patsopoulos NA, Analatos AA, Ioannidis JP . Relative citation impact of various study designs in the health sciences. JAMA 2005; 293: 2362–2366.
Harbour R, Miller J . A new system for grading recommendations in evidence-based guidelines. Br Med J 2001; 323: 334–336.
Kyzas PA, Denaxa-Kyza D, Ioannidis JP . Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer 2007; 43: 2559–2579.
Palma S, Delgado-Rodriguez M . Assessment of publication bias in meta-analyses of cardiovascular diseases. J Epidemiol Community Health 2005; 59: 864–869.
Sutton AJ, Cooper NJ, Abrams KR, Lambert PC, Jones DR . Bayesian approach to evaluating net clinical benefit allowed for parameter uncertainty. J Clin Epidemiol 2005; 58: 26–40.
Jørgensen AW, Hilden J, Gøtzsche PC . Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 2006; 333: 782.
LeLorier J, Gre’goire G, Benhaddad A, Lapierre J, Derderian F . Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997; 337: 536–542.
Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG et al. Comparison of evidence of treatment effects in randomized and non-randomized studies. JAMA 2001; 286: 821–830.
Shrier I . Cochrane reviews: new blocks on the kids. Br J Sports Med 2003; 37: 473–474.
Shrier I, Boivin J-F, Steele RJ, Platt RW, Furlan A, Kakuma R et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemol 2007; 166: 1203–1209.
Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ et al. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. Ann Intern Med 2007; 147: w163–w194.
Concato J, Shah N, Horwitz RI . Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342: 1887–1892.
Benson K, Hartz AJ . A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000; 342: 1878–1886.
Britton A, McPherson K, McKee M, Sanderson C, Black N, Bain C . Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998; 2: i–iv.
Guyatt GH, DiCenso A, Farewell V, Willan A, Griffith L . Randomized trials versus observational studies in adolescent pregnancy prevention. J Clin Epidemiol 2000; 53: 167–174.
MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AM . A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technol Assess 2000; 4: 1–154.
Powles R, Mehta J, Kulkarni S, Treleaven J, Millar B, Marsden J et al. Allogeneic blood and bone marrow stem-cell transplantation in hematological malignant diseases: a randomized trial. Lancet 2000; 355: 1231–1237.
Bensinger WI, Martin PJ, Storer B, Clift R, Forman SJ, Negrin R et al. Transplantation of bone marrow as compared with peripheral blood cells from HLA-identical relatives in patients with hematological cancers. N Engl J Med 2001; 344: 175–181.
Schmitz N, Beksac M, Hasenclever D, Bacigalupo A, Ruutu T, Nagler A et al. Transplantation of mobilized peripheral blood cells to HLA-identical siblings with standard risk leukemia. Blood 2002; 100: 761–767.
Champlin RE, Schmitz N, Horowitz MM, Chapuis B, Chopra R, Cornelissen JJ et al. Blood stem cells compared with bone marrow as a source of hematopoietic cells for allogeneic transplantation. Blood 2000; 95: 3702–3709.
Schmitz N, Eapen M, Horowitz MM, Zhang MJ, Klein JP, Rizzo JD et al. Long-term outcome of patients given transplants of mobilized blood or bone marrow: a report from the International Bone Marrow Transplant Registry and the European Group for Blood and Marrow Transplantation. Blood 2006; 108: 4288–4290.
Eapen M, Logan BR, Confer DL, Haagenson M, Wagner JE, Weisdorf DJ et al. Peripheral blood grafts from unrelated donors are associated with increased acute and chronic graft-versus-host disease without improved survival. Biol Blood Marrow Transplant 2007; 13: 1461–1468.
Eapen M, Horowitz MM, Klein JP, Champlin RE, Loberiza Jr FR, Ringdén O et al. Higher mortality after allogeneic peripheral-blood transplantation compared with bone marrow in children and adolescents: the Histocompatibility and Alternate Stem Cell Source Working Committee of the International Bone Marrow Transplant Registry. J Clin Oncol 2004; 22: 4872–4880.
Schrezenmeier H, Passweg JR, Marsh JC, Bacigalupo A, Bredeson CN, Bullorsky E et al. Worse outcome and more chronic GvHD with peripheral blood progenitor cells than bone marrow in HLA-matched sibling donor transplants for young patients with severe acquired aplastic anemia. Blood 2007; 110: 1397–1400.
Smith GC, Pell JP . Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ 2003; 327: 1459–1461.
Berry DA, Broadwater G, Klein JP, Aisner J, Bitran J, Costanza M et al. High-dose versus standard chemotherapy in metastatic breast cancer: comparison of cancer and leukemia group B trials with data from the autologous blood and marrow transplant registry. J Clin Oncol 2002; 20: 743–750.
Lazarus HM, Stiff PJ, Carreras J, Logan BR, Akard L, Bolwell BJ et al. Utility of single versus tandem autotransplants for advanced testes/germ cancer. Biol Blood Marrow Transplant 2007; 13: 944–952.
Surowiecki J . The Wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Random House: New York, NY, 2004.
Herrin J, Etchason JA, Kahan JP, Brook RH, Ballard DJ . Effect of panel composition on physician ratings of appropriateness of abdominal aortic aneurysm surgery: elucidating differences between multispecialty panel results and specialty society recommendations. Health Policy 1997; 42: 67–81.
Krupski TL, Kwan L, Afifi AA, Litwin MS . Geographic and socioeconomic variation in the treatment of prostate cancer. J Clin Oncol 2005; 23: 7881–7888.
Chassin MR, Kosecoff J, Park RE, Winslow CM, Kahn KL, Merrick NJ et al. Does inappropriate use explain geographic variations in the use of health care services? A study of three procedures. JAMA 1987; 258: 2533–2537.
Goodman C, Baratz SR (eds). Improving Consensus Development for Health Technology Assessment: an International Perspective. Council on Health Care Technology. Inst. Med. Natl Academy Press: Washington, DC, 1990.
Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE et al. A method for the detailed assessment of the appropriateness of medical technologies. Intl J Tech Assess Health Care 1986; 2: 53–63.
Leape LL, Hilborne LH, Park RE, Bernstein SJ, Kamberg CJ, Sherwood M et al. The appropriateness of coronary artery bypass surgery in New York State. JAMA 1993; 269: 753–760.
Bernstein SJ, Hilborne LH, Leape LL, Fiske ME, Park RE, Kamberg CJ et al. The appropriateness of coronary artery bypass surgery in New York State. JAMA 1993; 269: 766–769.
Bernstein SJ, McGlynn EA, Siu AL, Roth CP, Sherwood MJ, Keesey JW et al. The appropriateness of hysterectomy; a comparison of seven health plans. JAMA 1993; 269: 2398–2402.
Bengtson A, Herliz J, Karlsson T, Brandrup-Wognsen G, Hjalmarson A . The appropriateness of performing coronary angioplasty and coronary revascularization in a Swedish population. JAMA 1994; 271: 1260–1265.
Gray D, Hampton JR, Bernstein SJ, Kosecoff J, Brook RH . Audit of coronary angiography and bypass surgery. Lancet 1990; 335: 1317–1320.
Pearson SD, Margolis CZ, Davis S, Schreier LK, Sokol HN, Gottlieb LK . Is consensus reproducible? A study of an algorithmic guidelines development process. Med Care 1995; 33: 643–660.
Shekelle PG, Chassin MR, Park RE . Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care 1998; 14: 707–727.
Kravitz RL, Laouri M, Kahan JP, Guzy P, Sherman T, Hilborne L et al. Validity of criteria used for detecting underuse of coronary revascularization. JAMA 1995; 274: 632–638.
Hahn T, Wolff SN, Czuczman M, Fisher RI, Lazarus HM, McCarthy PL et al. American Society for Blood and Marrow Transplantation (ASBMT). The role of cytotoxic therapy with hematopoietic stem cell transplantation in the treatment of diffuse large cell B-cell non-Hodgkin's lymphoma. Biol Blood Marrow Transplant 2003; 7: 308–331.
Hahn T, Wingard JR, Anderson KC, Bensinger WI, Berenson JR, Brozeit G et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of multiple myeloma: an evidence-based review. Biol Blood Marrow Transplant 2003; 9: 4–37.
Hahn T, Wall D, Camitta B, Davies S, Dillon H, Gaynon P et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute lymphoblastic leukemia in adults: an evidence-based review. Biol Blood Marrow Transplant 2006; 12: 1–30.
Hahn T, Wall D, Camitta B, Davies S, Dillon H, Gaynon P et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute lymphoblastic leukemia in children: an evidence-based review. Biol Blood Marrow Transplant 2005; 11: 823–861.
Oliansky DM, Appelbaum F, Cassileth PA, Keating A, Kerr J, Nieto Y et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute myelogenous leukemia in adults: an evidence-based review. Biol Blood Marrow Transplant 2008; 14: 137–180.
Oliansky DM, Rizzo JD, Aplan PD, Arceci RJ, Leone L, Ravindranath Y et al. The role of cytotoxic therapy with hematopoietic stem cell transplantation in the therapy of acute myeloid leukemia in children: an evidence-based review. Biol Blood Marrow Transplant 2007; 13: 1–25.
Gale RP, Park RE, Dubois RW, Bitran JD, Buzdar A, Hortobagyi G et al. Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in women with breast cancer. Clin Transplant 2000; 14: 32–41.
Gale RP, Park RE, Dubois RW, Herzig GP, Hocking WG, Horowitz MM et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in chronic myelogenous leukemia in chronic phase. Leuk Res 1998; 23: 817–826.
Gale RP, Park RE, Dubois RW, Herzig GP, Hocking WG, Horowitz MM et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute myelogenous leukemia in 1st remission. Leuk Res 1999; 23: 709–718.
Gale RP, Park RE, Dubois RW, Anderson KC, Audeh WM, Bergsagel D et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow autotransplants in newly diagnosed multiple myeloma. Leuk Lymphoma 1999; 33: 511–519.
Gale RP, Park RE, Dubois RW, Bierman PJ, Fisher RI, Loh KK et al. Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in diffuse large-cell lymphoma. Leuk Lymphoma 1998; 32: 139–149.
Gale RP, Park RE, Dubois RW, Herzig GP, Hocking WG, Horowitz MM et al. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute lymphoblastic leukemia in first remission. Leuk Res 1998; 22: 973–981.
Nathan PC, Sung L, Crump M, Beyene J . Consolidation therapy with autologous bone marrow transplantation in adults with acute myeloid leukemia: a meta-analysis. J Natl Cancer Inst 2004; 96: 38–45.
Levi I, Grotto I, Yerushalmi R, Ben-Bassat I, Shpilberg O . Meta-analysis of autologous bone marrow transplantation versus chemotherapy in adult patients with acute myeloid leukemia in first remission. Leuk Res 2004; 28: 605–612.
Breems DA, Boogaerts MA, Dekker AW, Van Putten WL, Sonneveld P, Huijgens PC et al. Autologous bone marrow transplantation as consolidation therapy in the treatment of adult patients under 60 years with acute myeloid leukaemia in first complete remission: a prospective randomized Dutch–Belgian Haemato-Oncology Co-operative Group (HOVON) and Swiss Group for Clinical Cancer Research (SAKK) trial. Br J Haematol 2005; 128: 59–65.
Dolnick E . Fish or foul?. New York Times, 2 September 2008. Opinions.
Karanicolas PJ, Montori VM, Devereaux PJ, Schuenemann H, Guyatt GH . A new ‘mechanic-practical’ framework for designing and interpreting randomized trials. J Clin Epidemiol; e-pub ahead of print, 9 May 2008.
We are grateful to our many colleagues at more than 150 transplant centers worldwide who, over the past 35 years, have challenged us to tackle these issues. We are especially indebted to our severest critics (you know who you are) who forced us to think harder about the use of ODBs to answer important clinical questions. Dr R Ed Park taught one of us (RPG) quite a lot about structured quantification of expert opinion. We shamelessly pirated the brilliant parody of Smith and Pell43 to illustrate some points. The late Drs Mortimer Bortin, Albert Rimm and their colleagues pioneered the concept of using ODBs to address issues in transplants; we dedicate this article to Mort. Mary M Horowitz, and John Klein continued and expanded Mort's legacy. Sabine Jacob kindly struggled through endless iterations of the typescript.
Rights and permissions
About this article
Cite this article
Gale, R., Eapen, M., Logan, B. et al. Are there roles for observational database studies and structured quantification of expert opinion to answer therapy controversies in transplants?. Bone Marrow Transplant 43, 435–446 (2009). https://doi.org/10.1038/bmt.2008.447
- observational databases
- randomized clinical trials
- expert opinion
This article is cited by
Statistical challenges in haematopoietic cell transplantation
Bone Marrow Transplantation (2022)
Transplant indications, guidelines and recommendations: Caveat Emptor
Bone Marrow Transplantation (2022)
Can guidelines inhibit innovation and critical thinking?
Bone Marrow Transplantation (2020)
Does a durian smell like a rose? The dangers of jargon
Bone Marrow Transplantation (2020)
Statistical analyses of clinical trials in haematopoietic cell transplantation or why there is a strong correlation between people drowning after falling out of a fishing boat and marriage rate in Kentucky
Bone Marrow Transplantation (2020)