Review Article | Published:

Meta-analysis and the science of research synthesis

Nature volume 555, pages 175182 (08 March 2018) | Download Citation

Abstract

Meta-analysis is the quantitative, scientific synthesis of research results. Since the term and modern approaches to research synthesis were first introduced in the 1970s, meta-analysis has had a revolutionary effect in many scientific fields, helping to establish evidence-based practice and to resolve seemingly contradictory research outcomes. At the same time, its implementation has engendered criticism and controversy, in some cases general and others specific to particular disciplines. Here we take the opportunity provided by the recent fortieth anniversary of meta-analysis to reflect on the accomplishments, limitations, recent advances and directions for future developments in the field of research synthesis.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    ., & in The Handbook of Meta-analysis in Ecology and Evolution (eds . et al.) Ch. 23, 364–380 (Princeton Univ. Press, 2013)

  2. 2.

    ., & Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 132, 409–423 (2006)

  3. 3.

    ., . & Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7, e1000326 (2010)

  4. 4.

    & in The Handbook of Research Synthesis and Meta-analysis 2nd edn (eds . et al.) 497–519 (Russell Sage Foundation, 2009)

  5. 5.

    The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 94, 485–514 (2016)

  6. 6.

    . & Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102, 828–844 (2014)

  7. 7.

    . & Making sense of meta-analysis: a critique of “effectiveness of long-term psychodynamic psychotherapy”. Clin. Soc. Work J. 39, 340–346 (2011)

  8. 8.

    Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J. Evol. Biol. 29, 1882–1904 (2016)

  9. 9.

    Meta-analyses and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology 91, 2522–2533 (2010)

  10. 10.

    . & Drug development: Raise standards for preclinical cancer research. Nature 483, 531–533 (2012); clarification 485, 41 (2012)

  11. 11.

    . & A critique for meta-analyses and the productivity-diversity relationship. Ecology 91, 2545–2549 (2010)

  12. 12.

    . et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6, e1000097 (2009).This paper provides a consensus regarding the reporting requirements for medical meta-analysis and has been highly influential in ensuring good reporting practice and standardizing language in evidence-based medicine, with further guidance for protocols, individual patient data meta-analyses and animal studies.

  13. 13.

    . et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4, 1 (2015)

  14. 14.

    . & Methodological issues and advances in biological meta-analysis. Evol. Ecol. 26, 1253–1274 (2012)

  15. 15.

    ., ., & Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 15, 18 (2017)

  16. 16.

    & Statistical Methods for Meta-analysis (Academic Press, 1985)

  17. 17.

    Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1–48 (2010)

  18. 18.

    . & Graphical displays for meta-analysis: an overview with suggestions for practice. Res. Synth. Methods 1, 66–80 (2010)

  19. 19.

    ., ., & Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315, 629–634 (1997)

  20. 20.

    . & Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, 455–463 (2000)

  21. 21.

    . & Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc. R. Soc. Lond. B 271, 1961–1966 (2004)

  22. 22.

    . & (eds) Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0 (Wiley, 2011).This large collaborative work provides definitive guidance for the production of systematic reviews in medicine and is of broad interest for methods development outside the medical field.

  23. 23.

    ., . & in The Handbook of Meta-analysis in Ecology and Evolution (eds . et al.) Ch. 25, 407–419 (Princeton Univ. Press, 2013)

  24. 24.

    ., ., & How to critically read ecological meta-analyses. Res. Synth. Methods 6, 124–133 (2015)

  25. 25.

    . & Synthesizing evidence: shifting the focus from individual studies to the body of evidence. J. Am. Med. Assoc. 309, 2217–2218 (2013)

  26. 26.

    ., ., ., . & Maternal obesity and risk of neural tube defects: a meta-analysis. Am. J. Obstet. Gynecol. 198, 611–619 (2008)

  27. 27.

    ., ., . & Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10–17. Cochrane Database Syst. Rev. (2005)

  28. 28.

    What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. Am. Psychol. 47, 1173–1181 (1992)

  29. 29.

    . et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013); erratum 14, 451 (2013)

  30. 30.

    . et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31, 711–719 (2016)

  31. 31.

    Meta-analysis in applied ecology. Biol. Lett. 6, 78–81 (2010)

  32. 32.

    ., ., . & The need for evidence-based conservation. Trends Ecol. Evol. 19, 305–308 (2004)

  33. 33.

    . et al. Biological invasions: a field synopsis, systematic review, and database of the literature. Ecol. Evol. 3, 182–196 (2013)

  34. 34.

    . & A globally coherent fingerprint of climate change impacts across natural systems. Nature 421, 37–42 (2003)

  35. 35.

    ., . & in The Handbook of Meta-analysis in Ecology and Evolution (eds . et al.) Ch. 24, 381–403 (Princeton Univ. Press, 2013)

  36. 36.

    . et al. Quantifying the evidence for biodiversity effects on ecosystem functioning and services. Ecol. Lett. 9, 1146–1156 (2006)

  37. 37.

    . et al. Effects of biodiversity on the functioning of trophic groups and ecosystems. Nature 443, 989–992 (2006)

  38. 38.

    ., ., & Enhancement of biodiversity and ecosystem services by ecological restoration: a meta-analysis. Science 325, 1121–1124 (2009)

  39. 39.

    ., ., & How general are positive relationships between plant population size, fitness and genetic variation? J. Ecol. 94, 942–952 (2006)

  40. 40.

    On the generality of the latitudinal diversity gradient. Am. Nat. 163, 192–211 (2004)

  41. 41.

    in The Handbook of Meta-analysis in Ecology and Evolution (eds . et al.) Ch. 19, 313–320 (Princeton Univ. Press, 2013)

  42. 42.

    . et al. A meta-analysis of the response of soil respiration, net nitrogen mineralization, and aboveground plant growth to experimental ecosystem warming. Oecologia 126, 543–562 (2001)

  43. 43.

    Phylogenetic meta-analysis. Evolution 62, 567–572 (2008)

  44. 44.

    . & General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J. Evol. Biol. 23, 494–508 (2010)

  45. 45.

    Meta-analysis and the comparative phylogenetic method. Am. Nat. 174, 369–381 (2009)

  46. 46.

    ., . & MetaWin: Statistical Software for Meta-Analysis with Resampling Tests Version 1 (Sinauer Associates, 1997)

  47. 47.

    . et al. OpenMEE: intuitive, open-source software for meta-analysis in ecology and evolutionary biology. Methods Ecol. Evol. 8, 941–947 (2016)

  48. 48.

    ., . & The interaction between competition and predation: a meta-analysis of field experiments. Am. Nat. 155, 435–453 (2000)

  49. 49.

    ., . & Resampling tests for meta-analysis of ecological data. Ecology 78, 1277–1283 (1997)

  50. 50.

    . & Statistical issues in ecological meta-analyses. Ecology 80, 1142–1149 (1999)

  51. 51.

    . & in The Handbook of Meta-analysis in Ecology and Evolution (eds . et al.) Ch. 11, 145–173 (Princeton Univ. Press, 2013)

  52. 52.

    Exercise in mega-silliness. Am. Psychol. 33, 517 (1978)

  53. 53.

    Rejoinder to: Don’t calculate effect sizes; study ecological effects. Ecol. Lett. 9, 921–922 (2006)

  54. 54.

    ., . & Gauging the impact of meta-analysis on ecology. Evol. Ecol. 26, 1153–1167 (2012)

  55. 55.

    ., & in The Handbook of Meta-analysis in Ecology and Evolution (eds . et al.) Ch. 15, 237–254 (Princeton Univ. Press, 2013)

  56. 56.

    ., ., ., . & The case of the misleading funnel plot. Br. Med. J. 333, 597–600 (2006)

  57. 57.

    ., & Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere 4, 1–24 (2013)

  58. 58.

    ., & in The Handbook of Meta-analysis in Ecology and Evolution (eds et al.) Ch. 16, 255–283 (Princeton Univ. Press, 2013)

  59. 59.

    ., & Relative citation impact of various study designs in the health sciences. J. Am. Med. Assoc. 293, 2362–2366 (2005)

  60. 60.

    . et al. Fame, glory and neglect in meta-analyses. Trends Ecol. Evol. 26, 493–494 (2011)

  61. 61.

    & Review Articles: The black-market of scientific currency. Ann. Entomol. Soc. Am. 110, 90 (2017)

  62. 62.

    & Data sharing. N. Engl. J. Med. 374, 276–277 (2016)

  63. 63.

    Scientific Method in Practice (Cambridge Univ. Press, 2003)

  64. 64.

    Science Staff. Dealing with data: introduction. Challenges and opportunities. Science 331, 692–693 (2011)

  65. 65.

    . et al. Promoting an open research culture. Science 348, 1422–1425 (2015)

  66. 66.

    . et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. J. Am. Med. Assoc. 313, 1657–1665 (2015)

  67. 67.

    . et al. Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Syst. Rev. 5, 196 (2016)

  68. 68.

    & Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J. Educ. Behav. Stat. 40, 604–634 (2015)

  69. 69.

    ., . & The potential for meta-analysis to support decision analysis in ecology. Res. Synth. Methods 6, 111–121 (2015)

  70. 70.

    Bayesian statistics in medicine: a 25 year review. Stat. Med. 25, 3589–3631 (2006)

  71. 71.

    . et al. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology 97, 3293–3299 (2016)

  72. 72.

    ., ., . & Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 356, 1228–1231 (2000)

  73. 73.

    ., . & (eds) The Handbook of Meta-Analysis in Ecology and Evolution (Princeton Univ. Press, 2013)This book provides the first comprehensive guide to undertaking meta-analyses in ecology and evolution and is also relevant to other fields where heterogeneity is expected, incorporating explicit consideration of the different approaches used in different domains.

  74. 74.

    Network meta-analysis for indirect treatment comparisons. Stat. Med. 21, 2313–2324 (2002)

  75. 75.

    . et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 15, 3 (2017)

  76. 76.

    . et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 11, e1001603 (2014)

  77. 77.

    ., . & Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews? BMC Med. 14, 59 (2016)

  78. 78.

    A meta-analytic study of the effects of female age on laying date and clutch size in the Great Tit Parus major and the Pied Flycatcher Ficedula hypoleuca. Ibis 133, 62–67 (1991)

  79. 79.

    & Meta-analysis: synthesizing research findings in ecology and evolution. Trends Ecol. Evol. 10, 236–240 (1995)

  80. 80.

    ., . & The meta-analysis of response ratios in experimental ecology. Ecology 80, 1150–1156 (1999)

  81. 81.

    ., & Meta-analysis in ecology. Adv. Ecol. Res 32, 199–247 (2001)

  82. 82.

    phyloMeta: a program for phylogenetic comparative analyses with meta-analysis. Bioinformatics 27, 2603–2604 (2011)

  83. 83.

    Report on certain enteric fever inoculation statistics. Br. Med. J. 2, 1243–1246 (1904)

  84. 84.

    Statistical Methods for Research Workers (Oliver and Boyd, 1925)

  85. 85.

    & The analysis of groups of experiments. J. Agric. Sci. 28, 556–580 (1938)

  86. 86.

    The combination of estimates from different experiments. Biometrics 10, 101–129 (1954)

  87. 87.

    . & Meta-analysis of psychotherapy outcome studies. Am. Psychol. 32, 752–760 (1977)

  88. 88.

    Meta-analysis at middle age: a personal history. Res. Synth. Methods 6, 221–231 (2015)

  89. 89.

    ., . & (eds) The Handbook of Research Synthesis and Meta-analysis 2nd edn (Russell Sage Foundation, 2009).This book is an important compilation that builds on the ground-breaking first edition to set the standard for best practice in meta-analysis, primarily in the social sciences but with applications to medicine and other fields.

  90. 90.

    Meta-analytic Procedures for Social Research (Sage, 1991)

  91. 91.

    ., & Meta-analysis: Cumulating Research Findings Across Studies (Sage, 1982)

  92. 92.

    ., ., . & A meta-analysis of competition in field experiments. Am. Nat. 140, 539–572 (1992).This influential early ecological meta-analysis reports multiple experimental outcomes on a longstanding and controversial topic that introduced a wide range of ecologists to research synthesis methods.

  93. 93.

    An historical perspective on meta-analysis: dealing quantitatively with varying study results. J. R. Soc. Med. 100, 579–582 (2007)

  94. 94.

    . & The meta-analytic big bang. Res. Synth. Methods 6, 246–264 (2015)

  95. 95.

    Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3–8 (1976)

  96. 96.

    . & Meta-analysis in clinical trials. Control. Clin. Trials 7, 177–188 (1986)

  97. 97.

    . & The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. Am. Psychol. 48, 1181–1209 (1993)

  98. 98.

    & Systematic Reviews (BMJ Publishing Group, 1995)

  99. 99.

    . et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet 354, 1896–1900 (1999)

  100. 100.

    & Quantifying heterogeneity in a meta-analysis. Stat. Med. 21, 1539–1558 (2002)

Download references

Acknowledgements

We dedicate this Review to the memory of Ingram Olkin and William Shadish, founding members of the Society for Research Synthesis Methodology who made tremendous contributions to the development of meta-analysis and research synthesis and to the supervision of generations of students. We thank L. Lagisz for help in preparing the figures. We are grateful to the Center for Open Science and the Laura and John Arnold Foundation for hosting and funding a workshop, which was the origination of this article. S.N. is supported by Australian Research Council Future Fellowship (FT130100268). J.G. acknowledges funding from the US National Science Foundation (ABI 1262402).

Author information

Affiliations

  1. Department of Ecology and Evolution, Stony Brook University, Stony Brook, New York 11794-5245, USA

    • Jessica Gurevitch
  2. School of Biological Sciences, Royal Holloway University of London, Egham, Surrey, TW20 0EX, UK

    • Julia Koricheva
  3. Evolution and Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, New South Wales 2052, Australia

    • Shinichi Nakagawa
  4. Diabetes and Metabolism Division, Garvan Institute of Medical Research, 384 Victoria Street, Darlinghurst, Sydney, New South Wales 2010, Australia

    • Shinichi Nakagawa
  5. School of Natural and Environmental Sciences, Newcastle University, Newcastle upon Tyne, NE1 7RU, UK

    • Gavin Stewart

Authors

  1. Search for Jessica Gurevitch in:

  2. Search for Julia Koricheva in:

  3. Search for Shinichi Nakagawa in:

  4. Search for Gavin Stewart in:

Contributions

All authors contributed equally in designing the study and writing the manuscript, and so are listed alphabetically.

Competing interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to Jessica Gurevitch or Julia Koricheva or Shinichi Nakagawa or Gavin Stewart.

Reviewer Information Nature thanks D. Altman, M. Lajeunesse, D. Moher and G. Romero for their contribution to the peer review of this work.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nature25753

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.