Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Eight problems with literature reviews and how to fix them

Nature Ecology & Evolution volume 4, pages 1582–1589 (2020)Cite this article

Subjects

An Author Correction to this article was published on 19 October 2020

This article has been updated

Abstract

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a ‘critical friend’ role in supporting would-be systematic reviews and avoiding individual responses to police use of the ‘systematic review’ label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematic showing the main stages necessary for the conduct of a systematic review as defined by the Collaboration for Environmental Evidence.

Similar content being viewed by others

Change history

  • 19 October 2020

    An amendment to this paper has been published and can be accessed via a link at the top of the paper.

References

  1. Grant, M. J. & Booth, A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr. J. 26, 91–108 (2009).

    PubMed  Google Scholar 

  2. Haddaway, N. R. & Macura, B. The role of reporting standards in producing robust literature reviews. Nat. Clim. Change 8, 444–447 (2018).

    Google Scholar 

  3. Pullin, A. S. & Knight, T. M. Science informing policy–a health warning for the environment. Environ. Evid. 1, 15 (2012).

    Google Scholar 

  4. Haddaway, N., Woodcock, P., Macura, B. & Collins, A. Making literature reviews more reliable through application of lessons from systematic reviews. Conserv. Biol. 29, 1596–1605 (2015).

    CAS  PubMed  Google Scholar 

  5. Pullin, A., Frampton, G., Livoreil, B. & Petrokofsky, G. Guidelines and Standards for Evidence Synthesis in Environmental Management (Collaboration for Environmental Evidence, 2018).

  6. White, H. The twenty-first century experimenting society: the four waves of the evidence revolution. Palgrave Commun. 5, 47 (2019).

    Google Scholar 

  7. O’Leary, B. C. et al. The reliability of evidence review methodology in environmental science and conservation. Environ. Sci. Policy 64, 75–82 (2016).

    Google Scholar 

  8. Woodcock, P., Pullin, A. S. & Kaiser, M. J. Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: a methodology. Biol. Conserv. 176, 54–62 (2014).

    Google Scholar 

  9. Campbell Systematic Reviews: Policies and Guidelines (Campbell Collaboration, 2014).

  10. Higgins, J. P. et al. Cochrane Handbook for Systematic Reviews of Interventions (John Wiley & Sons, 2019).

  11. Shea, B. J. et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358, j4008 (2017).

    PubMed  PubMed Central  Google Scholar 

  12. Haddaway, N. R., Land, M. & Macura, B. “A little learning is a dangerous thing”: a call for better understanding of the term ‘systematic review’. Environ. Int. 99, 356–360 (2017).

    PubMed  Google Scholar 

  13. Freeman, R. E. Strategic Management: A Stakeholder Approach (Cambridge Univ. Press, 2010).

  14. Haddaway, N. R. et al. A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environ. Evid. 6, 11 (2017).

    Google Scholar 

  15. Land, M., Macura, B., Bernes, C. & Johansson, S. A five-step approach for stakeholder engagement in prioritisation and planning of environmental evidence syntheses. Environ. Evid. 6, 25 (2017).

    Google Scholar 

  16. Oliver, S. & Dickson, K. Policy-relevant systematic reviews to strengthen health systems: models and mechanisms to support their production. Evid. Policy 12, 235–259 (2016).

    Google Scholar 

  17. Savilaakso, S. et al. Systematic review of effects on biodiversity from oil palm production. Environ. Evid. 3, 4 (2014).

    Google Scholar 

  18. Savilaakso, S., Laumonier, Y., Guariguata, M. R. & Nasi, R. Does production of oil palm, soybean, or jatropha change biodiversity and ecosystem functions in tropical forests. Environ. Evid. 2, 17 (2013).

    Google Scholar 

  19. Haddaway, N. R. & Crowe, S. Experiences and lessons in stakeholder engagement in environmental evidence synthesis: a truly special series. Environ. Evid. 7, 11 (2018).

    Google Scholar 

  20. Sánchez-Bayo, F. & Wyckhuys, K. A. Worldwide decline of the entomofauna: a review of its drivers. Biol. Conserv. 232, 8–27 (2019).

    Google Scholar 

  21. Agarwala, M. & Ginsberg, J. R. Untangling outcomes of de jure and de facto community-based management of natural resources. Conserv. Biol. 31, 1232–1246 (2017).

    PubMed  Google Scholar 

  22. Gurevitch, J., Curtis, P. S. & Jones, M. H. Meta-analysis in ecology. Adv. Ecol. Res. 32, 199–247 (2001).

    CAS  Google Scholar 

  23. Haddaway, N. R., Macura, B., Whaley, P. & Pullin, A. S. ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ. Evid. 7, 7 (2018).

    Google Scholar 

  24. Lwasa, S. et al. A meta-analysis of urban and peri-urban agriculture and forestry in mediating climate change. Curr. Opin. Environ. Sustain. 13, 68–73 (2015).

    Google Scholar 

  25. Pacifici, M. et al. Species’ traits influenced their response to recent climate change. Nat. Clim. Change 7, 205–208 (2017).

    Google Scholar 

  26. Owen-Smith, N. Ramifying effects of the risk of predation on African multi-predator, multi-prey large-mammal assemblages and the conservation implications. Biol. Conserv. 232, 51–58 (2019).

    Google Scholar 

  27. Prugh, L. R. et al. Designing studies of predation risk for improved inference in carnivore-ungulate systems. Biol. Conserv. 232, 194–207 (2019).

    Google Scholar 

  28. Li, Y. et al. Effects of biochar application in forest ecosystems on soil properties and greenhouse gas emissions: a review. J. Soil Sediment. 18, 546–563 (2018).

    CAS  Google Scholar 

  29. Moher, D., Liberati, A., Tetzlaff, J. & Altman, D. G., The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6, e1000097 (2009).

    PubMed  PubMed Central  Google Scholar 

  30. Bernes, C. et al. What is the influence of a reduction of planktivorous and benthivorous fish on water quality in temperate eutrophic lakes? A systematic review. Environ. Evid. 4, 7 (2015).

    Google Scholar 

  31. McDonagh, M., Peterson, K., Raina, P., Chang, S. & Shekelle, P. Avoiding bias in selecting studies. Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet] (Agency for Healthcare Research and Quality, 2013).

  32. Burivalova, Z., Hua, F., Koh, L. P., Garcia, C. & Putz, F. A critical comparison of conventional, certified, and community management of tropical forests for timber in terms of environmental, economic, and social variables. Conserv. Lett. 10, 4–14 (2017).

    Google Scholar 

  33. Min-Venditti, A. A., Moore, G. W. & Fleischman, F. What policies improve forest cover? A systematic review of research from Mesoamerica. Glob. Environ. Change 47, 21–27 (2017).

    Google Scholar 

  34. Bramer, W. M., Giustini, D. & Kramer, B. M. R. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst. Rev. 5, 39 (2016).

    PubMed  PubMed Central  Google Scholar 

  35. Bramer, W. M., Giustini, D., Kramer, B. M. R. & Anderson, P. F. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst. Rev. 2, 115 (2013).

    PubMed  PubMed Central  Google Scholar 

  36. Gusenbauer, M. & Haddaway, N. R. Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods 11, 181–217 (2020).

    PubMed  PubMed Central  Google Scholar 

  37. Livoreil, B. et al. Systematic searching for environmental evidence using multiple tools and sources. Environ. Evid. 6, 23 (2017).

    Google Scholar 

  38. Mlinarić, A., Horvat, M. & Šupak Smolčić, V. Dealing with the positive publication bias: why you should really publish your negative results. Biochem. Med. 27, 447–452 (2017).

    Google Scholar 

  39. Lin, L. & Chu, H. Quantifying publication bias in meta‐analysis. Biometrics 74, 785–794 (2018).

    PubMed  Google Scholar 

  40. Haddaway, N. R. & Bayliss, H. R. Shades of grey: two forms of grey literature important for reviews in conservation. Biol. Conserv. 191, 827–829 (2015).

    Google Scholar 

  41. Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1–48 (2010).

    Google Scholar 

  42. Bilotta, G. S., Milner, A. M. & Boyd, I. On the use of systematic reviews to inform environmental policies. Environ. Sci. Policy 42, 67–77 (2014).

    Google Scholar 

  43. Englund, G., Sarnelle, O. & Cooper, S. D. The importance of data‐selection criteria: meta‐analyses of stream predation experiments. Ecology 80, 1132–1141 (1999).

    Google Scholar 

  44. Burivalova, Z., Şekercioğlu, Ç. H. & Koh, L. P. Thresholds of logging intensity to maintain tropical forest biodiversity. Curr. Biol. 24, 1893–1898 (2014).

    CAS  PubMed  Google Scholar 

  45. Bicknell, J. E., Struebig, M. J., Edwards, D. P. & Davies, Z. G. Improved timber harvest techniques maintain biodiversity in tropical forests. Curr. Biol. 24, R1119–R1120 (2014).

    CAS  PubMed  Google Scholar 

  46. Damette, O. & Delacote, P. Unsustainable timber harvesting, deforestation and the role of certification. Ecol. Econ. 70, 1211–1219 (2011).

    Google Scholar 

  47. Blomley, T. et al. Seeing the wood for the trees: an assessment of the impact of participatory forest management on forest condition in Tanzania. Oryx 42, 380–391 (2008).

    Google Scholar 

  48. Haddaway, N. R. et al. How does tillage intensity affect soil organic carbon? A systematic review. Environ. Evid. 6, 30 (2017).

    Google Scholar 

  49. Higgins, J. P. et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343, d5928 (2011).

    PubMed  PubMed Central  Google Scholar 

  50. Stewart, G. Meta-analysis in applied ecology. Biol. Lett. 6, 78–81 (2010).

    PubMed  Google Scholar 

  51. Koricheva, J. & Gurevitch, J. Uses and misuses of meta‐analysis in plant ecology. J. Ecol. 102, 828–844 (2014).

    Google Scholar 

  52. Vetter, D., Ruecker, G. & Storch, I. Meta‐analysis: a need for well‐defined usage in ecology and conservation biology. Ecosphere 4, 1–24 (2013).

    Google Scholar 

  53. Stewart, G. B. & Schmid, C. H. Lessons from meta-analysis in ecology and evolution: the need for trans-disciplinary evidence synthesis methodologies. Res. Synth. Methods 6, 109–110 (2015).

    PubMed  Google Scholar 

  54. Macura, B. et al. Systematic reviews of qualitative evidence for environmental policy and management: an overview of different methodological options. Environ. Evid. 8, 24 (2019).

    Google Scholar 

  55. Koricheva, J. & Gurevitch, J. in Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J. et al.) Ch. 1 (Princeton Scholarship Online, 2013).

  56. Britt, M., Haworth, S. E., Johnson, J. B., Martchenko, D. & Shafer, A. B. The importance of non-academic coauthors in bridging the conservation genetics gap. Biol. Conserv. 218, 118–123 (2018).

    Google Scholar 

  57. Graham, L., Gaulton, R., Gerard, F. & Staley, J. T. The influence of hedgerow structural condition on wildlife habitat provision in farmed landscapes. Biol. Conserv. 220, 122–131 (2018).

    Google Scholar 

  58. Delaquis, E., de Haan, S. & Wyckhuys, K. A. On-farm diversity offsets environmental pressures in tropical agro-ecosystems: a synthetic review for cassava-based systems. Agric. Ecosyst. Environ. 251, 226–235 (2018).

    Google Scholar 

  59. Popay, J. et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product from the ESRC Methods Programme Version 1 (Lancaster Univ., 2006).

  60. Pullin, A. S. et al. Human well-being impacts of terrestrial protected areas. Environ. Evid. 2, 19 (2013).

    Google Scholar 

  61. Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S. & Pieper, D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med. Res. Methodol. 19, 132 (2019).

    PubMed  PubMed Central  Google Scholar 

  62. Rallo, A. & García-Arberas, L. Differences in abiotic water conditions between fluvial reaches and crayfish fauna in some northern rivers of the Iberian Peninsula. Aquat. Living Resour. 15, 119–128 (2002).

    Google Scholar 

  63. Glasziou, P. & Chalmers, I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ 363, k4645 (2018).

    Google Scholar 

  64. Haddaway, N. R. Open Synthesis: on the need for evidence synthesis to embrace Open Science. Environ. Evid. 7, 26 (2018).

    Google Scholar 

Download references

Acknowledgements

We thank C. Shortall from Rothamstead Research for useful discussions on the topic.

Author information

Authors and Affiliations

  1. Mercator Research Institute on Climate Change and Global Commons, Berlin, Germany

    Neal R. Haddaway

  2. Stockholm Environment Institute, Stockholm, Sweden

    Neal R. Haddaway & Biljana Macura

  3. Africa Centre for Evidence, University of Johannesburg, Johannesburg, South Africa

    Neal R. Haddaway

  4. College of Medicine and Health, Exeter University, Exeter, UK

    Alison Bethel

  5. Department of Zoology, University of Cambridge, Cambridge, UK

    Lynn V. Dicks

  6. School of Biological Sciences, University of East Anglia, Norwich, UK

    Lynn V. Dicks

  7. Department of Biological Sciences, Royal Holloway University of London, Egham, UK

    Julia Koricheva

  8. Department of Zoology, University of Oxford, Oxford, UK

    Gillian Petrokofsky

  9. Collaboration for Environmental Evidence, UK Centre, School of Natural Sciences, Bangor University, Bangor, UK

    Andrew S. Pullin

  10. Liljus ltd, London, UK

    Sini Savilaakso

  11. Department of Forest Sciences, University of Helsinki, Helsinki, Finland

    Sini Savilaakso

  12. Evidence Synthesis Lab, School of Natural and Environmental Sciences, University of Newcastle, Newcastle-upon-Tyne, UK

    Gavin B. Stewart

Authors
  1. Neal R. Haddaway
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alison Bethel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lynn V. Dicks
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julia Koricheva
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Biljana Macura
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gillian Petrokofsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Andrew S. Pullin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sini Savilaakso
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Gavin B. Stewart
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

N.R.H. developed the manuscript idea and a first draft. All authors contributed to examples and edited the text. All authors have read and approve of the final submission.

Corresponding author

Correspondence to Neal R. Haddaway.

Ethics declarations

Competing interests

S.S. is a co-founder of Liljus ltd, a firm that provides research services in sustainable finance as well as forest conservation and management. The other authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Table

Examples of literature reviews and common problems identified.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haddaway, N.R., Bethel, A., Dicks, L.V. et al. Eight problems with literature reviews and how to fix them. Nat Ecol Evol 4, 1582–1589 (2020). https://doi.org/10.1038/s41559-020-01295-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41559-020-01295-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing Anthropocene

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: Anthropocene