Abstract

Peer review is widely considered fundamental to maintaining the rigour of science, but it often fails to ensure transparency and reduce bias in published papers, and this systematically weakens the quality of published inferences. In part, this is because many reviewers are unaware of important questions to ask with respect to the soundness of the design and analyses, and the presentation of the methods and results; also some reviewers may expect others to be responsible for these tasks. We therefore present a reviewers’ checklist of ten questions that address these critical components. Checklists are commonly used by practitioners of other complex tasks, and we see great potential for the wider adoption of checklists for peer review, especially to reduce bias and facilitate transparency in published papers. We expect that such checklists will be well received by many reviewers.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Parker, T. H. et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31, 711–719 (2016).

  2. 2.

    TTEE Working Group Tools for Transparency in Ecology and Evolution (TTEE) (Open Science Framework, 2016); https://osf.io/g65cb/

  3. 3.

    Arriaga, A. F. et al. Simulation-based trial of surgical-crisis checklists. New Engl. J. Med. 368, 246–253 (2013).

  4. 4.

    Gawande, A. A. The Checklist Manifesto: How to Get Things Right (Metropolitan Books, New York, 2009).

  5. 5.

    Gerstner, K. et al. Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting. Methods Ecol. Evol. 8, 777–784 (2017).

  6. 6.

    Ferreira, V. et al. A meta-analysis of the effects of nutrient enrichment on litter decomposition in streams. Biol. Rev. 90, 669–688 (2015).

  7. 7.

    Fidler, F., Burgman, M. A., Cumming, G., Buttrose, R. & Thomason, N. Impact of criticism of null-hypothesis significance testing on statistical reporting practices in conservation biology. Conserv. Biol. 20, 1539–1544 (2006).

  8. 8.

    Zhang, Y., Chen, H. Y. H. & Reich, P. B. Forest productivity increases with evenness, species richness and trait variation: a global meta-analysis. J. Ecol. 100, 742–749 (2012).

  9. 9.

    Nickerson, R. S. Confirmation bias: a ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175–220 (1998).

  10. 10.

    Fischhoff, B. Hindsight not equal to foresight – effect of outcome knowledge on judgment under uncertainty. J. Exp. Psychol. Human. 1, 288–299 (1975).

  11. 11.

    Kozlov, M. V., Zverev, V. & Zvereva, E. L. Confirmation bias leads to overestimation of losses of woody plant foliage to insect herbivores in tropical regions. PeerJ 2, e709 (2014).

  12. 12.

    van Wilgenburg, E. & Elgar, M. A. Confirmation bias in studies of nestmate recognition: a cautionary note for research into the behaviour of animals. PLoS ONE 8, e53548 (2013).

  13. 13.

    Holman, L., Head, M. L., Lanfear, R. & Jennions, M. D. Evidence of experimental bias in the life sciences: why we need blind data recording. PLoS Biol. 13, e1002190 (2015).

  14. 14.

    Lee, C. J., Sugimoto, C. R., Zhang, G. & Cronin, B. Bias in peer review. Adv. Inf. Sci. 64, 2–17 (2013).

  15. 15.

    Mislan, K. A. S., Heer, J. M. & White, E. P. Elevating the status of code in ecology. Trends Ecol. Evol. 31, 4–7 (2016).

  16. 16.

    Fidler, F. et al. Metaresearch for evaluating reproducibility in ecology and evolution. BioScience 67, 282–289 (2017).

  17. 17.

    Parker, T. H. What do we really know about the signalling role of plumage colour in blue tits? A case study of impediments to progress in evolutionary biology. Biol. Rev. 88, 511–536 (2013).

  18. 18.

    Simmons, J. P., Nelson, L. D. & Simonsohn, U. A 21 word solution. Dialogue 26, 4–7 (2012).

  19. 19.

    Kardish, M. R. et al. Blind trust in unblinded observation in ecology, evolution and behavior. Front. Ecol. Evol. 3, 51 (2015).

  20. 20.

    Simmons, J. P., Nelson, L. D. & Simonsohn, U. False positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).

  21. 21.

    Forstmeier, W., Wagenmakers, E.-J. & Parker, T. H. Detecting and avoiding likely false-positive findings – a practical guide. Biol. Rev. 92, 1941–1968 (2017).Forstmeier et al. present insights that can help reviewers recognize and guide authors away from potentially biased and unreliable reporting.

  22. 22.

    Forstmeier, W. & Schielzeth, H. Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner’s curse. Behav. Ecol. Sociobiol. 65, 47–55 (2011).

  23. 23.

    Palmer, A. R. Quasireplication and the contract of error: lessons from sex ratios, heritabilities and fluctuating asymmetry. Annu. Rev. Ecol. Syst. 31, 441–480 (2000).

  24. 24.

    Halsey, L. G., Curran-Everett, D., Vowler, S. L. & Drummond, G. B. The fickle P value generates irreproducible results. Nat. Methods 12, 179–185 (2015).

  25. 25.

    Gelman, A. & Weakliem, D. Of beauty, sex, and power. Am. Sci. 97, 310–316 (2009).

  26. 26.

    Barto, E. K. & Rillig, M. C. Dissemination biases in ecology: effect sizes matter more than quality. Oikos 121, 228–235 (2012).Barto and Rillig provide evidence that various forms of bias, rather than concerns about data quality, have often influenced publication patterns in ecology.

  27. 27.

    Lemoine, N. P. et al. Underappreciated problems of low replication in ecological field studies. Ecology 97, 2554–2561 (2016).Lemoine et al. discuss how bias can emerge from low-powered studies, and also how bias can be avoided, even in systems where low power is inevitable due to logistical constraints.

  28. 28.

    Møller, A. P. & Jennions, M. D. How much variance can be explained by ecologists and evolutionary biologists? Oecologia 132, 492–500 (2002).

  29. 29.

    Duffy, J. E., Godwin, C. M. & Cardinale, B. J. Biodiversity effects in the wild are common and as strong as key drivers of productivity. Nature 549, 261–264 (2017).

  30. 30.

    Nakagawa, S. & Cuthill, I. C. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol. Rev. 82, 591–605 (2007).

  31. 31.

    Benjamin, D. J. et al. Redefine statistical significance. Nat. Hum. Behav. 2, 6–10 (2018).

Download references

Acknowledgements

We thank A. Moore for suggestions that improved the manuscript.

Author information

Affiliations

  1. Department of Biology, Whitman College, Walla Walla, WA, USA

    • Timothy H. Parker
  2. Department of Biological Sciences, Macquarie University, North Ryde, New South Wales, Australia

    • Timothy H. Parker
    •  & Simon C. Griffith
  3. Department of Ecology and Evolutionary Biology, University of Arizona, Tucson, AZ, USA

    • Judith L. Bronstein
  4. School of BioSciences, University of Melbourne, Melbourne, Victoria, Australia

    • Fiona Fidler
    •  & Hannah Fraser
  5. History & Philosophy of Science, School of Historical & Philosophical Studies, University of Melbourne, Melbourne, Victoria, Australia

    • Fiona Fidler
  6. Department of Biology, Clark University, Worcester, MA, USA

    • Susan Foster
  7. Department of Behavioural Ecology and Evolutionary Genetics, Max Planck Institute for Ornithology, Seewiesen, Germany

    • Wolfgang Forstmeier
  8. Department of Ecology and Evolution, Stony Brook University, Stony Brook, NY, USA

    • Jessica Gurevitch
  9. School of Biological Sciences, Royal Holloway University of London, Egham, Surrey, UK

    • Julia Koricheva
  10. Department of Computational Landscape Ecology, UFZ – Helmholtz Centre for Environmental Research, Leipzig, Germany

    • Ralf Seppelt
  11. Institute of Geoscience and Geography, Martin-Luther-University Halle-Wittenberg, Halle (Saale), Germany

    • Ralf Seppelt
  12. iDiv – German Centre for Integrative Biodiversity Research Halle-Jena-Leipzig, Leipzig, Germany

    • Ralf Seppelt
  13. Department of Ecology and Evolutionary Biology, University of Connecticut, Storrs, CT, USA

    • Morgan W. Tingley
  14. Evolution & Ecology Research Centre, School of Biological, Earth and Environmental Sciences, University of New South Wales, Randwick, New South Wales, Australia

    • Shinichi Nakagawa

Authors

  1. Search for Timothy H. Parker in:

  2. Search for Simon C. Griffith in:

  3. Search for Judith L. Bronstein in:

  4. Search for Fiona Fidler in:

  5. Search for Susan Foster in:

  6. Search for Hannah Fraser in:

  7. Search for Wolfgang Forstmeier in:

  8. Search for Jessica Gurevitch in:

  9. Search for Julia Koricheva in:

  10. Search for Ralf Seppelt in:

  11. Search for Morgan W. Tingley in:

  12. Search for Shinichi Nakagawa in:

Contributions

T.H.P. composed the original draft of this manuscript in consultation with S.C.G. and S.N. S.N. made the figure. The manuscript was edited substantially over multiple rounds with input from all co-authors.

Competing interests

The authors declare no competing interests.

Corresponding author

Correspondence to Timothy H. Parker.

Supplementary information

  1. Supplementary Information

    Supplementary Text

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41559-018-0545-z