Peer review is widely considered fundamental to maintaining the rigour of science, but it often fails to ensure transparency and reduce bias in published papers, and this systematically weakens the quality of published inferences. In part, this is because many reviewers are unaware of important questions to ask with respect to the soundness of the design and analyses, and the presentation of the methods and results; also some reviewers may expect others to be responsible for these tasks. We therefore present a reviewers’ checklist of ten questions that address these critical components. Checklists are commonly used by practitioners of other complex tasks, and we see great potential for the wider adoption of checklists for peer review, especially to reduce bias and facilitate transparency in published papers. We expect that such checklists will be well received by many reviewers.
Subscribe to Journal
Get full journal access for 1 year
only $9.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Parker, T. H. et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31, 711–719 (2016).
TTEE Working Group Tools for Transparency in Ecology and Evolution (TTEE) (Open Science Framework, 2016); https://osf.io/g65cb/
Arriaga, A. F. et al. Simulation-based trial of surgical-crisis checklists. New Engl. J. Med. 368, 246–253 (2013).
Gawande, A. A. The Checklist Manifesto: How to Get Things Right (Metropolitan Books, New York, 2009).
Gerstner, K. et al. Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting. Methods Ecol. Evol. 8, 777–784 (2017).
Ferreira, V. et al. A meta-analysis of the effects of nutrient enrichment on litter decomposition in streams. Biol. Rev. 90, 669–688 (2015).
Fidler, F., Burgman, M. A., Cumming, G., Buttrose, R. & Thomason, N. Impact of criticism of null-hypothesis significance testing on statistical reporting practices in conservation biology. Conserv. Biol. 20, 1539–1544 (2006).
Zhang, Y., Chen, H. Y. H. & Reich, P. B. Forest productivity increases with evenness, species richness and trait variation: a global meta-analysis. J. Ecol. 100, 742–749 (2012).
Nickerson, R. S. Confirmation bias: a ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175–220 (1998).
Fischhoff, B. Hindsight not equal to foresight – effect of outcome knowledge on judgment under uncertainty. J. Exp. Psychol. Human. 1, 288–299 (1975).
Kozlov, M. V., Zverev, V. & Zvereva, E. L. Confirmation bias leads to overestimation of losses of woody plant foliage to insect herbivores in tropical regions. PeerJ 2, e709 (2014).
van Wilgenburg, E. & Elgar, M. A. Confirmation bias in studies of nestmate recognition: a cautionary note for research into the behaviour of animals. PLoS ONE 8, e53548 (2013).
Holman, L., Head, M. L., Lanfear, R. & Jennions, M. D. Evidence of experimental bias in the life sciences: why we need blind data recording. PLoS Biol. 13, e1002190 (2015).
Lee, C. J., Sugimoto, C. R., Zhang, G. & Cronin, B. Bias in peer review. Adv. Inf. Sci. 64, 2–17 (2013).
Mislan, K. A. S., Heer, J. M. & White, E. P. Elevating the status of code in ecology. Trends Ecol. Evol. 31, 4–7 (2016).
Fidler, F. et al. Metaresearch for evaluating reproducibility in ecology and evolution. BioScience 67, 282–289 (2017).
Parker, T. H. What do we really know about the signalling role of plumage colour in blue tits? A case study of impediments to progress in evolutionary biology. Biol. Rev. 88, 511–536 (2013).
Simmons, J. P., Nelson, L. D. & Simonsohn, U. A 21 word solution. Dialogue 26, 4–7 (2012).
Kardish, M. R. et al. Blind trust in unblinded observation in ecology, evolution and behavior. Front. Ecol. Evol. 3, 51 (2015).
Simmons, J. P., Nelson, L. D. & Simonsohn, U. False positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).
Forstmeier, W., Wagenmakers, E.-J. & Parker, T. H. Detecting and avoiding likely false-positive findings – a practical guide. Biol. Rev. 92, 1941–1968 (2017).Forstmeier et al. present insights that can help reviewers recognize and guide authors away from potentially biased and unreliable reporting.
Forstmeier, W. & Schielzeth, H. Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner’s curse. Behav. Ecol. Sociobiol. 65, 47–55 (2011).
Palmer, A. R. Quasireplication and the contract of error: lessons from sex ratios, heritabilities and fluctuating asymmetry. Annu. Rev. Ecol. Syst. 31, 441–480 (2000).
Halsey, L. G., Curran-Everett, D., Vowler, S. L. & Drummond, G. B. The fickle P value generates irreproducible results. Nat. Methods 12, 179–185 (2015).
Gelman, A. & Weakliem, D. Of beauty, sex, and power. Am. Sci. 97, 310–316 (2009).
Barto, E. K. & Rillig, M. C. Dissemination biases in ecology: effect sizes matter more than quality. Oikos 121, 228–235 (2012).Barto and Rillig provide evidence that various forms of bias, rather than concerns about data quality, have often influenced publication patterns in ecology.
Lemoine, N. P. et al. Underappreciated problems of low replication in ecological field studies. Ecology 97, 2554–2561 (2016).Lemoine et al. discuss how bias can emerge from low-powered studies, and also how bias can be avoided, even in systems where low power is inevitable due to logistical constraints.
Møller, A. P. & Jennions, M. D. How much variance can be explained by ecologists and evolutionary biologists? Oecologia 132, 492–500 (2002).
Duffy, J. E., Godwin, C. M. & Cardinale, B. J. Biodiversity effects in the wild are common and as strong as key drivers of productivity. Nature 549, 261–264 (2017).
Nakagawa, S. & Cuthill, I. C. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol. Rev. 82, 591–605 (2007).
Benjamin, D. J. et al. Redefine statistical significance. Nat. Hum. Behav. 2, 6–10 (2018).
We thank A. Moore for suggestions that improved the manuscript.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Parker, T.H., Griffith, S.C., Bronstein, J.L. et al. Empowering peer reviewers with a checklist to improve transparency. Nat Ecol Evol 2, 929–935 (2018). https://doi.org/10.1038/s41559-018-0545-z
Scientific Reports (2021)
Spinal Cord (2021)