Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

Nature Ecology & Evolution volume 7, pages 1525–1536 (2023)Cite this article

Subjects

Abstract

In many scientific disciplines, common research practices have led to unreliable and exaggerated evidence about scientific phenomena. Here we describe some of these practices and quantify their pervasiveness in recent ecology publications in five popular journals. In an analysis of over 350 studies published between 2018 and 2020, we detect empirical evidence of exaggeration bias and selective reporting of statistically significant results. This evidence implies that the published effect sizes in ecology journals exaggerate the importance of the ecological relationships that they aim to quantify. An exaggerated evidence base hinders the ability of empirical ecology to reliably contribute to science, policy, and management. To increase the credibility of ecology research, we describe a set of actions that ecologists should take, including changes to scientific norms about what high-quality ecology looks like and expectations about what high-quality studies can deliver.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Percentage of statistical tests that meet and do not meet the conventional 0.8 threshold for statistical power.
Fig. 2: The percentage of under-powered estimates from ecological studies that are exaggerated.
Fig. 3: Evidence of selective reporting of statistically significant results.
Fig. 4: The percentage of ecology studies that use multiple hypothesis testing.
Fig. 5: Ecology studies that have data available and provide code for their analyses.

Similar content being viewed by others

Data availability

Our dataset is available at https://osf.io/9yd2b.

Code availability

Our analysis code is available at https://osf.io/9yd2b.

References

  1. Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012).

    Article  PubMed  Google Scholar 

  2. Leimu, R. & Koricheva, J. Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc. R. Soc. B 271, 1961–1966 (2004).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Møller, A. P. & Jennions, M. D. Testing and adjusting for publication bias. Trends Ecol. Evol. 16, 580–586 (2001).

    Article  Google Scholar 

  4. Barto, E. K. & Rillig, M. C. Dissemination biases in ecology: effect sizes matter more than quality. Oikos 121, 228–235 (2012).

    Article  Google Scholar 

  5. Christensen, G. & Miguel, E. Transparency, reproducibility, and the credibility of economics research. J. Econ. Lit. 56, 920–980 (2018).

    Article  Google Scholar 

  6. Collaboration, O. S. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

    Article  Google Scholar 

  7. Ferraro, P. J. & Shukla, P. Is a replicability crisis on the horizon for environmental and resource economics? Rev. Environ. Econ. Policy 14, 339–351 (2020).

    Article  Google Scholar 

  8. Martinson, B. C., Anderson, M. S. & de Vries, R. Scientists behaving badly. Nature 435, 737–738 (2005).

    Article  CAS  PubMed  Google Scholar 

  9. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, 696–701 (2005).

    Article  Google Scholar 

  10. Fraser, H., Parker, T., Nakagawa, S., Barnett, A. & Fidler, F. Questionable research practices in ecology and evolution. PLoS ONE 13, e0200303 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Fraser, H., Barnett, A., Parker, T. H. & Fidler, F. The role of replication studies in ecology. Ecol. Evol. 10, 5197–5207 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  12. Fidler, F. et al. Metaresearch for evaluating reproducibility in ecology and evolution. Bioscience 67, 282–289 (2017).

    PubMed  PubMed Central  Google Scholar 

  13. Cassey, P. & Blackburn, T. M. Reproducibility and repeatability in ecology. Bioscience 56, 958–959 (2006).

    Article  Google Scholar 

  14. Parker, T. H. et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31, 711–719 (2016).

    Article  PubMed  Google Scholar 

  15. Ioannidis, J. P. A., Stanley, T. D. & Doucouliagos, H. The power of bias in economics research. Econ. J. 127, F236–F265 (2017).

    Article  Google Scholar 

  16. Jennions, M. D. & Møller, A. P. A survey of the statistical power of research in behavioral ecology and animal behavior. Behav. Ecol. 14, 438–445 (2003).

    Article  Google Scholar 

  17. Lemoine, N. P. et al. Underappreciated problems of low replication in ecological field studies. Ecology 97, 2562–2569 (2016).

    Article  Google Scholar 

  18. Yang, Y. et al. Publication bias impacts on effect size, statistical power, and magnitude (type M) and sign (type S) errors in ecology and evolutionary biology. BMC Bio. 21, 71 (2023).

    Article  Google Scholar 

  19. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  PubMed  Google Scholar 

  20. Fidler, F., Burgman, M. A., Cumming, G., Buttrose, R. & Thomason, N. Impact of criticism of null-hypothesis significance testing on statistical reporting practices in conservation biology. Conserv. Biol. 20, 1539–1544 (2006).

    Article  PubMed  Google Scholar 

  21. Gelman, A. & Carlin, J. Beyond power calculations: assessing type S (sign) and type M (magnitude) errors. Perspect. Psychol. Sci. 9, 641–651 (2014).

    Article  PubMed  Google Scholar 

  22. Nichols, J. D., Oli, M. K., Kendall, W. L. & Scott Boomer, G. A better approach for dealing with reproducibility and replicability in science. Proc. Natl Acad. Sci. USA 118, 1–5 (2021).

    Article  Google Scholar 

  23. Koricheva, J. Non-significant results in ecology: a burden or a blessing in disguise? Oikos 102, 397–401 (2003).

    Article  Google Scholar 

  24. Ceausu, I. et al. High impact journals in ecology cover proportionally more statistically significant findings. Preprint at bioRxiv https://doi.org/10.1093/sw/38.6.771 (2018).

  25. Nichols, J. D., Kendall, W. L. & Boomer, G. S. Accumulating evidence in ecology: once is not enough. Ecol. Evol. 9, 13991–14004 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012).

    Article  Google Scholar 

  27. Fanelli, D. Is science really facing a reproducibility crisis, and do we need it to? Proc. Natl Acad. Sci. USA 115, 2628–2631 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Yoccoz, N. G. Use, overuse, and misuse of significance tests in evolutionary biology and ecology. Bull. Ecol. Soc. Am. 72, 106–111 (1991).

    Article  Google Scholar 

  29. Fidler, F., Fraser, H., McCarthy, M. A. & Game, E. T. Improving the transparency of statistical reporting in Conservation Letters. Conserv. Lett. 11, 1–5 (2018).

    Article  Google Scholar 

  30. Murtaugh, P. A. In defense of P values. Ecology 95, 611–617 (2014).

    Article  PubMed  Google Scholar 

  31. Anderson, D. R., Burnham, K. P. & Thompson, W. L. Null hypothesis testing: problems, prevalence, and an alternative. J. Wildl. Manag. 64, 912–923 (2000).

    Article  Google Scholar 

  32. Callaham, M., Wears, R. L. & Weber, E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. J. Am. Med. Assoc. 287, 2847–2850 (2002).

    Article  Google Scholar 

  33. Brodeur, A., Lé, M., Sangnier, M. & Zylberberg, Y. Star wars: the empirics strike back. Am. Econ. J. Appl. Econ. 8, 1–32 (2016).

    Article  Google Scholar 

  34. Gopalakrishna, G. et al. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: a survey among academic researchers in the Netherlands. PLoS ONE 17, 1–16 (2022).

    Article  Google Scholar 

  35. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).

    Article  PubMed  Google Scholar 

  36. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T. & Jennions, M. D. The extent and consequences of P-hacking in science. PLoS Biol. 13, 1–15 (2015).

    Article  CAS  Google Scholar 

  37. Hartgerink, C. H. J., Van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M. & Van Assen, M. A. L. M. Distributions of p-values smaller than .05 in psychology: what is going on? PeerJ 2016, e1935 (2016).

    Article  Google Scholar 

  38. Shaffer, J. P. Multiple hypothesis testing. Annu. Rev. Psychol. 46, 561–584 (1995).

    Article  Google Scholar 

  39. Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B 57, 289–300 (1995).

    Google Scholar 

  40. Dunnett, C. W. A multiple comparison procedure for comparing several treatments with a control. J. Am. Stat. Assoc. 50, 1096–1121 (1955).

    Article  Google Scholar 

  41. Yekutieli, D. & Benjamini, Y. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Stat. Plan. Inference 82, 171–196 (1999).

    Article  Google Scholar 

  42. Berry, D. A. & Hochberg, Y. Bayesian perspectives on multiple comparisons. J. Stat. Plan. Inference 82, 215–227 (1999).

    Article  Google Scholar 

  43. Gelman, A., Hill, J. & Yajima, M. Why we (usually) don’t have to worry about multiple comparisons. J. Res. Educ. Eff. 5, 189–211 (2012).

    Google Scholar 

  44. Rubin, M. Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Rev. Gen. Psychol. 21, 269–275 (2017).

    Article  Google Scholar 

  45. Rubin, M. When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Rev. Gen. Psychol. 21, 308–320 (2017).

    Article  Google Scholar 

  46. Nakagawa, S. A farewell to Bonferroni: the problems of low statistical power and publication bias. Behav. Ecol. 15, 1044–1045 (2004).

    Article  Google Scholar 

  47. Forstmeier, W., Wagenmakers, E. J. & Parker, T. H. Detecting and avoiding likely false-positive findings—a practical guide. Biol. Rev. 92, 1941–1968 (2017).

    Article  PubMed  Google Scholar 

  48. Baker, M. & Penny, D. Is there a reproducibility crisis? Nature 533, 452–454 (2016).

    Article  CAS  PubMed  Google Scholar 

  49. Gelman, A. & Loken, E. The statistical crisis in science. Am. Sci. 102, 460–465 (2014).

    Article  Google Scholar 

  50. O’Dea, R. E. et al. Towards open, reliable, and transparent ecology and evolutionary biology. BMC Biol. 19, 1–5 (2021).

    Article  Google Scholar 

  51. Parker, T. H., Nakagawa, S. & Gurevitch, J. Promoting transparency in evolutionary biology and ecology. Ecol. Lett. 19, 726–728 (2016).

    Article  CAS  PubMed  Google Scholar 

  52. Parker, T., Fraser, H. & Nakagawa, S. Making conservation science more reliable with preregistration and registered reports. Conserv. Biol. 33, 747–750 (2019).

    Article  PubMed  Google Scholar 

  53. Buxton, R. T. et al. Avoiding wasted research resources in conservation science. Conserv. Sci. Pract. 3, 1–11 (2021).

    Google Scholar 

  54. Powers, S. M. & Hampton, S. E. Open science, reproducibility, and transparency in ecology. Ecol. Appl. 29, 1–8 (2019).

    Article  Google Scholar 

  55. Archmiller, A. A. et al. Computational reproducibility in the Wildlife Society’s flagship journals. J. Wildl. Manag. 84, 1012–1017 (2020).

    Article  Google Scholar 

  56. Whitlock, M. C., McPeek, M. A., Rausher, M. D., Rieseberg, L. & Moore, A. J. Data archiving. Am. Nat. 175, 145–146 (2010).

    Article  PubMed  Google Scholar 

  57. Whitlock, M. C. Data archiving in ecology and evolution: best practices. Trends Ecol. Evol. 26, 61–65 (2011).

    Article  PubMed  Google Scholar 

  58. Mislan, K. A. S., Heer, J. M. & White, E. P. Elevating the status of code in ecology. Trends Ecol. Evol. 31, 4–7 (2016).

    Article  CAS  PubMed  Google Scholar 

  59. Culina, A., van den Berg, I., Evans, S. & Sánchez-Tójar, A. Low availability of code in ecology: a call for urgent action. PLoS Biol. 18, 1–9 (2020).

    Article  Google Scholar 

  60. Wilkinson, M. D. et al. Comment: the FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 1–9 (2016).

    Article  Google Scholar 

  61. Gopalakrishna, G. et al. Prevalence of responsible research practices among academics in the Netherlands. F1000Research 11, 1–34 (2022).

    Article  Google Scholar 

  62. Hardwicke, T. E. et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. R. Soc. Open Sci. 5, 180448 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  63. Stodden, V., Seiler, J. & Ma, Z. An empirical analysis of journal policy effectiveness for computational reproducibility. Proc. Natl Acad. Sci. USA 115, 2584–2589 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Roche, D. G., Kruuk, L. E. B., Lanfear, R. & Binning, S. A. Public data archiving in ecology and evolution: how well are we doing? PLoS Biol. 13, 1–12 (2015).

    Article  Google Scholar 

  65. Roche, D. G. et al. Slow improvement to the archiving quality of open datasets shared by researchers in ecology and evolution. Proc. R. Soc. B 289, 20212780 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Lindsey, P. A. et al. The bushmeat trade in African savannas: impacts, drivers, and possible solutions. Biol. Conserv. 160, 80–96 (2013).

    Article  Google Scholar 

  67. Roche, D. G. et al. Paths towards greater consensus building in experimental biology. J. Exp. Biol. 225, jeb243559 (2022).

    Article  PubMed  Google Scholar 

  68. Smaldino, P. E. & McElreath, R. The natural selection of bad science. R. Soc. Open Sci. 3, 160384 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  69. R Core Team. R: a language and environment for statistical computing (R Foundation for Statistical Computing, 2019); https://www.R-project.org/

  70. Müller, K. here: a simpler way to find your files. R package version 1.0.1 (2017). https://CRAN.R-project.org/package=here

  71. Wickham, H., Francois, R., Henry, L. & Muller, K. dplyr: a grammar of data manipulation R package version 1.0.7 (2020). https://CRAN.R-project.org/package=dplyr

  72. Wickham, H. & Henry, L. tidyr: tidy messy data R package version 1.1.4 (2020). https://CRAN.R-project.org/package=tidyr

  73. Wickham, H. ggplot2: elegant graphics for data analysis (Springer-Verlag, 2016).

  74. Kassambara, A. ggpubr: ‘ggplot2’ based publication ready plots. R package version 0.4.0 (2020). https://CRAN.R-project.org/package=ggpubr

  75. Pedersen, T. L. patchwork: the composer of plots. R package version 1.1.1 (2021). https://CRAN.R-project.org/package=patchwork

  76. Wickham, H. & Seidel, D. scales: scale functions for visualization. R package version 1.1.1 (2020). https://CRAN.R-project.org/package=scales

  77. Bloom, H. S. Minimum detectable effects: a simple way to report the statistical power of experimental designs. Eval. Rev. 19, 547–556 (1995).

    Article  Google Scholar 

  78. Djimeu, E. W. & Houndolo, D. G. Power calculation for causal inference in social science: sample size and minimum detectable effect determination. J. Dev. Eff. 8, 508–527 (2016).

    Article  Google Scholar 

  79. Havranek, T., Horvath, R. & Zeynalov, A. Natural resources and economic growth: a meta-analysis. World Dev. 88, 134–151 (2016).

    Article  Google Scholar 

  80. Stanley, T. D., Carter, E. C. & Doucouliagos, H. What meta-analyses reveal about the replicability of psychological research. Psychol. Bull. 144, 1325–1346 (2018).

    Article  CAS  PubMed  Google Scholar 

  81. Parker, T. H. et al. Empowering peer reviewers with a checklist to improve transparency. Nat. Ecol. Evol. 2, 929–935 (2018).

    Article  PubMed  Google Scholar 

  82. Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 1–9 (2017).

    Article  Google Scholar 

  83. Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422–1425 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  84. Nakagawa, S. & Parker, T. H. Replicating research in ecology and evolution: feasibility, incentives, and the cost–benefit conundrum. BMC Biol. 13, 1–6 (2015).

    Article  CAS  Google Scholar 

  85. Kaplan, R. M. & Irvin, V. L. Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS ONE 10, 1–12 (2015).

    Article  Google Scholar 

  86. Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. The preregistration revolution. Proc. Natl Acad. Sci. USA 115, 2600–2606 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  87. Allen, C. & Mehler, D. M. A. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 17, 1–14 (2019).

    Google Scholar 

  88. Scheel, A. M., Schijen, M. R. M. J. & Lakens, D. An excess of positive results: comparing the standard psychology literature with registered reports. Adv. Methods Pract. Psychol. Sci. 4, 1–12 (2021).

    Google Scholar 

  89. Nosek, B. A. et al. Preregistration is hard, and worthwhile. Trends Cogn. Sci. 23, 815–818 (2019).

    Article  PubMed  Google Scholar 

  90. Button, K. S., Bal, L., Clark, A. & Shipley, T. Preventing the ends from justifying the means: withholding results to address publication bias in peer-review. BMC Psychol. 4, 1–7 (2016).

    Article  Google Scholar 

  91. Soderberg, C. K. et al. Initial evidence of research quality of registered reports compared with the standard publishing model. Nat. Hum. Behav. 5, 990–997 (2021).

    Article  PubMed  Google Scholar 

  92. Smulders, Y. M. A two-step manuscript submission process can reduce publication bias. J. Clin. Epidemiol. 66, 946–947 (2013).

    Article  PubMed  Google Scholar 

  93. Anderson, M. S., Martinson, B. C. & De Vries, R. Normative dissonance in science: results from a national survey of U.S. scientists. J. Empir. Res. Hum. Res. Ethics 3, 3–14 (2007).

    Article  Google Scholar 

Download references

Acknowledgements

We thank the Glenadore and Howard L. Pim Postdoctoral Fellowship in Global Change for funding K.K. We thank T. Parker for his helpful comments on revising the manuscript. We thank M. Buchanan, P. Dye, Z. Ellis, Y. Li, L. Wang and L. Williams for helping in the data collection for this paper. We thank P. Shukla for providing sample code for the analyses.

Author information

Authors and Affiliations

  1. Mad Agriculture, Boulder, CO, USA

    Kaitlin Kimmel

  2. Department of Earth and Planetary Sciences, Johns Hopkins University, Baltimore, MD, USA

    Kaitlin Kimmel & Meghan L. Avolio

  3. Carey Business School, Johns Hopkins University, Baltimore, MD, USA

    Paul J. Ferraro

  4. Department of Environmental Health and Engineering, a joint department of the Bloomberg School of Public Health and the Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA

    Paul J. Ferraro

Authors
  1. Kaitlin Kimmel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meghan L. Avolio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul J. Ferraro
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.J.F. and K.K. designed the study. K.K. analysed the data. M.L.A., P.J.F. and K.K. wrote the paper.

Corresponding author

Correspondence to Paul J. Ferraro.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Ecology & Evolution thanks Timothy Parker, Antica Culina, Dominique Roche and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary text, Fig. 1 and Table 1.

Reporting Summary

Peer Review File

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kimmel, K., Avolio, M.L. & Ferraro, P.J. Empirical evidence of widespread exaggeration bias and selective reporting in ecology. Nat Ecol Evol 7, 1525–1536 (2023). https://doi.org/10.1038/s41559-023-02144-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41559-023-02144-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing Anthropocene

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: Anthropocene