Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reproducibility crisis and gravitation towards a consensus in ocean acidification research

Nature Climate Change volume 13, pages 1266–1271 (2023)Cite this article

Subjects

Abstract

Reproducibility is a persistent concern in science and recently attracts considerable attention in assessing biological responses to ocean acidification. Here we track the reproducibility of the harmful effects of ocean acidification on calcification of shell-building organisms by conducting a meta-analysis of 373 studies across 24 years. The pioneering studies tended to report large negative effects, but as other researchers assimilated this research into understanding their biological systems, the size of negative effects declined. Such declines represent a scientific process by which discoveries are initially assimilated and their limitations are subsequently explored. We suggest that scientific novelties can polarize a discipline where researchers fail to distinguish between different motivations for testing a phenomenon, that is, its existence (theory proposal) versus its influence within ever-widening contexts (theory development). Where context dependency is high, the lack of reproducibility may not represent a crisis but a part of theory development and eventual gravitation towards a consensus position.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Responses of marine calcifiers to ocean acidification in terms of calcification across the publication years.
Fig. 2: Potential factors that drive the effect sizes of calcification.
Fig. 3: Different methods of manipulation of seawater carbonate chemistry used for ocean acidification research across the publication years.
Fig. 4: Various taxa of marine calcifiers selected for ocean acidification research across the publication years.

Data availability

The datasets analysed in the current study are available in the Supplementary Information.

Code availability

The figures were created and analysed using a commercial software GraphPad Prism v.9.0.0, and the meta-analysis was conducted using an open-source software R version 4.3.0 with the metafor package version 4.2-0. The codes used to reproduce the meta-analysis are presented in the Supplementary Information.

References

  1. Schooler, J. Unpublished results hide the decline effect. Nature 470, 437 (2011).

    Article  CAS  Google Scholar 

  2. Fang, F. C. & Casadevall, A. Competitive science: is competition ruining science? Infect. Immun. 83, 1229–1233 (2015).

    Article  CAS  Google Scholar 

  3. Clements, J. C., Sundin, J., Clark, T. D. & Jutfelt, F. Meta-analysis reveals an extreme ‘decline effect’ in the impacts of ocean acidification on fish behavior. PLoS Biol. 20, e3001511 (2022).

    Article  CAS  Google Scholar 

  4. Gattuso, J. P., Frankignoulle, M., Bourge, I., Romaine, S. & Buddemeier, R. W. Effect of calcium carbonate saturation of seawater on coral calcification. Glob. Planet. Change 18, 37–46 (1998).

    Article  Google Scholar 

  5. Riebesell, U. et al. Reduced calcification of marine plankton in response to increased atmospheric CO2. Nature 407, 364–367 (2000).

    Article  CAS  Google Scholar 

  6. Zondervan, I., Zeebe, R. E., Rost, B. & Riebesell, U. Decreasing marine biogenic calcification: a negative feedback on rising atmospheric \(p_{{\mathrm{co}}_2}\). Glob. Biogeochem. Cycles 15, 507–516 (2001).

    Article  CAS  Google Scholar 

  7. Feely, R. A. et al. Impact of anthropogenic CO2 on the CaCO3 system in the oceans. Science 305, 362–366 (2004).

    Article  CAS  Google Scholar 

  8. Orr, J. C. et al. Anthropogenic ocean acidification over the twenty-first century and its impact on calcifying organisms. Nature 437, 681–686 (2005).

    Article  CAS  Google Scholar 

  9. Leung, J. Y. S., Zhang, S. & Connell, S. D. Is ocean acidification really a threat to marine calcifiers? A systematic review and meta-analysis of 980+ studies spanning two decades. Small 18, 2107407 (2022).

    Article  CAS  Google Scholar 

  10. Leung, J. Y. S. et al. Calcifiers can adjust shell building at the nanoscale to resist ocean acidification. Small 16, 2003186 (2020).

    Article  CAS  Google Scholar 

  11. Cornwall, C. E. et al. Resistance of corals and coralline algae to ocean acidification: physiological control of calcification under natural pH variability. Proc. R. Soc. B 285, 20181168 (2018).

    Article  Google Scholar 

  12. Rajan, K. C., Meng, Y., Yu, Z., Roberts, S. B. & Vengatesen, T. Oyster biomineralization under ocean acidification: from genes to shell. Glob. Change Biol. 27, 3779–3797 (2021).

    Article  Google Scholar 

  13. Liu, Y., Eagle, R. A., Aciego, S. M., Gilmore, R. E. & Ries, J. B. A coastal coccolithophore maintains pH homeostasis and switches carbon sources in response to ocean acidification. Nat. Commun. 9, 2857 (2018).

    Article  Google Scholar 

  14. Wahl, M. et al. Macroalgae may mitigate ocean acidification effects on mussel calcification by increasing pH and its fluctuations. Limnol. Oceanogr. 63, 3–21 (2018).

    Article  CAS  Google Scholar 

  15. Lau, J., Schmid, C. H. & Chalmers, T. C. Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J. Clin. Epidemiol. 48, 45–57 (1995).

    Article  CAS  Google Scholar 

  16. Pietschnig, J., Voracek, M. & Formann, A. K. Mozart effect—Shmozart effect: a meta-analysis. Intelligence 38, 314–323 (2010).

    Article  Google Scholar 

  17. Pietschnig, J., Penke, L., Wicherts, J. M., Zeiler, M. & Voracek, M. Meta-analysis of associations between human brain volume and intelligence differences: how strong are they and what do they mean? Neurosci. Biobehav. Rev. 57, 411–432 (2015).

    Article  Google Scholar 

  18. Connell, S. D. et al. How ocean acidification can benefit calcifiers. Curr. Biol. 27, R95–R96 (2017).

    Article  CAS  Google Scholar 

  19. Goldenberg, S. U. et al. Ecological complexity buffers the impacts of future climate on marine consumers. Nat. Clim. Change 8, 229–233 (2018).

    Article  Google Scholar 

  20. Leung, J. Y. S. et al. How calorie-rich food could help marine calcifiers in a CO2-rich future. Proc. R. Soc. B 286, 20190757 (2019).

    Article  CAS  Google Scholar 

  21. Cornwall, C. E. et al. Understanding coralline algal responses to ocean acidification: meta-analysis and synthesis. Glob. Change Biol. 28, 362–374 (2022).

    Article  CAS  Google Scholar 

  22. Cornwall, C. E. & Hurd, C. L. Experimental design in ocean acidification research: problems and solutions. ICES J. Mar. Sci. 73, 572–581 (2015).

    Article  Google Scholar 

  23. Cornwall, C. E. et al. Global declines in coral reef calcium carbonate production under ocean acidification and warming. Proc. Natl Acad. Sci. USA 118, e2015265118 (2021).

    Article  CAS  Google Scholar 

  24. Mlinarić, A., Horvat, M. & Smolčić, V. S. Dealing with the positive publication bias: why you should really publish your negative results. Biochem. Med. 27, 030201 (2017).

    Article  Google Scholar 

  25. Browman, H. I. Applying organized scepticism to ocean acidification research. ICES J. Mar. Sci. 73, 529–536 (2016).

    Article  Google Scholar 

  26. Leung, J. Y. S., Harvey, B. P. & Russell, B. D. Editorial: fitness of marine calcifiers in the future acidifying ocean. Front. Mar. Sci. 8, 752635 (2021).

    Article  Google Scholar 

  27. McPherson, M., Smith-Lovin, L. & Cook, J. M. Birds of a feather: homophily in social networks. Annu. Rev. Socio 27, 415–444 (2001).

    Article  Google Scholar 

  28. Norberg, J., Blenckner, T., Cornell, S. E., Petchey, O. L. & Hillebrand, H. Failures to disagree are essential for environmental science to effectively influence policy development. Ecol. Lett. 25, 1075–1093 (2022).

    Article  Google Scholar 

  29. Kahneman, D. & Klein, G. Conditions for intuitive expertise: a failure to disagree. Am. Psychol. 64, 515–526 (2009).

    Article  Google Scholar 

  30. Chamberlain, S. A., Bronstein, J. L. & Rudgers, J. A. How context dependent are species interactions? Ecol. Lett. 17, 881–890 (2014).

    Article  Google Scholar 

  31. Shanteau, J. How much information does an expert use? Is it relevant? Acta Psychol. 81, 75–86 (1992).

    Article  Google Scholar 

  32. Protzko, J. et al. 2020. High replicability of newly-discovered social–behavioral findings is achievable. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/n2a9x (2020).

  33. Amaral, O. B. & Neves, K. Reproducibility: expect less of the scientific paper. Nature 597, 329–331 (2021).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We acknowledge the Australian Research Council grant to S.D.C. (ARC DP230101932) and the funding from the National Natural Science Foundation of China to J.Y.S.L. (42176199, 42076121, M-0163, 42211530423). We thank N. K. M. Cheung for his advice on the statistical analyses used in this study.

Author information

Author notes

  1. These authors contributed equally: Sean D. Connell, Jonathan Y. S. Leung.

Authors and Affiliations

  1. Southern Seas Ecology Laboratories, School of Biological Sciences, The University of Adelaide, Adelaide, South Australia, Australia

    Sean D. Connell & Jonathan Y. S. Leung

  2. Guangdong Provincial Key Laboratory of Marine Disaster Prediction and Prevention, Shantou University, Shantou, China

    Jonathan Y. S. Leung

  3. Fisheries College, Guangdong Ocean University, Zhanjiang, China

    Jonathan Y. S. Leung

Authors
  1. Sean D. Connell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan Y. S. Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.D.C. and J.Y.S.L. conceived the study, interpreted the results and wrote the manuscript. J.Y.S.L. collected and analysed the data. S.D.C. provided in-depth interpretation of the data.

Corresponding authors

Correspondence to Sean D. Connell or Jonathan Y. S. Leung.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Climate Change thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Trends in the experimental protocol used for ocean acidification research across the publication years.

The scatter plots showing the gradual increases in (a) experimental duration and (b) average sample size used for experimentation across the years. Each dot represents an observation, whereas the solid grey line represents the best fit line of linear regression.

Extended Data Fig. 2 Assessment of potential publication bias based on funnel plot asymmetry.

The funnel plot showing the relationship between effect size and standard error using Egger’s regression test (two-sided, z = –10.8, p = 2.33 × 10–27). Each dot represents an observation.

Supplementary information

Supplementary Information

Supplementary Fig. 1, Tables 1 and 2 and Methods.

Reporting Summary

Supplementary Data 1

A comprehensive dataset of the studies included in the meta-analysis.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Connell, S.D., Leung, J.Y.S. Reproducibility crisis and gravitation towards a consensus in ocean acidification research. Nat. Clim. Chang. 13, 1266–1271 (2023). https://doi.org/10.1038/s41558-023-01828-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41558-023-01828-9

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing