Abstract

Many scientific disciplines are currently experiencing a 'reproducibility crisis' because numerous scientific findings cannot be repeated consistently. A novel but controversial hypothesis postulates that stringent levels of environmental and biotic standardization in experimental studies reduce reproducibility by amplifying the impacts of laboratory-specific environmental factors not accounted for in study designs. A corollary to this hypothesis is that a deliberate introduction of controlled systematic variability (CSV) in experimental designs may lead to increased reproducibility. To test this hypothesis, we had 14 European laboratories run a simple microcosm experiment using grass (Brachypodium distachyon L.) monocultures and grass and legume (Medicago truncatula Gaertn.) mixtures. Each laboratory introduced environmental and genotypic CSV within and among replicated microcosms established in either growth chambers (with stringent control of environmental conditions) or glasshouses (with more variable environmental conditions). The introduction of genotypic CSV led to 18% lower among-laboratory variability in growth chambers, indicating increased reproducibility, but had no significant effect in glasshouses where reproducibility was generally lower. Environmental CSV had little effect on reproducibility. Although there are multiple causes for the 'reproducibility crisis', deliberately including genetic variability may be a simple solution for increasing the reproducibility of ecological studies performed under stringently controlled environmental conditions.

  • Subscribe to Nature Ecology & Evolution for full access:

    $99

    Subscribe

Additional access options:

Already a subscriber?  Log in  now or  Register  for online access.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Cassey, P. & Blackburn, T. Reproducibility and repeatability in ecology. Bioscience 56, 958–959 (2006).

  2. 2.

    Ellison, A. M. Repeatability and transparency in ecological research. Ecology 91, 2536–2539 (2010).

  3. 3.

    Lawton, J. H. The Ecotron facility at Silwood Park: the value of ‘big bottle’ experiments. Ecology 77, 665–669 (1996).

  4. 4.

    Benton, T. G., Solan, M., Travis, J. M. & Sait, S. M. Microcosm experiments can inform global ecological problems. Trends Ecol. Evol. 22, 516–521 (2007).

  5. 5.

    Drake, J. M. & Kramer, A. M. Mechanistic analogy: how microcosms explain nature. Theor. Ecol. 5, 433–444 (2012).

  6. 6.

    Fraser, L. H. & Keddy, P. The role of experimental microcosms in ecological research. Trends Ecol. Evol. 12, 478–481 (1997).

  7. 7.

    Srivastava, D. S. et al. Are natural microcosms useful model systems for ecology? Trends Ecol. Evol. 19, 379–384 (2004).

  8. 8.

    De Boeck, H. J. et al. Global change experiments: challenges and opportunities. BioScience 65, 922–931 (2015).

  9. 9.

    Richter, S. H. et al. Effect of population heterogenization on the reproducibility of mouse behavior: a multi-laboratory study. PLoS ONE 6, e16461 (2011).

  10. 10.

    Richter, S. H., Garner, J. P. & Würbel, H. Environmental standardization: cure or cause of poor reproducibility in animal experiments? Nat. Methods 6, 257–261 (2009).

  11. 11.

    Richter, S. H., Garner, J. P., Auer, C., Kunert, J. & Würbel, H. Systematic variation improves reproducibility of animal experiments. Nat. Methods 7, 167–168 (2010).

  12. 12.

    Massonnet, C. et al Probing the reproducibility of leaf growth and molecular phenotypes: a comparison of three Arabidopsis accessions cultivated in ten laboratories. Plant Physiol. 152, 2142–2157 (2010).

  13. 13.

    Begley, C. G. & Ellis, M. L. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012).

  14. 14.

    Open Science Collaboration Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

  15. 15.

    Parker, T. H. et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31, 711–719 (2016).

  16. 16.

    Moore, R. P. & Robinson, W. D. Artificial bird nests, external validity, and bias in ecological field studies. Ecology 85, 1562–1567 (2004).

  17. 17.

    Temperton, V. M., Mwangi, P. N., Scherer-Lorenzen, M., Schmid, B. & Buchmann, N. Positive interactions between nitrogen-fixing legumes and four different neighbouring species in a biodiversity experiment. Oecologia 151, 190–205 (2007).

  18. 18.

    Meng, L. et al. Arbuscular mycorrhizal fungi and rhizobium facilitate nitrogen uptake and transfer in soybean/maize intercropping system. Front. Plant Sci. 6, 339 (2015).

  19. 19.

    Sleugh, B., Moore, K. J., George, J. R. & Brummer, E. C. Binary legume–grass mixtures improve forage yield, quality, and seasonal distribution. Agron. J. 92, 24–29 (2000).

  20. 20.

    Keuskamp, J. A., Dingemans, B. J. J., Lehtinen, T., Sarneel, J. M. & Hefting, M. M. Tea bag index: a novel approach to collect uniform decomposition data across ecosystems. Methods Ecol. Evol. 4, 1070–1075 (2013).

  21. 21.

    Nyfeler, D., Huguenin-Elie, O., Suter, M., Frossard, E. & Lüscher, A. Grass–legume mixtures can yield more nitrogen than legume pure stands due to mutual stimulation of nitrogen uptake from symbiotic and non-symbiotic sources. Agric. Ecosyst. Environ. 140, 155–163 (2011).

  22. 22.

    Suter, M. et al Nitrogen yield advantage from grass–legume mixtures is robust over a wide range of legume proportions and environmental conditions. Glob. Change Biol. 21, 2424–2438 (2015).

  23. 23.

    Loreau, M. & de Mazancourt, C. Biodiversity and ecosystem stability: a synthesis of underlying mechanisms. Ecol. Lett. 16, 106–115 (2013).

  24. 24.

    Reusch, T. B., Ehlers, A., Hämmerli, A. & Worm, B. Ecosystem recovery after climatic extremes enhanced by genotypic diversity. Proc. Natl Acad. Sci. USA 102, 2826–2831 (2005).

  25. 25.

    Hughes, A. R., Inouye, B. D., Johnson, M. T. J., Underwood, N. & Vellend, M. Ecological consequences of genetic diversity. Ecol. Lett. 11, 609–623 (2008).

  26. 26.

    Prieto, I. et al. Complementary effects of species and genetic diversity on productivity and stability of sown grasslands. Nat. Plants 1, 1–5 (2015).

  27. 27.

    Wasserstein, R. L. & Lazar, N. A. The ASA’s statement on P-values: context, process, and purpose. Am. Stat. 70, 129–133 (2016).

  28. 28.

    Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).

  29. 29.

    Nuzzo, R. How scientists fool themselves—and how they can stop. Nature 526, 182–185 (2015).

  30. 30.

    R Development Core Team R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, 2017).

  31. 31.

    Tukey, J. W. Exploratory Data Analysis (Addison–Wesley, Reading, USA, 1977).

  32. 32.

    Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D. & R Core Team. nlme: Linear and Nonlinear Mixed-Effects Models R Package Version 3.1-122 (The R Foundation, 2016); http://CRAN.R-project.org/package=nlme

  33. 33.

    Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A. & Smith, G. M. Mixed-Effects Models and Extensions in Ecology with R (Springer, New York, 2009).

  34. 34.

    Lê, S., Josse, J. & Husson, F. FactoMineR: an R package for multivariate analysis. J. Stat. Softw. 25, 1–18 (2008).

  35. 35.

    Josse, J., Chavent, M., Liquet, B. & Husson, F. Handling missing values with regularized iterative multiple correspondence analysis. J. Classif. 29, 91–116 (2010).

  36. 36.

    Josse, J. & Husson, F. missMDA: a package for handling missing values in multivariate data analysis. J. Stat. Softw. 70, 1–31 (2016).

Download references

Acknowledgements

This study benefited from the Centre Nationnal de la Recherche Scientifique human and technical resources allocated to the Ecotrons research infrastructures, the state allocation ‘Investissement d’Avenir’ ANR-11-INBS-0001 and financial support from the ExpeER (grant 262060) consortium funded under the EU-FP7 research programme (FP2007–2013). Brachypodium seeds were provided by R. Sibout (Observatoire du Végétal, Institut Jean-Pierre Bourgin) and Medicago seeds were supplied by J.-M. Prosperi (Institut National de la Recherche Agronomique Biological Resource Centre). We further thank J. Varale, G. Hoffmann, P. Werthenbach, O. Ravel, C. Piel, D. Landais, D. Degueldre, T. Mathieu, P. Aury, N. Barthès, B. Buatois and R. Leclerc for assistance during the study. For additional acknowledgements, see the Supplementary Information.

Author information

Affiliations

  1. Ecotron (Unité Propre de Service 3248), Centre National de la Recherche Scientifique, Campus Baillarguet, Montferrier-sur-Lez, France

    • Alexandru Milcu
    • , Sebastien Devidal
    •  & Jacques Roy
  2. Centre d’Ecologie Fonctionnelle et Evolutive, Centre National de la Recherche Scientifique, Unité Mixte de Recherche 5175, Université de Montpellier/Université Paul Valéry – École Pratique des Hautes Études, Montpellier, France

    • Alexandru Milcu
    • , Grégoire T. Freschet
    •  & Katherine Urban-Mead
  3. Institut de l’Ecologie et des Sciences de l’Environnement de Paris, Université Paris-Est Créteil, Créteil, France

    • Ruben Puga-Freitas
    • , Manuel Blouin
    •  & Agnès Gigon
  4. Harvard Forest, Harvard University, Petersham, MA, USA

    • Aaron M. Ellison
  5. Tropical Forests and People Research Centre, University of the Sunshine Coast, Maroochydore DC, Queensland, Australia

    • Aaron M. Ellison
  6. Agroécologie, AgroSup Dijon, Institut National de la Recherche Agronomique, Université Bourgogne Franche-Comté, Dijon, France

    • Manuel Blouin
    •  & Anne Pando
  7. Johann-Friedrich-Blumenbach Institute for Zoology and Anthropology, Georg August University Göttingen, Göttingen, Germany

    • Stefan Scheu
    •  & Olaf Butenschoen
  8. Department of Geobotany, Faculty of Biology, University of Freiburg, Freiburg, Germany

    • Laura Rose
    • , Anna Greiner
    • , Paula Rotter
    •  & Michael Scherer-Lorenzen
  9. Institut de Recherche pour le Développement, Institut de l’Ecologie et des Sciences de l’Environnement de Paris, Université Pierre et Marie Curie, Paris, France

    • Sebastien Barot
  10. German Centre for Integrative Biodiversity Research, Halle-Jena-Leipzig, Leipzig, Germany

    • Simone Cesarz
    • , Nico Eisenhauer
    •  & Alexandra Weigelt
  11. Institute of Biology, Leipzig University, Leipzig, Germany

    • Simone Cesarz
    • , Nico Eisenhauer
    •  & Alexandra Weigelt
  12. Institut Jean-Pierre Bourgin, INRA, AgroParisTech, Centre National de la Recherche Scientifique, Université Paris-Saclay, Versailles, France

    • Thomas Girin
  13. Department of Agricultural, Forest and Food Sciences, University of Turin, Grugliasco, Italy

    • Davide Assandri
    • , Carlo Grignani
    •  & Laura Zavattaro
  14. Cluster of Excellence on Plant Sciences, Terrestrial Ecology Group, Institute for Zoology, University of Cologne, Cologne, Germany

    • Michael Bonkowski
  15. Institute of Agricultural Sciences, ETH Zurich, Zurich, Switzerland

    • Nina Buchmann
    •  & Neringa Mannerheim
  16. Senckenberg Biodiversität und Klima Forschungszentrum, Frankfurt, Germany

    • Olaf Butenschoen
  17. Max Planck Institute for Biogeochemistry, Postfach 100164, Jena, Germany

    • Gerd Gleixner
    •  & Markus Lange
  18. Leibniz Centre for Agricultural Landscape Research, Institute of Landscape Biogeochemistry, Müncheberg, Germany

    • Arthur Gessler
    •  & Marina E. H. Müller
  19. Swiss Federal Research Institute, Zürcherstrasse 111, Birmensdorf, Switzerland

    • Arthur Gessler
    •  & Zachary Kayler
  20. Département de Biologie, Ecole Normale Supérieure, Université de recherche Paris Sciences & Lettres Research University, Centre National de la Recherche Scientifique, Unité Mixte de Service 3194 (Centre de Recherche en Écologie Expérimentale et Prédictive-Ecotron IleDeFrance), Saint-Pierre-lès-Nemours, France

    • Amandine Hansart
    •  & Jean-François Le Galliard
  21. Department of Soil and Water Systems, University of Idaho, Moscow, ID, USA

    • Zachary Kayler
  22. Institut de l’Ecologie et des Sciences de l’Environnement de Paris, Sorbonne Universités, Paris, France

    • Jean-Christophe Lata
    • , Jean-François Le Galliard
    •  & Rahme Seyhun
  23. School of Agriculture, Policy and Development, University of Reading, Reading, UK

    • Martin Lukac
  24. Faculty of Forestry and Wood Sciences, Czech University of Life Sciences, Prague, Czech Republic

    • Martin Lukac

Authors

  1. Search for Alexandru Milcu in:

  2. Search for Ruben Puga-Freitas in:

  3. Search for Aaron M. Ellison in:

  4. Search for Manuel Blouin in:

  5. Search for Stefan Scheu in:

  6. Search for Grégoire T. Freschet in:

  7. Search for Laura Rose in:

  8. Search for Sebastien Barot in:

  9. Search for Simone Cesarz in:

  10. Search for Nico Eisenhauer in:

  11. Search for Thomas Girin in:

  12. Search for Davide Assandri in:

  13. Search for Michael Bonkowski in:

  14. Search for Nina Buchmann in:

  15. Search for Olaf Butenschoen in:

  16. Search for Sebastien Devidal in:

  17. Search for Gerd Gleixner in:

  18. Search for Arthur Gessler in:

  19. Search for Agnès Gigon in:

  20. Search for Anna Greiner in:

  21. Search for Carlo Grignani in:

  22. Search for Amandine Hansart in:

  23. Search for Zachary Kayler in:

  24. Search for Markus Lange in:

  25. Search for Jean-Christophe Lata in:

  26. Search for Jean-François Le Galliard in:

  27. Search for Martin Lukac in:

  28. Search for Neringa Mannerheim in:

  29. Search for Marina E. H. Müller in:

  30. Search for Anne Pando in:

  31. Search for Paula Rotter in:

  32. Search for Michael Scherer-Lorenzen in:

  33. Search for Rahme Seyhun in:

  34. Search for Katherine Urban-Mead in:

  35. Search for Alexandra Weigelt in:

  36. Search for Laura Zavattaro in:

  37. Search for Jacques Roy in:

Contributions

A.M. and J.R. designed the study with input from M. Blouin, S.B., M. Bonkowski and J.-C.L. Substantial methodological contributions were provided by S.S., T.G., L.R. and M.S.-L. Conceptual feedback on an early version was provided by G.T.F., N.E., J.R. and A.M.E. Data were analysed by A.M. with input from A.M.E. A.M. wrote the manuscript with input from all authors. All authors were involved in carrying out the experiments and/or analyses.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Alexandru Milcu.

Supplementary information

  1. Supplementary information

    Supplementary Tables 1–3; Supplementary Figures 1–5; Model outputs; Supplementary Acknowledgements.

  2. Life Sciences Reporting Summary