To the Editor:
Reproducibility of results is a fundamental tenet of science. In this journal, Richter et al.1 tested whether systematic variation in experimental conditions (heterogenization) affects the reproducibility of results. Comparing this approach with the current standard of ensuring reproducibility through minimizing variation in experimental conditions (standardization), they concluded that heterogenization improved reproducibility1. However, in our view, they did not account for significant sources of dependency in their data, which resulted in an inflated type I error rate through pseudoreplication (defined as “the use of inferential statistics to test for treatment effects with data from experiments where either treatments are not replicated (though samples may be) or replicates are not statistically independent”2). We show that this leads to strong overconfidence in their analyses and that their hypothesis is unsupported.
Richter et al.1 compared F ratios of strain-by-experiment interactions of 36 behavioral measures to test for differences in reproducibility between series of standardized and heterogenized experiments. Although these measures were treated as independent, most are strongly intercorrelated, thus causing interdependency of the F ratios. Two sources contribute to interdependence of behavioral measures and, hence, pseudoreplication. First, the measures within each of their three behavioral tests are strongly intercorrelated. For example, an animal exploring the edge of an arena cannot explore the center simultaneously. Second, the measures are correlated across tests because many animals show temporal and cross-contextual consistency in behavioral traits (known as 'animal personalities'3,4). However, to allow meaningful comparison of reproducibility for standardized versus heterogenized experiments, F ratios must be obtained independently5.
We reanalyzed their data (Supplementary Note and Supplementary Figs. 1–6) by identifying supposedly independent variables using hierarchical clustering. This analysis showed that there was no detectable difference in reproducibility between standardization and heterogenization. Hence, in our view, their data do not support their hypothesis.
We caution that overconfidence resulting from pseudoreplication may lead to premature conclusions in studies designed to prove this principle1,6,7. Unjustifiably assuming that heterogenization yields better reproducibility may prompt a reduction in the number of replicate experiments, possibly decreasing the chance of detecting desired and/or unwanted effects8. Hence, further studies validating the benefits of heterogenization for reproducibility are required before it can be adopted as the new standard.
References
Richter, S.H., Garner, J.P., Auer, C., Kunert, J. & Würbel, H. Nat. Methods 7, 167–168 (2010).
Hurlbert, S.H. Ecol. Monogr. 54, 187–211 (1984).
Wolf, M. & Weissing, F.J. Trends Ecol. Evol. 27, 452–461 (2012).
Lewejohann, L., Zipser, B. & Sachser, N. Dev. Psychobiol. 53, 624–630 (2011).
Schumann, D.E.W. & Bradley, R.A. Ann. Math. Stat. 28, 902–920 (1957).
Richter, S.H. et al. PLoS ONE 6, e16461 (2011).
Richter, S.H., Garner, J.P. & Würbel, H. Nat. Methods 6, 257–261 (2009).
van der Staay, F.J., Arndt, S.S. & Nordquist, R.E. Genes Brain Behav. 9, 849–855 (2010).
Acknowledgements
We thank H. Würbel and H. Richter for making the data used in Richter et al.1 available to us. We thank the members of the Stats Club of Bielefeld University's Evolutionary Biology, Animal Behaviour and Behavioural Biology departments; H. Würbel for discussion; and F. Schurr for helpful comments on an earlier version of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–6 and Supplementary Note 1 (PDF 700 kb)
Rights and permissions
About this article
Cite this article
Jonker, R., Guenther, A., Engqvist, L. et al. Does systematic variation improve the reproducibility of animal experiments?. Nat Methods 10, 373 (2013). https://doi.org/10.1038/nmeth.2439
Published:
Issue Date:
DOI: https://doi.org/10.1038/nmeth.2439
This article is cited by
-
Reproducibility of animal research in light of biological variation
Nature Reviews Neuroscience (2020)
-
Systematic heterogenization for better reproducibility in animal experimentation
Lab Animal (2017)
-
Reanalysis of Richter et al. (2010) on reproducibility
Nature Methods (2013)
-
Reply to: "Reanalysis of Richter et al. (2010) on reproducibility"
Nature Methods (2013)
-
The way you say it
Nature Methods (2013)