Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reproducibility of animal research in light of biological variation

Nature Reviews Neuroscience volume 21, pages 384–393 (2020)Cite this article

Subjects

An Author Correction to this article was published on 08 June 2020

This article has been updated

Abstract

Context-dependent biological variation presents a unique challenge to the reproducibility of results in experimental animal research, because organisms’ responses to experimental treatments can vary with both genotype and environmental conditions. In March 2019, experts in animal biology, experimental design and statistics convened in Blonay, Switzerland, to discuss strategies addressing this challenge. In contrast to the current gold standard of rigorous standardization in experimental animal research, we recommend the use of systematic heterogenization of study samples and conditions by actively incorporating biological variation into study design through diversifying study samples and conditions. Here we provide the scientific rationale for this approach in the hope that researchers, regulators, funders and editors can embrace this paradigm shift. We also present a road map towards better practices in view of improving the reproducibility of animal research.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Context-dependent treatment effect.

Change history

References

  1. Agassi, J. The very idea of modern science: Francis Bacon and Robert Boyle Vol. 298 (Springer Science+Business Media, 2013).

  2. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    PubMed  PubMed Central  Google Scholar 

  3. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

    Google Scholar 

  4. Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–453 (2016).

    CAS  PubMed  Google Scholar 

  5. Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Loken, E. & Gelman, A. Measurement error and the replication crisis. Science 355, 584–585 (2017).

    CAS  PubMed  Google Scholar 

  7. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: How much can we rely on published data on potential drug targets? Nat. Rev. Drug. Discov. 10, 712–713 (2011).

    CAS  PubMed  Google Scholar 

  8. Begley, C. G. & Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature 483, 531–533 (2012).

    CAS  PubMed  Google Scholar 

  9. Lithgow, G. J., Driscoll, M. & Phillips, P. A long journey to reproducible results. Nature 548, 387–388 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  10. Collins, F. S. & Tabak, L. A. Policy: NIH plans to enhance reproducibility. Nature 505, 612–613 (2014).

    PubMed  PubMed Central  Google Scholar 

  11. Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13, e1002165 (2015).

    PubMed  PubMed Central  Google Scholar 

  12. Goodman, S. N., Fanelli, D. & Ioannidis, J. P. A. What does research reproducibility mean? Sci. Transl. Med. 8, 341ps12 (2016).

    PubMed  Google Scholar 

  13. Forsman, A. Rethinking phenotypic plasticity and its consequences for individuals, populations and species. Heredity 115, 276–284 (2015).

    CAS  PubMed  Google Scholar 

  14. West-Eberhardt, M. J. Developmental Plasticity and Evolution (Oxford University Press, 2003).

  15. Stearns, S. The evolutionary significance of phenotypic plasticity. Bioscience 39, 436–445 (2012).

    Google Scholar 

  16. Freund, J. et al. Emergence of individuality in genetically identical mice. Science 340, 756–759 (2013).

    CAS  PubMed  Google Scholar 

  17. Woltereck, R. Weitere experimentelle Untersuchungen über Artveränderung, speziell über das Wesen quantitativer Artunterschiede bei Daphnien. Verh. Dtsch. Zool. Ges. 19, 110–172 (1909).

    Google Scholar 

  18. Schmalhausen, I. Factors of Evolution; The Theory of Stabilizing Selection (Blakiston, 1949).

  19. Hartman IV, J. L., Garvik, B. & Hartwell, L. Cell biology: principles for the buffering of genetic variation. Science 291, 1001–1004 (2001).

    CAS  PubMed  Google Scholar 

  20. Halldorsdottir, T. & Binder, E. B. Gene×environment interactions: from molecular mechanisms to behavior. Annu. Rev. Psychol. 3, 215–241 (2017).

    Google Scholar 

  21. Meaney, M. J. Epigenetics and the biological definition of gene x environment interactions. Child. Dev. 81, 41–79 (2010).

    PubMed  Google Scholar 

  22. Cortijo, S. et al. Mapping the epigenetic basis of complex traits. Science 343, 1145–1148 (2014).

    CAS  PubMed  Google Scholar 

  23. Chesler, E. J., Wilson, S. G., Lariviere, W. R., Rodriguez-Zas, S. L. & Mogil, J. S. Influences of laboratory environment on behavior. Nat. Neurosci. 5, 1101–1102 (2002).

    CAS  PubMed  Google Scholar 

  24. Gururajan, A., Reif, A., Cryan, J. F. & Slattery, D. A. The future of rodent models in depression research. Nat. Rev. Neurosci. 20, 686–701 (2019).

    CAS  PubMed  Google Scholar 

  25. Beynen, A. C., Gärtner, K. & van Zutphen, L. F. M. in Principles of Laboratory Animal Science Ch. 5 (eds Zutphen, L. F. M., Baumans, V. & Beynen, A. C.) 103–110 (Elsevier, 2003).

  26. Laukens, D., Brinkman, B. M., Raes, J., De Vos, M. & Vandenabeele, P. Heterogeneity of the gut microbiome in mice: guidelines for optimizing experimental design. FEMS Microbiol. Rev. 40, 117–132 (2015).

    PubMed  PubMed Central  Google Scholar 

  27. Willmann, R. et al. Enhancing translation: guidelines for standard pre-clinical experiments in mdx mice. Neuromuscul. Disord. 22, 43–49 (2012).

    PubMed  Google Scholar 

  28. Holmes, C., McDonald, F., Jones, M., Ozdemir, V. & Graham, J. E. Standardization and omics science: technical and social dimensions are inseparable and demand symmetrical study. OMICS 14, 327–332 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  29. Richter, S. H., Garner, J. P. & Würbel, H. Environmental standardization: cure or cause of poor reproducibility in animal experiments? Nat. Methods 6, 257–261 (2009).

    CAS  PubMed  Google Scholar 

  30. Weihe, W. H. in Welfare and Science: Proceedings of the Fifth Symposium of the Federation of European Laboratory Animal Science Associations (eds Bunyan, J. & FELASA) 295–299 (Royal Society of Medicine Press, 1994).

  31. Gur, E. & Waner, T. The variability of organ weight background data in rats. Lab. Anim. 27, 65–72 (1993).

    CAS  PubMed  Google Scholar 

  32. Roe, F. J. C. Historical histopathological control data for laboratory rodents: valuable treasure or worthless trash? Lab. Anim. 28, 148–154 (1994).

    CAS  PubMed  Google Scholar 

  33. Festing, M. F. Refinement and reduction through the control of variation. Altern. Lab. Anim. 32, 259–263 (2004).

    CAS  PubMed  Google Scholar 

  34. Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Thomas Hartung’s Publications, 1959).

  35. Festing, M. F. W. Evidence should trump intuition by preferring inbred strains to outbred stocks in preclinical research. ILAR J. 55, 399–404 (2014).

    CAS  PubMed  Google Scholar 

  36. Tsai, P. P., Stelzer, H. D., Hedrich, H. J. & Hackbarth, H. Are the effects of different enrichment designs on the physiology and behaviour of DBA/2 mice consistent? Lab. Anim. 37, 314–327 (2003).

    CAS  PubMed  Google Scholar 

  37. Mogil, J. S. Sex differences in pain and pain inhibition: multiple explanations of a controversial phenomenon. Nat. Rev. Neurosci. 13, 859–866 (2012).

    CAS  PubMed  Google Scholar 

  38. Sorge, R. E. et al. Different immune cells mediate mechanical pain hypersensitivity in male and female mice. Nat. Neurosci. 18, 1081–1083 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  39. Crabbe, J. C., Wahlsten, D. & Dudek, B. C. Genetics of mouse behavior: interactions with laboratory environment. Science 284, 1670–1672 (1999).

    CAS  PubMed  Google Scholar 

  40. Loos, M. et al. Within-strain variation in behavior differs consistently between common inbred strains of mice. Mamm. Genome 26, 348–354 (2015).

    PubMed  Google Scholar 

  41. Prendergast, B. J., Onishi, K. G. & Zucker, I. Female mice liberated for inclusion in neuroscience and biomedical research. Neurosci. Biobehav. Rev. 40, 1–5 (2014).

    PubMed  Google Scholar 

  42. Kitano, H. Biological robustness. Nat. Rev. Genet. 5, 826–837 (2004).

    CAS  PubMed  Google Scholar 

  43. Mueller, F. S., Polesel, M., Richetto, J., Meyer, U. & Weber-Stadlbauer, U. Mouse models of maternal immune activation: mind your caging system! Brain Behav. Immun. 73, 643–660 (2018).

    CAS  PubMed  Google Scholar 

  44. Kallnik, M. et al. Impact of IVC housing on emotionality and fear learning in male C3HeB/FeJ and C57BL/6J mice. Mamm. Genome 18, 173–186 (2007).

    PubMed  Google Scholar 

  45. Logge, W., Kingham, J. & Karl, T. Behavioural consequences of IVC cages on male and female C57BL/6J mice. Neuroscience 237, 285–293 (2013).

    CAS  PubMed  Google Scholar 

  46. Logge, W., Kingham, J. & Karl, T. Do individually ventilated cage systems generate a problem for genetic mouse model research? Genes. Brain Behav. 13, 713–720 (2014).

    CAS  PubMed  Google Scholar 

  47. Åhlgren, J. & Voikar, V. Housing mice in the individually ventilated or open cages — does it matter for behavioral phenotype? Genes Brain Behav. 18, e12564 (2019).

    PubMed  PubMed Central  Google Scholar 

  48. Lazic, S. E. & Essioux, L. Improving basic and translational science by accounting for litter-to-litter variation in animal models. BMC Neurosci. 14, 37 (2013).

    PubMed  PubMed Central  Google Scholar 

  49. Kimmelman, J., Mogil, J. S. & Dirnagl, U. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol. 12, e1001863 (2014).

    PubMed  PubMed Central  Google Scholar 

  50. Garner, J. P. The significance of meaning: why do over 90% of behavioral neuroscience results fail to translate to humans, and what can we do to fix it? ILAR J. 55, 438–456 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  51. Nosek, B. A. & Errington, T. M. Reproducibility in cancer biology: making sense of replications. eLife 6, e23383 (2017).

    PubMed  PubMed Central  Google Scholar 

  52. Corrigan, J. K. et al. A big-data approach to understanding metabolic rate and response to obesity in laboratory mice. BioRxiv https://doi.org/10.1101/839076 (2019).

    Article  Google Scholar 

  53. van der Staay, F. J., Arndt, S. S. & Nordquist, R. E. The standardization–generalization dilemma: a way out. Genes, Brain Behav. 9, 849–855 (2010).

    Google Scholar 

  54. Amrhein, V., Trafimow, D. & Greenland, S. Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication. Am. Stat. 73, 262–270 (2019).

    Google Scholar 

  55. Servick, K. Of mice and microbes. Science 353, 741–743 (2016).

    CAS  PubMed  Google Scholar 

  56. Stappenbeck, T. S. & Virgin, H. W. Accounting for reciprocal host-microbiome interactions in experimental science. Nature 534, 191–199 (2016).

    CAS  PubMed  Google Scholar 

  57. van Driel, K. S. & Talling, J. C. Familiarity increases consistency in animal tests. Behav. Brain Res. 159, 243–245 (2005).

    PubMed  Google Scholar 

  58. Sorge, R. E. et al. Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nat. Methods 11, 629–632 (2014).

    CAS  PubMed  Google Scholar 

  59. Wahlsten, D. et al. Different data from different labs: Lessons from studies of gene-environment interaction. J. Neurobiol. 54, 283–311 (2003).

    PubMed  Google Scholar 

  60. Karp, N. A. et al. Impact of temporal variation on design and analysis of mouse knockout phenotyping studies. PLoS One 9, e111239 (2014).

    PubMed  PubMed Central  Google Scholar 

  61. Fisher, R. A. The Design of Experiments (Oliver and Boyd, 1935).

  62. Voelkl, B. & Würbel, H. Reproducibility crisis: are we ignoring reaction norms? Trends Pharmacol. Sci. 37, 509–510 (2016).

    CAS  PubMed  Google Scholar 

  63. Würbel, H. Behaviour and the standardization fallacy. Nat. Genet. 26, 263 (2000).

    PubMed  Google Scholar 

  64. Kafkafi, N. et al. Addressing reproducibility in single-laboratory phenotyping experiments. Nat. Methods 14, 462–464 (2017).

    CAS  PubMed  Google Scholar 

  65. Richter, S. H. et al. Effect of population heterogenization on the reproducibility of mouse behavior: a multi-laboratory study. PLoS One 6, e16461 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  66. Richter, S. H., Garner, J. P., Auer, C., Kunert, J. & Würbel, H. Systematic variation improves reproducibility of animal experiments. Nat. Methods 7, 167–168 (2010).

    CAS  PubMed  Google Scholar 

  67. Voelkl, B., Vogt, L., Sena, E. S. & Würbel, H. Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLoS Biol. 16, e2003693 (2018).

    PubMed  PubMed Central  Google Scholar 

  68. Bodden, C. et al. Heterogenising study samples across testing time improves reproducibility of behavioural data. Sci. Rep. 9, 8247 (2019).

    PubMed  PubMed Central  Google Scholar 

  69. Jonker, R. M., Guenther, A., Engqvist, L. & Schmoll, T. Does systematic variation improve the reproducibility of animal experiments? Nat. Methods 10, 373 (2013).

    CAS  PubMed  Google Scholar 

  70. Wolfinger, R. D. Reanalysis of Richter et al. (2010) on reproducibility. Nat. Methods 10, 373–374 (2013).

    CAS  PubMed  Google Scholar 

  71. Nelder, J. A. Statistics, science and technology. J. R. Stat. Soc. A 149, 109–121 (1986).

    Google Scholar 

  72. Mogil, J. S. & Macleod, M. R. No publication without confirmation. Nature 542, 409–411 (2017).

    CAS  PubMed  Google Scholar 

  73. Tukey, J. W. Exploratory Data Analysis (Addison-Wesley, 1977).

  74. Box, G. E. P. Science and statistics. J. Am. Stat. Assoc. 71, 791–799 (1976).

    Google Scholar 

  75. Will, T. R. et al. Problems and progress regarding sex bias and omission in neuroscience research. eNeuro e0278 (2017).

  76. Zucker, I. & Beery, A. K. Males still dominate animal studies. Nature 465, 690 (2010).

    CAS  PubMed  Google Scholar 

  77. Clayton, J. A. & Collins, F. S. NIH to balance sex in cell and animal studies. Nature 509, 282–283 (2014).

    PubMed  PubMed Central  Google Scholar 

  78. Krzywinski, M. & Altman, N. Points of significance: analysis of variance and blocking. Nat. Methods 11, 699–700 (2014).

    CAS  PubMed  Google Scholar 

  79. Miller, L. R. et al. Considering sex as a biological variable in preclinical research. FASEB J. 31, 29–34 (2016).

    PubMed  PubMed Central  Google Scholar 

  80. Würbel, H. More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research. Lab. Anim. 46, 164–166 (2017).

    Google Scholar 

  81. Paylor, R. Questioning standardization in science. Nat. Methods 6, 253–254 (2009).

    CAS  PubMed  Google Scholar 

  82. Karp, N. A. Reproducible preclinical research — is embracing variability the answer? PLoS Biol. 16, e2005413 (2018).

    PubMed  PubMed Central  Google Scholar 

  83. van der Staay, F. J., Arndt, S. S. & Nordquist, R. E. Evaluation of animal models of neurobehavioral disorders. Behav. Brain Funct. 5, 11 (2009).

    PubMed  PubMed Central  Google Scholar 

  84. Lewin, K. Frontiers in group dynamics: concept, method and reality in social science; social equilibria and social change. Hum. Relat. 1, 5–41 (1947).

    Google Scholar 

  85. Karp, N. A. & Reavey, N. Sex bias in preclinical research and an exploration of how to change the status quo. Br. J. Pharmacol. 176, 4107–4118 (2019).

    CAS  PubMed  Google Scholar 

  86. McNutt, M. Journals unite for reproducibility. Science 346, 679 (2014).

    CAS  PubMed  Google Scholar 

  87. Chia, R., Achilli, F., Festing, M. F. W. & Fisher, E. M. C. The origins and uses of mouse outbred stocks. Nat. Genet. 37, 1181–1186 (2005).

    CAS  PubMed  Google Scholar 

  88. National Centre for the Replacement, Refinement and Reduction of Animals in Research. Definitions of the 3Rs https://www.nc3rs.org.uk/the-3rs (2019).

  89. National Institutes of Health. Consideration of sex as a biological variable in NIH-funded research (notice no. NOT-OD-15-102). (National Institutes of Health, 2015).

  90. Organisation for Economic Co-operation and Development. OECD guidelines for the testing of chemicals 408 (Organisation for Economic Co-operation and Development, 2018).

  91. European Medicines Agency. ICH guideline M3(R2) on non-clinical safety studies for the conduct of human clinical trials and marketing authorisation for pharmaceuticals. EMA/CPMP/ICH/286/1995. (2013).

  92. National Centre for the Replacement, Refinement and Reduction of Animals in Research. ARRIVE guidelines https://www.nc3rs.org.uk/arrive-guidelines (2020).

  93. Nature. Nature Research Reporting Summary https://www.nature.com/documents/nr-reporting-summary.pdf (2020).

  94. Ioannidis, J. P. A., Fanelli, D., Dunne, D. D. & Goodman, S. N. Meta-research: evaluation and improvement of research methods and practices. PLoS Biol. 13, e1002264 (2015).

    PubMed  PubMed Central  Google Scholar 

  95. Forstmeier, W., Wagenmakers, E. J. & Parker, T. H. Detecting and avoiding likely false-positive findings–a practical guide. Biol. Rev. 92, 1941–1968 (2017).

    PubMed  Google Scholar 

  96. Jarvis, M. F. & Williams, M. Irreproducibility in preclinical biomedical research: perceptions, uncertainties, and knowledge gaps. Trends Pharmacol. Res. 37, 290–302 (2015).

    Google Scholar 

  97. Bishop, D. Rein in the four horsemen of irreproducibility. Nature 568, 435 (2019).

    CAS  PubMed  Google Scholar 

  98. Festing, M. F. Warning: the use of heterogeneous mice may seriously damage your research. Neurobiol. Aging 20, 237–244 (1999).

    CAS  PubMed  Google Scholar 

  99. Beck, J. A. et al. Genealogies of mouse inbred strains. Nat. Genet. 24, 23–25 (2000).

    CAS  PubMed  Google Scholar 

  100. Hsieh, L. S., Wen, J. H., Miyares, L., Lombroso, P. J. & Bordey, A. Outbred CD1 mice are as suitable as inbred C57BL/6J mice in performing social tasks. Neurosci. Lett. 637, 142–147 (2017).

    CAS  PubMed  Google Scholar 

  101. Silva, A. J. et al. Mutant mice and neuroscience: recommendations concerning genetic background. Neuron 19, 755–759 (1997).

    Google Scholar 

  102. Bogue, M. A., Churchill, G. A. & Chesler, E. J. Collaborative cross and diversity outbred data resources in the mouse phenome database. Mamm. Genome 26, 511–520 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  103. Tannenbaum, C., Ellis, R. P., Eyssel, F., Zou, J. & Schiebinger, L. Sex and gender analysis improves science and engineering. Nature 575, 137–146 (2019).

    CAS  PubMed  Google Scholar 

  104. Buch, T. et al. Benefits of a factorial design focusing on inclusion of female and male animals in one experiment. J. Mol. Med. 97, 871–877 (2019).

    PubMed  Google Scholar 

  105. Biggers, J. D. & Claringbold, P. J. Why use inbred lines? Nature 174, 596–597 (1954).

    CAS  PubMed  Google Scholar 

  106. Jensen, V. S., Porsgaard, T., Lykkesfeldt, J. & Hvid, H. Rodent model choice has major impact on variability of standard preclinical readouts associated with diabetes and obesity research. Am. J. Transl. Res. 8, 3574–3584 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  107. Tuttle, A. H., Philip, V. M., Chesler, E. J. & Mogil, J. S. Comparing phenotypic variation between inbred and outbred mice. Nat. Methods 15, 994–996 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  108. Lerner, I. M. Genetic Homeostasis (Oliver & Boyd, 1954).

  109. Crusio, W. E. Inheritance of behavioral and neuroanatomical phenotypical variance: Hybrid mice are not always more stable than inbreds. Behav. Genet. 36, 723–731 (2006).

    PubMed  Google Scholar 

  110. Gingrich, J. A. & Hen, R. The broken mouse: the role of development, plasticity and environment in the interpretation of phenotypic changes in knockout mice. Curr. Opin. Neurobiol. 10, 146–152 (2000).

    CAS  PubMed  Google Scholar 

  111. Ricceri, L., Moles, A. & Crawley, J. Behavioral phenotyping of mouse models of neurodevelopmental disorders: relevant social behavior patterns across the life span. Behav. Brain Res. 176, 40–52 (2007).

    PubMed  Google Scholar 

  112. Huang, K., Rabold, R., Schofield, B., Mitzner, W. & Tankersley, C. G. Age-dependent changes of airway and lung parenchyma in C57BL/6J mice. J. Appl. Physiol. 102, 200–206 (2007).

    PubMed  Google Scholar 

  113. Walker, C. L. et al. Protective effect of pregnancy for development of uterine leiomyoma. Carcinogenesis 22, 2049–2052 (2001).

    CAS  PubMed  Google Scholar 

  114. Carvalho-Freitas de, M. I. R. et al. Reproductive experience modifies dopaminergic function, serum levels of prolactin, and macrophage activity in female rats. Life Sci. 81, 128–136 (2007).

    Google Scholar 

  115. Ritzel, R. M. et al. Multiparity improves outcomes after cerebral ischemia in female mice despite features of increased metabovascular risk. Proc. Natl Acad. Sci. 114, E5673–E5682 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  116. Grimm, S. A. et al. DNA methylation in mice is influenced by genetics as well as sex and life experience. Nat. Commun. 10, 305 (2019).

    PubMed  PubMed Central  Google Scholar 

  117. Richetto, J., Polesel, M. & Weber-Stadlbauer, U. Effects of light and dark phase testing on the investigation of behavioural paradigms in mice: Relevance for behavioural neuroscience. Pharmacol. Biochem. Behav. 178, 19–29 (2019).

    CAS  PubMed  Google Scholar 

  118. Sousa, N., Almeida, O. F. X. & Wotjak, C. T. A hitchhiker’s guide to behavioral analysis in laboratory rodents. Genes Brain Behav. 5, 5–24 (2006).

    PubMed  Google Scholar 

  119. Beura, L. K. et al. Normalizing the environment recapitulates adult human immune traits in laboratory mice. Nature 532, 512–516 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  120. Rampon, C. et al. Enrichment induces structural changes and recovery from nonspatial memory deficits in CA1 NMDAR1-knockout mice. Nat. Neurosci. 3, 238–244 (2000).

    CAS  PubMed  Google Scholar 

  121. Freedman, L. S. et al. Inclusion of women and minorities in clinical trials and the NIH Revitalization Act of 1993 — the perspective of NIH clinical trialists. Control. Clin. Trials 16, 277–285 (1995).

    CAS  PubMed  Google Scholar 

  122. Gesensway, D. Reasons for sex-specific and gender-specific study of health topics. Ann. Intern. Med. 135, 935–938 (2001).

    CAS  PubMed  Google Scholar 

  123. Clayton, J. A. Studying both sexes: a guiding principle for biomedicine. FASEB J. 30, 519–524 (2015).

    PubMed  PubMed Central  Google Scholar 

  124. Clayton, J. A. Applying the new SABV (sex as a biological variable) policy to research and clinical care. Physiol. Behav. 187, 2–5 (2018).

    CAS  PubMed  Google Scholar 

  125. Arnold, A. P., van Nas, A. & Lusis, A. J. Systems biology asks new questions about sex differences. Trends Endocrinol. Metab. 20, 471–476 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  126. Hughes, R. N. Sex does matter: comments on the prevalence of male-only investigations of drug effects on rodent behaviour. Behav. Pharmacol. 18, 583–589 (2007).

    PubMed  Google Scholar 

  127. Wald, C. & Wu, C. Of mice and women: the bias in animal models. Science 327, 1571–1572 (2010).

    CAS  PubMed  Google Scholar 

  128. Jazin, E. & Cahill, L. Sex differences in molecular neuroscience: from fruit flies to humans. Nat. Rev. Neurosci. 11, 9–17 (2010).

    CAS  PubMed  Google Scholar 

  129. Arnold, A. P. et al. Ischemic nitric oxide and poly (ADP-ribose) polymerase-1 in cerebral ischemia: male toxicity, female protection. Proc. Natl Acad. Sci. 20, 565–572 (2015).

    Google Scholar 

  130. Beery, A. K. & Zucker, I. Sex bias in neuroscience and biomedical research. Neurosci. Biobehav. Rev. 35, 565–572 (2011).

    PubMed  Google Scholar 

  131. Klein, S. L. et al. Opinion: sex inclusion in basic research drives discovery. Proc. Natl Acad. Sci. 112, 5257–5258 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  132. Forsman, A. On the role of sex differences for evolution in heterogeneous and changing fitness landscapes: insights from pygmy grasshoppers. Philos. Trans. R. Soc. B Biol. Sci. 373, 20170429 (2018).

    Google Scholar 

  133. Yang, X. et al. Tissue-specific expression and regulation of sexually dimorphic genes in mice. Genome Res. 16, 995–1004 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  134. McCullough, L. D., Zeng, Z., Blizzard, K. K., Debchoudhury, I. & Hurn, P. D. Ischemic nitric oxide and poly (ADP-ribose) polymerase-1 in cerebral ischemia: male toxicity, female protection. J. Cereb. Blood Flow. Metab. 25, 502–512 (2005).

    CAS  PubMed  Google Scholar 

  135. Sandberg, K., Verbalis, J. G., Yosten, G. L. C. & Samson, W. K. Sex and basic science. A title IX position. Am. J. Physiol. Integr. Comp. Physiol. 307, R361–R365 (2014).

    CAS  Google Scholar 

  136. McCullough, L. D., McCarthy, M. M. & de Vries, G. J. NIH policy: status quo is also costly. Nature 510, 340 (2014).

    CAS  PubMed  Google Scholar 

  137. Fields, R. D. NIH policy: mandate goes too far. Nature 510, 340 (2014).

    CAS  PubMed  Google Scholar 

  138. Becker, J. B., Prendergast, B. J. & Liang, J. W. Female rats are not more variable than male rats: a meta-analysis of neuroscience studies. Biol. Sex Differ. 7, 34 (2016).

    PubMed  PubMed Central  Google Scholar 

  139. Cochran, W. G. & Cox, G. M. Experimental Design (John Wiley and Sons, 1957).

Download references

Acknowledgements

The Swiss National Science Foundation (IZSEZ0_184010) provided funding to B.V. for the workshop ‘Variation in in vivo experiments: the norm of reaction and reproducibility’. B.V., H.W. and M.J.K. were funded by the European Union Horizon 2020 research and innovation programme and the European Federation of Pharmaceutical Industries and Associations (Innovative Medicines Initiative, IMI2, grant agreement no. 777364, European Quality in Preclinical Data). A.F. thanks Linnaeus University for funding. H.S. was supported by the German Research Foundation (INST 215/543-1, 396782608). I.J. was funded by the Swiss National Science Foundation (310030_179254).

Author information

Authors and Affiliations

  1. Animal Welfare Division, Vetsuisse, University of Bern, Bern, Switzerland

    Bernhard Voelkl, Ivana Jaric & Hanno Würbel

  2. Department of Statistics, The Pennsylvania State University, University Park, PA, USA

    Naomi S. Altman

  3. Department of Biology and Environmental Science, Linnaeus University, Kalmar, Sweden

    Anders Forsman

  4. Department of Behavioural Ecology and Evolutionary Genetics, Max Planck Institute for Ornithology, Seewiesen, Germany

    Wolfgang Forstmeier

  5. Department of Ecology and Evolution, Stony Brook University, Stony Brook, NY, USA

    Jessica Gurevitch

  6. Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK

    Natasha A. Karp

  7. Groningen Institute for Evolutionary Life Sciences, University of Groningen, Groningen, Netherlands

    Martien J. Kas

  8. Institute of Ecology and Evolution, Friedrich Schiller University Jena, Jena, Germany

    Holger Schielzeth

  9. Statistics and Decision Sciences, Janssen R&D, Johnson & Johnson, Beerse, Belgium

    Tom Van de Casteele

Authors
  1. Bernhard Voelkl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Naomi S. Altman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anders Forsman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wolfgang Forstmeier
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jessica Gurevitch
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ivana Jaric
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Natasha A. Karp
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Martien J. Kas
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Holger Schielzeth
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tom Van de Casteele
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Hanno Würbel
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to all aspects of the article.

Corresponding author

Correspondence to Hanno Würbel.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Neuroscience thanks S. C. Stanford and F. J. van der Staay for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Glossary

Biological variation

The variation of phenotypes in a population of organisms. It is the result of genetic variation, environmental influences on the organism and gene × environment interactions.

Confirmatory studies

Studies designed to test specific hypotheses about the existence of a relationship or effect, its direction and magnitude using inferential statistical methods. The hypotheses are based on previous knowledge of the study system.

Exploratory studies

Studies designed to probe for relationships or treatment effects of novel interventions without specific hypotheses about the direction and size of effects. The outcome of an exploratory study is a descriptive account of the observed effects.

External validity

The extent to which findings can be generalized to the desired inference space of animals (including humans) and/or other environmental conditions.

Gene × environment interactions

These subsume the non-additive joint effect of genetic and environmental influences on the development of the phenotype. As a consequence, environmental influences can have different effects on the phenotype depending on the organism’s genotype or genes can have differential effects depending on features of the environment.

Genotypes

Organisms’ hereditary information as encoded in the genome.

Heterogenization

The deliberate augmentation of systematic or random biological variation in the study population.

Inference space

The range of organisms and environmental contexts for which the inference of an experiment is valid.

Internal validity

Refers to whether the effects observed in a study are due to manipulation of the independent variables and not some other, unknown factors.

Norm of reaction

A property of a genotype describing how an environmental factor affects the development of the phenotype. It can be conceptualized as a function mapping expected phenotypic trait values onto environmental parameter values.

Phenotype

The sum of an organism’s observable characteristics or traits, including its morphological, biochemical and physiological processes, behaviour and responses to external stimulation and treatments.

Phenotypic plasticity

The extent to which an organism changes its phenotype in response to environmental influences.

Random noise

Also known as measurement error, refers to unexplained variability in the data. It affects the variation but not the size of an experimental treatment effect.

Reproducibility

The ability to produce similar results by an independent replicate experiment using the same methodology in the same or a different laboratory.

Robustness

The ability of an organism to maintain a functioning phenotype under varying environmental conditions. It also refers to the stability of a response to an experimental treatment in the face of variation in environmental conditions.

3Rs principles

The guiding principles for a responsible approach to experimental animal research. They imply that a study involving the use of animals should be conducted only if the intended outcome cannot be achieved by use of no or non-sentient animals (replace), fewer animals (reduce) or procedures that are less harmful or improve animal well-being (refine).

Scientific rigour

As defined by the US National Institutes of Health, this means “the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. This includes full transparency in reporting experimental details so that others may reproduce and extend the findings”.

Standardization

The practice of minimizing both technical and biological variation in the study outcomes by identifying and controlling sources of variation that are believed to be putative confounders. Standardization can aim at aspects of the environment in which a study is conducted (environmental standardization), aspects of the study subjects (phenotype standardization) or aspects of how procedures and interventions are conducted and how measurements are taken (operational standardization).

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Voelkl, B., Altman, N.S., Forsman, A. et al. Reproducibility of animal research in light of biological variation. Nat Rev Neurosci 21, 384–393 (2020). https://doi.org/10.1038/s41583-020-0313-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41583-020-0313-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing