A call for transparent reporting to optimize the predictive value of preclinical research

Journal name:
Nature
Volume:
490,
Pages:
187–191
Date published:
DOI:
doi:10.1038/nature11556
Received
Accepted
Published online

The US National Institute of Neurological Disorders and Stroke convened major stakeholders in June 2012 to discuss how to improve the methodological reporting of animal studies in grant applications and publications. The main workshop recommendation is that at a minimum studies should report on sample-size estimation, whether and how animals were randomized, whether investigators were blind to the treatment, and the handling of data. We recognize that achieving a meaningful improvement in the quality of reporting will require a concerted effort by investigators, reviewers, funding agencies and journal editors. Requiring better reporting of animal studies will raise awareness of the importance of rigorous study design to accelerate scientific progress.

References

  1. Begley, C. G. & Ellis, L. M. Raise standards for preclinical cancer research. Nature 483, 531533 (2012)
  2. Hess, K. R. Statistical design considerations in animal studies published recently in Cancer Research. Cancer Res. 71, 625 (2011)
  3. Kilkenny, C. et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS ONE 4, e7824 (2009)
  4. Moher, D., Simera, I., Schulz, K. F., Hoey, J. & Altman, D. G. Helping editors, peer reviewers and authors improve the clarity, completeness and transparency of reporting health research. BMC Med. 6, 13 (2008)
  5. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discov. 10, 712 (2011)The first report that many published studies cannot be reproduced by the pharmaceutical industry.
  6. Sena, E., van der Worp, H. B., Howells, D. & Macleod, M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 30, 433439 (2007)
  7. Steward, O., Popovich, P. G., Dietrich, W. D. & Kleitman, N. Replication and reproducibility in spinal cord injury research. Exp. Neurol. 233, 597605 (2012)
  8. van der Worp, H. B. & Macleod, M. R. Preclinical studies of human disease: time to take methodological quality seriously. J. Mol. Cell. Cardiol. 51, 449450 (2011)
  9. Hackam, D. G. & Redelmeier, D. A. Translation of research evidence from animals to humans. J. Am. Med. Assoc. 296, 17271732 (2006)A study reporting that a large fraction of high-impact publications in highly reputable journals lack important information related to experimental design.
  10. Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 28242829 (2008)A study demonstrating that lack of reporting of key methodological parameters is associated with bias.
  11. Bebarta, V., Luyten, D. & Heard, K. Emergency medicine animal research: does use of randomization and blinding affect the results? Acad. Emerg. Med. 10, 684687 (2003)
  12. Crossley, N. A. et al. Empirical evidence of bias in the design of experimental stroke studies – A metaepidemiologic approach. Stroke 39, 929934 (2008)
  13. Rooke, E. D., Vesterinen, H. M., Sena, E. S., Egan, K. J. & Macleod, M. R. Dopamine agonists in animal models of Parkinson’s disease: a systematic review and meta-analysis. Parkinsonism Relat. Disord. 17, 313320 (2011)
  14. Vesterinen, H. M. et al. Improving the translational hit of experimental treatments in multiple sclerosis. Mult. Scler. J. 16, 10441055 (2010)
  15. Stroke Therapy Academic Industry Roundtable (STAIR). Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 30, 27522758 (1999)
  16. Fanelli, D. “Positive” results increase down the hierarchy of the sciences. PLoS ONE 5, e10068 (2010)
  17. Jerndal, M. et al. A systematic review and meta-analysis of erythropoietin in experimental stroke. J. Cereb. Blood Flow Metab. 30, 961968 (2010)
  18. Macleod, M. R., O’Collins, T., Horky, L. L., Howells, D. W. & Donnan, G. A. Systematic review and metaanalysis of the efficacy of FK506 in experimental stroke. J. Cereb. Blood Flow Metab. 25, 713721 (2005)
  19. Sena, E. S. et al. Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J. Cereb. Blood Flow Metab. 30, 19051913 (2010)
  20. Wheble, P. C. R., Sena, E. S. & Macleod, M. R. A systematic review and meta-analysis of the efficacy of piracetam and piracetam-like compounds in experimental stroke. Cerebrovasc. Dis. 25, 511 (2008)
  21. Festing, M. F. & Altman, D. G. Guidelines for the design and statistical analysis of experiments using laboratory animals. ILAR J. 43, 244258 (2002)
  22. Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010)
  23. van der Worp, H. B. et al. Can animal models of disease reliably inform human studies? PLoS Med. 7, e1000245 (2010)
  24. Fisher, M. et al. Update of the stroke therapy academic industry roundtable preclinical recommendations. Stroke 40, 22442250 (2009)
  25. Ludolph, A. C. et al. Guidelines for preclinical animal research in ALS/MND: a consensus meeting. Amyotroph. Lateral Scler. 11, 3845 (2010)
  26. Shineman, D. W. et al. Accelerating drug discovery for Alzheimer’s disease: best practices for preclinical animal studies. Alzheimers Res. Ther. 3, 28 (2011)
  27. Unger, E. F. All is not well in the world of translational research. J. Am. Coll. Cardiol. 50, 738740 (2007)
  28. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005)
  29. Dienes, Z. Bayesian versus orthodox statistics: which side are you on? Perspect. Psychol. Sci. 6, 274290 (2011)
  30. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 13591366 (2011)
  31. Beal, K. G. & Khamis, H. J. A problem in statistical-analysis: simultaneous inference. Condor 93, 10231025 (1991)
  32. Lazic, S. E. The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? BMC Neurosci. 11, 5 (2010)
  33. Scott, S. et al. Design, power, and interpretation of studies in the standard murine model of ALS. Amyotroph. Lateral Scler. 9, 415 (2008)An enlightening analysis of how small sample sizes can lead to false-positive outcomes.
  34. Proschan, M. A. & Waclawiw, M. A. Practical guidelines for multiplicity adjustment in clinical trials. Control. Clin. Trials 21, 527539 (2000)
  35. Festing, M. F. W. Design and statistical methods in studies using animal models of development. ILAR J. 47, 514 (2006)
  36. Nakagawa, S. & Cuthill, I. C. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol. Rev. Camb. Philos. Soc. 82, 591605 (2007)
  37. Chalmers, T. C., Celano, P., Sacks, H. S. & Smith, H. Bias in treatment assignment in controlled clinical-trials. N. Engl. J. Med. 309, 13581361 (1983)
  38. Jüni, P., Altman, D. G. & Egger, M. Systematic reviews in health care - assessing the quality of controlled clinical trials. Br. Med. J. 323, 42 (2001)
  39. Pildal, J. et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int. J. Epidemiol. 36, 847857 (2007)
  40. Pocock, S. J., Hughes, M. D. & Lee, R. J. Statistical problems in the reporting of clinical-trials. A survey of three medical journals. N. Engl. J. Med. 317, 426432 (1987)
  41. Schulz, K. F., Chalmers, I., Hayes, R. J. & Altman, D. G. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J. Am. Med. Assoc. 273, 408412 (1995)
  42. Wood, L. et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. Br. Med. J. 336, 601605 (2008)
  43. Moher, D. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. Br. Med. J. 340, c869 (2011)
  44. Moher, D., Schulz, K. F. & Altman, D. G. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357, 11911194 (2001)Revision of guidelines by the CONSORT group to improve the reporting of randomized clinical trials.
  45. Schulz, K. F., Altman, D. G. & Moher, D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 7, e1000251 (2010)
  46. Plint, A. C. et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med. J. Aust. 185, 263267 (2006)
  47. Kane, R. L., Wang, J. & Garrard, J. Reporting in randomized clinical trials improved after adoption of the CONSORT statement. J. Clin. Epidemiol. 60, 241249 (2007)
  48. Prady, S. L., Richmond, S. J., Morton, V. M. & Macpherson, H. A systematic evaluation of the impact of STRICTA and CONSORT recommendations on quality of reporting for acupuncture trials. PLoS ONE 3, e1577 (2008)
  49. Smith, B. A. et al. Quality of reporting randomized controlled trials (RCTs) in nursing literature: application of the consolidated standards reporting trials (CONSORT). Nurs. Outlook 56, 3137 (2008)
  50. Macleod, M. R., O’Collins, T., Howells, D. W. & Donnan, G. A. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 35, 12031208 (2004)
  51. Macleod, M. R., O’Collins, T., Horky, L. L., Howells, D. W. & Donnan, G. A. Systematic review and meta-analysis of the efficacy of melatonin in experimental stroke. J. Pineal Res. 38, 3541 (2005)
  52. Gallo, J. M. Pharmacokinetic/pharmacodynamic-driven drug development. Mt. Sinai J. Med. 77, 381388 (2010)
  53. Moher, D. et al. Describing reporting guidelines for health research: a systematic review. J. Clin. Epidemiol. 64, 718742 (2011)
  54. Callaham, M. L., Wears, R. L., Weber, E. J., Barton, C. & Young, G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. J. Am. Med. Assoc. 280, 254257 (1998)
  55. Dickersin, K. & Chalmers, I. Recognizing, investigation and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO. J. R. Soc. Med. 104, 532538 (2011)
  56. Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891904 (2012)
  57. Kyzas, P. A., Denaxa-Kyza, D. & Ioannidis, J. P. A. Almost all articles on cancer prognostic markers report statistically significant results. Eur. J. Cancer 43, 25592579 (2007)
  58. Liu, S. Dealing with publication bias in translational stroke research. J. Exp. Stroke Transl. Med. 2, 1621 (2009)
  59. Rockwell, S., Kimler, B. E. & Moulder, J. E. Publishing negative results: the problem of publication bias. Radiat. Res. 165, 623625 (2006)
  60. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638641 (1979)
  61. Sterling, T. D. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J. Am. Stat. Assoc. 54, 3034 (1959)
  62. Song, F. et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol. Assess. 14, 1220 (2010)
  63. Sena, E. S., van der Worp, H. B., Bath, P. M. W., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010)
  64. Fanelli, D. Do pressures to publish increase scientists’ bias? An empirical support from US states data. PLoS ONE 5, e10271 (2010)

Download references

Author information

Affiliations

  1. National Institute of Neurological Disorders and Stroke, NIH, Bethesda, Maryland 20892, USA

    • Story C. Landis,
    • Robert Finkelstein,
    • Amelie K. Gubitz,
    • Walter Koroshetz,
    • John D. Porter,
    • Ursula Utz &
    • Shai D. Silberberg
  2. Department of Neurobiology, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania 15213, USA

    • Susan G. Amara
  3. Bayer HealthCare, 13342 Berlin, Germany

    • Khusru Asadullah
  4. National Center for Advancing Translational Sciences, NIH, Rockville, Maryland 20854, USA

    • Chris P. Austin
  5. CHDI Management/CHDI Foundation, New York, New York 10001, USA

    • Robi Blumenstein
  6. Center for Review, NIH, Bethesda, Maryland 20892, USA

    • Eileen W. Bradley
  7. Department of Genetic Medicine, Weill Cornell Medical College, New York, New York 10021, USA

    • Ronald G. Crystal
  8. Howard Hughes Medical Institute, The Rockefeller University, New York, New York 10065, USA

    • Robert B. Darnell
  9. Department of Neurological Surgery, University of Pittsburgh, Pittsburgh, Pennsylvania 15213, USA

    • Robert J. Ferrante
  10. Alzheimer's Drug Discovery Foundation, New York, New York 10019, USA

    • Howard Fillit
  11. Department of Neurology, University of Massachusetts Medical School, Worcester, Massachusetts 01545, USA

    • Marc Fisher
  12. Department of Pharmacology and Experimental Neuroscience, University of Nebraska Medical Center, Omaha, Nebraska 68198, USA

    • Howard E. Gendelman
  13. JAMA, Chicago, Illinois 60654, USA

    • Robert M. Golub
  14. Department of Neurology, Michigan State University, East Lansing, Michigan 48824, USA

    • John L. Goudreau
  15. Department of Neurology, University of Rochester Medical Center, Rochester, New York 14642, USA

    • Robert A. Gross
  16. Parent Project Muscular Dystrophy, Hackensack, New Jersey 07601, USA

    • Sharon E. Hesterlee
  17. The Florey Institute of Neuroscience and Mental Health, University of Melbourne, Heidelberg 3081, Australia

    • David W. Howells
  18. Neurology and Neurological Sciences and Cellular and Molecular Physiology, Stanford University, Stanford, California 94305, USA

    • John Huguenard
  19. Science Translational Medicine, AAAS, Washington DC 22201, USA

    • Katrina Kelner
  20. Department of Neurology, Harvard Medical School, Massachusetts General Hospital, Boston, Massachusetts 02114, USA

    • Dimitri Krainc
  21. F. Hoffmann-La Roche, 4070 Basel, Switzerland

    • Stanley E. Lazic
  22. Department of Psychiatry and Biobehavioral Sciences, University of California Los Angeles, Los Angeles, California 90095, USA

    • Michael S. Levine
  23. Department of Clinical Neurosciences, University of Edinburgh, Western General Hospital, Edinburgh EH4 2XU, UK

    • Malcolm R. Macleod
  24. PharMac LLC, Boca Grande, Florida 33921, USA

    • John M. McCall
  25. University of Rochester Medical Center, School of Medicine and Dentistry, Rochester, New York 14642, USA

    • Richard T. Moxley III
  26. Nature Neuroscience, New York, New York 10013, USA

    • Kalyani Narasimhan
  27. Department of Neurological Surgery, University of California San Francisco, San Francisco, California 94143, USA

    • Linda J. Noble
  28. ALS Therapy Development Institute, Cambridge, Massachusetts 02139, USA

    • Steve Perrin
  29. Reeve-Irvine Research Center, University of California Irvine, Irvine, California 92697, USA

    • Oswald Steward
  30. Office of New Drugs, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, Maryland 20993, USA

    • Ellis Unger

Contributions

R.F., A.K.G., S.C.L., J.D.P., S.D.S., U.U. and W.K. organized the workshop. R.B.D., S.E.L., S.C.L., M.R.M. and S.D.S. wrote the manuscript. All authors participated in the workshop and contributed to the editing of the manuscript.

Competing financial interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to:

Author details

Additional data