Comment | Published:

Justify your alpha

Nature Human Behaviourvolume 2pages168171 (2018) | Download Citation

In response to recommendations to redefine statistical significance to P ≤ 0.005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Benjamin, D. J. et al. Nat. Hum. Behav. 2, 6–10 (2018).

  2. 2.

    Wacholder, S., Chanock, S., Garcia-Closas, M., El Ghormli, L. & Rothman, N. J. Natl Cancer Inst. 96, 434–442 (2004).

  3. 3.

    Open Science Collaboration Science 349, aac4716 (2015).

  4. 4.

    Senn, S. Statistical Issues in Drug Development 2nd edn (John Wiley & Sons, Chichester, 2007).

  5. 5.

    Mayo, D. Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (Cambridge Univ. Press, Cambridge, 2018).

  6. 6.

    Johnson, V. E., Payne, R. D., Wang, T., Asher, A. & Mandal, S. J. Am. Stat. Assoc. 112, 1–10 (2017).

  7. 7.

    Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P. & Willmes, K. Cortex 66, A1–A2 (2015).

  8. 8.

    Lyons, L. Preprint at http://arxiv.org/abs/1310.1284 (2013).

  9. 9.

    Field, S. A., Tyre, A. J., Jonzen, N., Rhodes, J. R. & Possingham, H. P. Ecol. Lett. 7, 669–675 (2004).

  10. 10.

    Grieve, A. P. Pharm. Stat. 14, 139–150 (2015).

  11. 11.

    Mudge, J. F., Baker, L. F., Edge, C. B. & Houlahan, J. E. PLoS ONE 7, e32734 (2012).

  12. 12.

    Skipper, J. K., Guenther, A. L. & Nass, G. Am. Sociol. 2, 16–18 (1967).

  13. 13.

    Neyman, J. & Pearson, E. S. Phil. Trans. R. Soc. Lond. Ser. A 231, 694–706 (1933).

  14. 14.

    Fisher R. A. Statistical Methods and Scientific Inferences (Hafner, Oxford, 1956).

  15. 15.

    Casadevall, A. & Fang, F. C. mBio 7, e01902–16 (2016).

Download references

Acknowledgements

We thank D. Barr, F. Cheung, D. Colquhoun, H. IJzerman, H. Motulsky and R. Morey for helpful discussions while drafting this Comment. D.L. was supported by Nederlandse Organisatie voor Wetenschappelijk Onderzoek 452-17-013. F.G.A. was supported by CONICET. M.A.J.A. was funded by a Biotechnology and Biological Sciences Research Council AFL Fellowship (BB/M013596/1). G.S.C. was supported by the National Institute for Health Research Biomedical Research Centre, Oxford. Z.C. was supported by the Economic and Social Research Council (grant number C106891X). E.S.C. was supported by the European Research Council (ERC-2015-StG-677270). L.D. is supported by the European Research Council (ERC-2014-CoG-647910 KINSHIP). A.-L.v.H. is funded by a Royal Society Dorothy Hodgkin Fellowship (DH150176). M.R.H. was supported by the National Science Foundation under grant SBE SPRF-FR 1714446. J.Lao was supported by the Swiss National Science Foundation grant 100014_156490/1. C.L.d.O. was supported by Conselho Nacional de Desenvolvimento Científico e Tecnológico. A.E.M. was supported by the Economic and Social Research Council of the United Kingdom (grant number ES/K009095/1). J.-J.O.d.X. is supported by an internal grant from the KU Leuven (STG/14/054) and by the Fonds voor Wetenschappelijk Onderzoek (1519916N). S.P. was supported by the European Research Council (FP7/2007–2013; ERC grant agreement no. 324176). G.M.A.L. was funded by Nederlandse Organisatie voor Wetenschappelijk Onderzoek 453-14-016. S.G.S. is supported by a Cancer Research UK Fellowship (C42785/A17965). V.S. was supported by the National Institute for Neurological Disorders and Stroke Intramural Research Program (IRP). M.A.V. was supported by grant 2016-T1/SOC-1395 from Comunidad de Madrid. T.Y. was supported by National Institutes of Health award R01MH109682.

Author information

Author notes

  1. Unaffiliated: michael.ingre@gmail.com

Affiliations

  1. Human-Technology Interaction, Eindhoven University of Technology, Eindhoven, The Netherlands

    • Daniel Lakens
  2. National Scientific and Technical Research Council (CONICET), Buenos Aires, Argentina

    • Federico G. Adolfi
  3. Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany

    • Federico G. Adolfi
  4. Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands

    • Casper J. Albers
  5. College of Education, Psychology and Social Work, Flinders University, Adelaide, South Australia, Australia

    • Farid Anvari
  6. Department of Experimental Psychology, University of Oxford, Oxford, UK

    • Matthew A. J. Apps
    •  & Sam Parsons
  7. Department of Computer Science, Illinois Institute of Technology, Chicago, IL, USA

    • Shlomo E. Argamon
  8. Department of Psychology, Nottingham Trent University, Nottingham, UK

    • Thom Baguley
  9. Faculty of Linguistics and Literature, Bielefeld University, Bielefeld, Germany

    • Raymond B. Becker
  10. Psychology, University of Nevada, Las Vegas, Las Vegas, NV, USA

    • Stephen D. Benning
  11. Psychology, University of Wisconsin-Madison, Madison, WI, USA

    • Daniel E. Bradford
  12. Psychology, Missouri State University, Springfield, MO, USA

    • Erin M. Buchanan
  13. Health, Human Performance, and Recreation, University of Arkansas, Fayetteville, AR, USA

    • Aaron R. Caldwell
  14. Department of Development and Regeneration, KU Leuven, Leuven, Belgium

    • Ben Van Calster
  15. Department of Medical Statistics and Bioinformatics, Leiden University Medical Center, Leiden, The Netherlands

    • Ben Van Calster
  16. Department of Psychology, Linnaeus University, Kalmar, Sweden

    • Rickard Carlsson
  17. Department of Human Development and Psychology, Tzu-Chi University, Hualien City, Taiwan

    • Sau-Chin Chen
  18. Department of Surgery, University of British Columbia, Victoria, British Columbia, Canada

    • Bryan Chung
  19. Department of Psychology, University of Cambridge, Cambridge, UK

    • Lincoln J. Colling
  20. Centre for Statistics in Medicine, University of Oxford, Oxford, UK

    • Gary S. Collins
  21. Department of Psychology, School of Philosophy, Psychology and Language Sciences, The University of Edinburgh, Edinburgh, UK

    • Zander Crook
    •  & Andrea E. Martin
  22. School of Psychology, Bangor University, Bangor, UK

    • Emily S. Cross
  23. Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, UK

    • Emily S. Cross
    •  & Lisa DeBruine
  24. Ramsey Decision Theoretics, Washington, DC, USA

    • Sameera Daniels
  25. Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden

    • Henrik Danielsson
  26. College of Social Work, Florida State University, Tallahassee, FL, USA

    • Daniel J. Dunleavy
  27. Departments of Psychology and Philosophy, Yale University, New Haven, CT, USA

    • Brian D. Earp
  28. Department of English, University of Louisiana at Lafayette, Lafayette, LA, USA

    • Michele I. Feist
  29. Department of Psychology, St. Edward’s University, Austin, TX, USA

    • Jason D. Ferrell
  30. Department of Psychology, University of Texas at Austin, Austin, TX, USA

    • Jason D. Ferrell
    •  & Tal Yarkoni
  31. Department of Management, West Virginia University, Morgantown, WV, USA

    • James G. Field
  32. Department of Psychology, Rutgers University, New Brunswick, NJ, USA

    • Nicholas W. Fox
  33. Department of Political Science, Indiana University Purdue University, Indianapolis, IN, USA

    • Amanda Friesen
  34. Booking.com, Amsterdam, The Netherlands

    • Caio Gomes
  35. Department of English, American and Romance Studies, RWTH Aachen University, Aachen, Germany

    • Monica Gonzalez-Marquez
  36. School of Psychology, Keele University, Keele, UK

    • James A. Grange
  37. Centre of Excellence for Statistical Innovation, UCB Celltech, Slough, UK

    • Andrew P. Grieve
  38. Translational Neurosurgery, Eberhard Karls University Tübingen, Tübingen, Germany

    • Robert Guggenberger
  39. International Centre for Ethics in Sciences and Humanities, Eberhard Karls University Tübingen, Tübingen, Germany

    • Robert Guggenberger
  40. Department of Radiology, University of Cambridge, Cambridge, UK

    • James Grist
  41. Department of Psychiatry, University of Cambridge, Cambridge, UK

    • Anne-Laura van Harmelen
  42. Behavioural Science Institute, Radboud University Nijmegen, Nijmegen, The Netherlands

    • Fred Hasselman
  43. Department of Psychology, University of Chester, Chester, UK

    • Kevin D. Hochard
  44. Department of Psychology, New York University, New York, NY, USA

    • Mark R. Hoffarth
  45. School of Psychology, University of Nottingham, Nottingham, UK

    • Nicholas P. Holmes
    •  & Christopher R. Madan
  46. Department of Psychology, Education, and Child Studies, Erasmus University Rotterdam, Rotterdam, The Netherlands

    • Michael Ingre
    •  & Rolf A. Zwaan
  47. Department of Clinical and Experimental Medicine, University of Linköping, Linköping, Sweden

    • Peder M. Isager
  48. School of Clinical Sciences, University of Bristol, Bristol, UK

    • Hanna K. Isotalus
  49. Occupational Orthopaedics and Research, Sahlgrenska University Hospital, Gothenburg, Sweden

    • Christer Johansson
  50. The Faculty of Modern Languages and Literatures, Institute of Linguistics, Psycholinguistics Department, Adam Mickiewicz University, Poznań, Poland

    • Konrad Juszczyk
  51. Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA

    • David A. Kenny
  52. Center for Stroke Research Berlin, Charité Universitätsmedizin Berlin, Berlin, Germany

    • Ahmed A. Khalil
  53. Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany

    • Ahmed A. Khalil
  54. Berlin School of Mind and Brain, Humboldt-Universität zu Berlin, Berlin, Germany

    • Ahmed A. Khalil
  55. Social Sciences, Adam Mickiewicz University, Poznań, Poland

    • Barbara Konat
  56. Department of Psychology, University of Fribourg, Fribourg, Switzerland

    • Junpeng Lao
  57. School of Politics and International Relations, University of Kent, Canterbury, UK

    • Erik Gahner Larsen
  58. Department of Sociology/ICS, University of Groningen, Groningen, The Netherlands

    • Gerine M. A. Lodder
  59. Institute of Psychology, Czech Academy of Sciences, Prague, Czech Republic

    • Jiří Lukavský
  60. Pardee RAND Graduate School, RAND Corporation, Arlington, VA, USA

    • David Manheim
  61. Psychology and Neuroscience, Baylor University, Waco, TX, USA

    • Stephen R. Martin
  62. Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands

    • Andrea E. Martin
  63. Department of Philosophy, Virginia Tech, Blacksburg, VA, USA

    • Deborah G. Mayo
  64. Center for the Study of Family Violence and Sexual Assault, Northern Illinois University, DeKalb, IL, USA

    • Randy J. McCarthy
  65. School of Mathematics and Statistics, The Open University, Milton Keynes, UK

    • Kevin McConway
  66. Skyscanner, Edinburgh, UK

    • Colin McFarland
  67. School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK

    • Amanda Q. X. Nio
  68. Stress Research Institute, Stockholm University, Stockholm, Sweden

    • Gustav Nilsonne
  69. Department of Clinical Neuroscience, Stockholm, Sweden

    • Gustav Nilsonne
  70. Department of Psychology, Stanford University, Stanford, CA, USA

    • Gustav Nilsonne
  71. Laboratory of Behavioral Neurobiology, Department of Physiological Sciences, Federal University of Santa Catarina, Florianópolis, Brazil

    • Cilene Lino de Oliveira
  72. Department of Kinesiology, KU Leuven, Leuven, Belgium

    • Jean-Jacques Orban de Xivry
  73. Department of Psychology, UiT The Arctic University of Norway, Tromsø, Norway

    • Gerit Pfuhl
  74. Department of Psychology, DePaul University, Chicago, IL, USA

    • Kimberly A. Quinn
  75. Center for Neural Science, New York University, New York, NY, USA

    • John J. Sakon
  76. Department of Psychology, Boğaziçi University, Istanbul, Turkey

    • S. Adil Saribay
  77. Psychology, University of Cologne, Cologne, Germany

    • Iris K. Schneider
  78. Saudi Human Genome Program, King Abdulaziz City for Science and Technology (KACST), Riyadh, Saudi Arabia

    • Manojkumar Selvaraju
  79. Integrated Gulf Biosystems, Riyadh, Saudi Arabia

    • Manojkumar Selvaraju
  80. Cognitive Psychology Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands

    • Zsuzsika Sjoerds
  81. Leiden Institute for Brain and Cognition, Leiden University, Leiden, The Netherlands

    • Zsuzsika Sjoerds
  82. Leeds Institute of Health Sciences, University of Leeds, Leeds, UK

    • Samuel G. Smith
  83. Institute for Media Studies, KU Leuven, Leuven, Belgium

    • Tim Smits
  84. Center for Open Science, Charlottesville, VA, USA

    • Jeffrey R. Spies
  85. Department of Engineering and Society, University of Virginia, Charlottesville, VA, USA

    • Jeffrey R. Spies
  86. Surgical Neurology Branch, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA

    • Vishnu Sreekumar
  87. Department of Psychology, University of Southern Indiana, Evansville, IN, USA

    • Crystal N. Steltenpohl
  88. Life Sciences Communication, University of Wisconsin-Madison, Madison, WI, USA

    • Neil Stenhouse
  89. Department of Social Psychology, Institute of Psychology, University of Lausanne, Lausanne, Switzerland

    • Wojciech Świątkowski
  90. Departamento de Psicología Básica, Universidad Autónoma de Madrid, Madrid, Spain

    • Miguel A. Vadillo
  91. Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands

    • Marcel A. L. M. Van Assen
  92. Department of Sociology, Utrecht University, Utrecht, The Netherlands

    • Marcel A. L. M. Van Assen
  93. School of Psychology, Massey University, Auckland, New Zealand

    • Matt N. Williams
  94. Psychology, Saint Louis University, St. Louis, MO, USA

    • Samantha E. Williams
  95. Psychology, University of California, Davis, Davis, CA, USA

    • Donald R. Williams
  96. Marketing Department, Ghent University, Ghent, Belgium

    • Ignazio Ziano

Authors

  1. Search for Daniel Lakens in:

  2. Search for Federico G. Adolfi in:

  3. Search for Casper J. Albers in:

  4. Search for Farid Anvari in:

  5. Search for Matthew A. J. Apps in:

  6. Search for Shlomo E. Argamon in:

  7. Search for Thom Baguley in:

  8. Search for Raymond B. Becker in:

  9. Search for Stephen D. Benning in:

  10. Search for Daniel E. Bradford in:

  11. Search for Erin M. Buchanan in:

  12. Search for Aaron R. Caldwell in:

  13. Search for Ben Van Calster in:

  14. Search for Rickard Carlsson in:

  15. Search for Sau-Chin Chen in:

  16. Search for Bryan Chung in:

  17. Search for Lincoln J. Colling in:

  18. Search for Gary S. Collins in:

  19. Search for Zander Crook in:

  20. Search for Emily S. Cross in:

  21. Search for Sameera Daniels in:

  22. Search for Henrik Danielsson in:

  23. Search for Lisa DeBruine in:

  24. Search for Daniel J. Dunleavy in:

  25. Search for Brian D. Earp in:

  26. Search for Michele I. Feist in:

  27. Search for Jason D. Ferrell in:

  28. Search for James G. Field in:

  29. Search for Nicholas W. Fox in:

  30. Search for Amanda Friesen in:

  31. Search for Caio Gomes in:

  32. Search for Monica Gonzalez-Marquez in:

  33. Search for James A. Grange in:

  34. Search for Andrew P. Grieve in:

  35. Search for Robert Guggenberger in:

  36. Search for James Grist in:

  37. Search for Anne-Laura van Harmelen in:

  38. Search for Fred Hasselman in:

  39. Search for Kevin D. Hochard in:

  40. Search for Mark R. Hoffarth in:

  41. Search for Nicholas P. Holmes in:

  42. Search for Michael Ingre in:

  43. Search for Peder M. Isager in:

  44. Search for Hanna K. Isotalus in:

  45. Search for Christer Johansson in:

  46. Search for Konrad Juszczyk in:

  47. Search for David A. Kenny in:

  48. Search for Ahmed A. Khalil in:

  49. Search for Barbara Konat in:

  50. Search for Junpeng Lao in:

  51. Search for Erik Gahner Larsen in:

  52. Search for Gerine M. A. Lodder in:

  53. Search for Jiří Lukavský in:

  54. Search for Christopher R. Madan in:

  55. Search for David Manheim in:

  56. Search for Stephen R. Martin in:

  57. Search for Andrea E. Martin in:

  58. Search for Deborah G. Mayo in:

  59. Search for Randy J. McCarthy in:

  60. Search for Kevin McConway in:

  61. Search for Colin McFarland in:

  62. Search for Amanda Q. X. Nio in:

  63. Search for Gustav Nilsonne in:

  64. Search for Cilene Lino de Oliveira in:

  65. Search for Jean-Jacques Orban de Xivry in:

  66. Search for Sam Parsons in:

  67. Search for Gerit Pfuhl in:

  68. Search for Kimberly A. Quinn in:

  69. Search for John J. Sakon in:

  70. Search for S. Adil Saribay in:

  71. Search for Iris K. Schneider in:

  72. Search for Manojkumar Selvaraju in:

  73. Search for Zsuzsika Sjoerds in:

  74. Search for Samuel G. Smith in:

  75. Search for Tim Smits in:

  76. Search for Jeffrey R. Spies in:

  77. Search for Vishnu Sreekumar in:

  78. Search for Crystal N. Steltenpohl in:

  79. Search for Neil Stenhouse in:

  80. Search for Wojciech Świątkowski in:

  81. Search for Miguel A. Vadillo in:

  82. Search for Marcel A. L. M. Van Assen in:

  83. Search for Matt N. Williams in:

  84. Search for Samantha E. Williams in:

  85. Search for Donald R. Williams in:

  86. Search for Tal Yarkoni in:

  87. Search for Ignazio Ziano in:

  88. Search for Rolf A. Zwaan in:

Contributions

D.L., N.W.F., M.G.-M., J.A.G., N.P.H., A.A.K., S.R.M., V.S., S.D.B. and C.N.S. participated in brainstorming, drafting the Comment and data analysis. C.J.A., S.E.A., T.B., E.M.B., B.V.C., Z.C., G.S.C., S.D., D.J.D., B.D.E., J.D.F., J.G.F., A.-L.v.H., M.I., P.M.I., H.K.I., J.Lao, G.M.A.L., D.M., A.E.M., K.M., A.Q.X.N., G.N., C.L.d.O., J.-J.O.d.X., G.P, K.A.Q., I.K.S., Z.S., S.G.S., J.R.S., M.A.L.M.V.A., M.N.W., D.R.W., T.Y. and R.A.Z. participated in brainstorming and drafting the Comment. F.G.A., R.B.B., M.I.F., F.H. and S.P. participated in drafting the Comment and data analysis. M.A.J.A., D.E.B., S.-C.C., B.C., L.J.C., H.D., L.D., M.R.H., E.G.L., R.J.M., J.J.S., S.A.S., T.S., N.S., W.Ś. and M.A.V. participated in brainstorming. F.A., A.R.C., R.C., E.S.C., A.F, C.G., A.P.G., R.G., J.G., K.D.H, C.J., K.J, D.A.K., B.K., J.Lukavský, C.R.M., D.G.M., C.M., M.S., S.E.W. I.Z. did not participate in drafting the Comment because the points that they would have raised had already been incorporated, or they endorse a sufficiently large part of the contents as if participation had occurred. Except for the first author, authorship order is alphabetical.

Competing interests

The authors declare no competing interests.

Corresponding author

Correspondence to Daniel Lakens.

About this article

Publication history

Published

DOI

https://doi.org/10.1038/s41562-018-0311-x

Newsletter Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing