Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Justify your alpha

In response to recommendations to redefine statistical significance to P ≤ 0.005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: The proportion of studies3 replicated at α = 0.05 (with a bin width of 0.005).

References

  1. 1.

    Benjamin, D. J. et al. Nat. Hum. Behav. 2, 6–10 (2018).

    Article  Google Scholar 

  2. 2.

    Wacholder, S., Chanock, S., Garcia-Closas, M., El Ghormli, L. & Rothman, N. J. Natl Cancer Inst. 96, 434–442 (2004).

    Article  PubMed  Google Scholar 

  3. 3.

    Open Science Collaboration Science 349, aac4716 (2015).

  4. 4.

    Senn, S. Statistical Issues in Drug Development 2nd edn (John Wiley & Sons, Chichester, 2007).

  5. 5.

    Mayo, D. Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (Cambridge Univ. Press, Cambridge, 2018).

  6. 6.

    Johnson, V. E., Payne, R. D., Wang, T., Asher, A. & Mandal, S. J. Am. Stat. Assoc. 112, 1–10 (2017).

    CAS  Article  Google Scholar 

  7. 7.

    Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P. & Willmes, K. Cortex 66, A1–A2 (2015).

    Article  PubMed  Google Scholar 

  8. 8.

    Lyons, L. Preprint at http://arxiv.org/abs/1310.1284 (2013).

  9. 9.

    Field, S. A., Tyre, A. J., Jonzen, N., Rhodes, J. R. & Possingham, H. P. Ecol. Lett. 7, 669–675 (2004).

    Article  Google Scholar 

  10. 10.

    Grieve, A. P. Pharm. Stat. 14, 139–150 (2015).

    Article  PubMed  Google Scholar 

  11. 11.

    Mudge, J. F., Baker, L. F., Edge, C. B. & Houlahan, J. E. PLoS ONE 7, e32734 (2012).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Skipper, J. K., Guenther, A. L. & Nass, G. Am. Sociol. 2, 16–18 (1967).

    Google Scholar 

  13. 13.

    Neyman, J. & Pearson, E. S. Phil. Trans. R. Soc. Lond. Ser. A 231, 694–706 (1933).

    Article  Google Scholar 

  14. 14.

    Fisher R. A. Statistical Methods and Scientific Inferences (Hafner, Oxford, 1956).

  15. 15.

    Casadevall, A. & Fang, F. C. mBio 7, e01902–16 (2016).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank D. Barr, F. Cheung, D. Colquhoun, H. IJzerman, H. Motulsky and R. Morey for helpful discussions while drafting this Comment. D.L. was supported by Nederlandse Organisatie voor Wetenschappelijk Onderzoek 452-17-013. F.G.A. was supported by CONICET. M.A.J.A. was funded by a Biotechnology and Biological Sciences Research Council AFL Fellowship (BB/M013596/1). G.S.C. was supported by the National Institute for Health Research Biomedical Research Centre, Oxford. Z.C. was supported by the Economic and Social Research Council (grant number C106891X). E.S.C. was supported by the European Research Council (ERC-2015-StG-677270). L.D. is supported by the European Research Council (ERC-2014-CoG-647910 KINSHIP). A.-L.v.H. is funded by a Royal Society Dorothy Hodgkin Fellowship (DH150176). M.R.H. was supported by the National Science Foundation under grant SBE SPRF-FR 1714446. J.Lao was supported by the Swiss National Science Foundation grant 100014_156490/1. C.L.d.O. was supported by Conselho Nacional de Desenvolvimento Científico e Tecnológico. A.E.M. was supported by the Economic and Social Research Council of the United Kingdom (grant number ES/K009095/1). J.-J.O.d.X. is supported by an internal grant from the KU Leuven (STG/14/054) and by the Fonds voor Wetenschappelijk Onderzoek (1519916N). S.P. was supported by the European Research Council (FP7/2007–2013; ERC grant agreement no. 324176). G.M.A.L. was funded by Nederlandse Organisatie voor Wetenschappelijk Onderzoek 453-14-016. S.G.S. is supported by a Cancer Research UK Fellowship (C42785/A17965). V.S. was supported by the National Institute for Neurological Disorders and Stroke Intramural Research Program (IRP). M.A.V. was supported by grant 2016-T1/SOC-1395 from Comunidad de Madrid. T.Y. was supported by National Institutes of Health award R01MH109682.

Author information

Affiliations

Authors

Contributions

D.L., N.W.F., M.G.-M., J.A.G., N.P.H., A.A.K., S.R.M., V.S., S.D.B. and C.N.S. participated in brainstorming, drafting the Comment and data analysis. C.J.A., S.E.A., T.B., E.M.B., B.V.C., Z.C., G.S.C., S.D., D.J.D., B.D.E., J.D.F., J.G.F., A.-L.v.H., M.I., P.M.I., H.K.I., J.Lao, G.M.A.L., D.M., A.E.M., K.M., A.Q.X.N., G.N., C.L.d.O., J.-J.O.d.X., G.P, K.A.Q., I.K.S., Z.S., S.G.S., J.R.S., M.A.L.M.V.A., M.N.W., D.R.W., T.Y. and R.A.Z. participated in brainstorming and drafting the Comment. F.G.A., R.B.B., M.I.F., F.H. and S.P. participated in drafting the Comment and data analysis. M.A.J.A., D.E.B., S.-C.C., B.C., L.J.C., H.D., L.D., M.R.H., E.G.L., R.J.M., J.J.S., S.A.S., T.S., N.S., W.Ś. and M.A.V. participated in brainstorming. F.A., A.R.C., R.C., E.S.C., A.F, C.G., A.P.G., R.G., J.G., K.D.H, C.J., K.J, D.A.K., B.K., J.Lukavský, C.R.M., D.G.M., C.M., M.S., S.E.W. I.Z. did not participate in drafting the Comment because the points that they would have raised had already been incorporated, or they endorse a sufficiently large part of the contents as if participation had occurred. Except for the first author, authorship order is alphabetical.

Corresponding author

Correspondence to Daniel Lakens.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lakens, D., Adolfi, F.G., Albers, C.J. et al. Justify your alpha. Nat Hum Behav 2, 168–171 (2018). https://doi.org/10.1038/s41562-018-0311-x

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing