Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Confidence intervals are no salvation from the alleged fickleness of the P value

Nature Methods volume 13, pages 605–606 (2016)Cite this article

Subjects

In this journal, Halsey et al.1 claimed that the “fickle” P value generates irreproducible results. They proposed discounting it and instead relying on a careful inspection of effect sizes and confidence intervals (CIs). Their condemnation relies on simulations with normally distributed random numbers showing the instability of the acceptance or rejection of the null hypothesis using the P value, especially with small samples. Interestingly, the Open Science Collaboration consortium2 recently reported an impressive replication of 100 published studies in psychological science, in which no more than 36% of the original results could be reproduced, and showed that the P value was the best predictor of replication success. How should one interpret this apparent contradiction? Should the P value be banned or praised? Has it been made a scapegoat for sins of other sources?

There is no doubt that Halsey et al.1 raise an important problem regarding the instability of small-sampled tests. They are right to criticize a common misinterpretation of the P value and to promote unit-based measurements (such as effect sizes and CIs) for their interpretative value. However, it is misleading to ascribe the lack of reproducibility to some intrinsic fickleness of the P value. A P value is neither stable nor unstable by itself; its fluctuations reflect only the lack of robustness of the underlying test, which may result from a variety of limitations (e.g., sample sizes, imprecision of the measures, design biases or batch effects).

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Combining volcano plots and CIs.

References

  1. Halsey, L.G., Curran-Everett, D., Vowler, S.L. & Drummond, G.B. Nat. Methods 12, 179–185 (2015).

    Article  CAS  Google Scholar 

  2. Open Science Collaboration Science 349, aac4716 (2015).

  3. Storey, J.D. & Tibshirani, R. Proc. Natl. Acad. Sci. USA 100, 9440–9445 (2003).

    Article  CAS  Google Scholar 

  4. Wasserstein, R.L. & Lazar, N.A. Am. Stat. 70, 129–133 (2016).

    Article  Google Scholar 

  5. Trafimow, D. & Marks, M. Basic Appl. Soc. Psych. 37, 1–2 (2015).

    Article  Google Scholar 

Download references

Acknowledgements

I am grateful to G. Lima-Mendez for detailed corrections of the successive versions of the manuscript, and for numerous suggestions that contributed to its improvement. I also thank M. Zytnicki, L. Spinelli, P. Rihet, D. Puthier, A. Saurin, G. Nuel and A. Stevens for constructive comments and discussions.

Author information

Authors and Affiliations

  1. Aix-Marseille Univ, INSERM, TAGC, Marseille, France

    Jacques van Helden

Authors
  1. Jacques van Helden
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jacques van Helden.

Ethics declarations

Competing interests

The author declares no competing financial interests.

Supplementary information

Supplementary Table 1

Equivalence between p-value based acceptance of H0 and overlap of the 0 value by the confidence interval. (PDF 86 kb)

Supplementary Data 1

R library stats4bioinfo (ZIP 974 kb)

Supplementary Data 2

Script used to produce the results and Figure 1 (TXT 18 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

van Helden, J. Confidence intervals are no salvation from the alleged fickleness of the P value. Nat Methods 13, 605–606 (2016). https://doi.org/10.1038/nmeth.3932

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.3932

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing