Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Solutions for quantifying P-value uncertainty and replication power

Nature Methods volume 13, pages 107–108 (2016)Cite this article

Subjects

To the Editor:

We congratulate Halsey et al. for their important discussion of the random nature of P values in an earlier issue of this journal1. P values from identical experiments can differ greatly in a way that is surprising to many2,3,4,5,6. The failure to appreciate this wide variability can lead researchers to expect, without adequate justification, that statistically significant findings will be replicated, only to be disappointed later. Thus, we agree with the authors that “Data ... interpretation must incorporate the uncertainty embedded in a P value.” Herein, by describing novel statistical intervals to assess P-value uncertainty, we provide the means to easily do just that.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Confidence limits for power based on observed P.

References

  1. Halsey, L.G., Curran-Everett, D., Vowler, S.L. & Drummond, G.B. Nat. Methods 12, 179–185 (2015).

    Article  CAS  Google Scholar 

  2. Goodman, S.N. Stat. Med. 11, 875–879 (1992).

    Article  CAS  Google Scholar 

  3. Cumming, G. Perspect. Psychol. Sci. 3, 286–300 (2008).

    Article  Google Scholar 

  4. Boos, D.D. & Stefanski, L.A. Am. Stat. 65, 213–221 (2011).

    Article  Google Scholar 

  5. Lazzeroni, L.C., Lu, Y. & Belitskaya-Lévy, I. Mol. Psychiatry 19, 1336–1340 (2014).

    Article  CAS  Google Scholar 

  6. Nuzzo, R. Nature 506, 150–152 (2014).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank the reviewers for their helpful comments and suggestions. This work was supported by the US National Institutes of Health (grant MH086135 to L.C.L. as part of the Consortium on the Genetics of Schizophrenia (COGS)) and the Cooperative Studies Program of the US Department of Veterans Affairs.

Author information

Authors and Affiliations

  1. Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, California, USA

    Laura C Lazzeroni

  2. Department of Veterans Affairs, Cooperative Studies Program Palo Alto Coordinating Center, Mountain View, California, USA

    Ying Lu & Ilana Belitskaya-Lévy

  3. Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California, USA

    Ying Lu

Authors
  1. Laura C Lazzeroni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ilana Belitskaya-Lévy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura C Lazzeroni.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Information

Supplementary Note (PDF 703 kb)

Supplementary Software

Statistical Intervals for P-value Evidence and Power (version 2.0). (XLS 183 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lazzeroni, L., Lu, Y. & Belitskaya-Lévy, I. Solutions for quantifying P-value uncertainty and replication power. Nat Methods 13, 107–108 (2016). https://doi.org/10.1038/nmeth.3741

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.3741

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics