Points of significance: P values and the search for significance

Journal name:
Nature Methods
Volume:
14,
Pages:
3–4
Year published:
DOI:
doi:10.1038/nmeth.4120
Published online

Little P value
What are you trying to say
Of significance?

—Steve Ziliak

At a glance

Figures

  1. P values are random variables.
    Figure 1: P values are random variables.

    In assessing statistical significance, we rely on their distribution when the null hypothesis, H0, is true. (a) Simulated P values from 1,000 statistical tests when H0 is true. The distribution is uniform and, on average, 5% of P < 0.05 (blue). (b) The distribution of the minimum P value across 1,000 simulations of 10 tests when H0 is true. Now, on average, 40% of P < 0.05 (blue). Note the difference in y-axis scale compared to a.

  2. Merely reporting 95% confidence intervals does not address selection bias.
    Figure 2: Merely reporting 95% confidence intervals does not address selection bias.

    (a) 95% confidence intervals for 100 one-sample t-tests with samples of size n = 100, mean zero and s.d. = 1. Intervals are vertically sorted in increasing order of statistical significance. (b) 100 instances of the 95% confidence interval corresponding to the most significant result from a set of 10 one-sample t-tests of the kind performed in a.

  3. Variable selection during model building greatly inflates statistical significance.
    Figure 3: Variable selection during model building greatly inflates statistical significance.

    (a) Number of times that 0 (the correct number) to 6 of predictors were selected as explanatory from 1,000 simulations. (b) Distribution of R2 (top) and P values of the F-test (bottom) for the 828 cases from a in which the incorrect number (k > 0) of predictors was selected.

References

  1. Krzywinski, M. & Altman, N. Nat. Methods 11, 355356 (2014).
  2. Altman, N. & Krzywinski, M. Nat. Methods 12, 9991000 (2015).
  3. Lever, J., Krzywinski, M. & Altman, N. Nat. Methods 13, 703704 (2016).
  4. Wasserstein, R.L. & Lazar, N.A. Am. Stat. 70, 129133 (2016).

Download references

Author information

Affiliations

  1. Naomi Altman is a Professor of Statistics at The Pennsylvania State University.

  2. Martin Krzywinski is a staff scientist at Canada's Michael Smith Genome Sciences Centre.

Competing financial interests

The authors declare no competing financial interests.

Author details

Additional data