Points of significance: Comparing samples—part I

Journal name:
Nature Methods
Volume:
11,
Pages:
215–216
Year published:
DOI:
doi:10.1038/nmeth.2858
Published online

Robustly comparing pairs of independent or related samples requires different approaches to the t-test.

At a glance

Figures

  1. The uncertainty in a sum or difference of random variables is the sum of the variables' individual uncertainties, as measured by the variance.
    Figure 1: The uncertainty in a sum or difference of random variables is the sum of the variables' individual uncertainties, as measured by the variance.

    Numerical values reflect sample estimates from Figure 2. Horizontal error bars show s.d., which is √Var. (a) Comparing a sample to a reference value involves only one measure of uncertainty: the variance of the sample's underlying population, Var(X). The variance of the sample mean is reduced in proportion to the sample size as Var(X)/n, which is also the uncertainty in the estimate of the difference between sample and reference. (b) When the reference is replaced by sample Y of size m, the variance of Y contributes to the uncertainty in the difference of means.

  2. In the two-sample test, both samples contribute to the uncertainty in the difference of means.
    Figure 2: In the two-sample test, both samples contribute to the uncertainty in the difference of means.

    (a) The difference between a sample and a reference value (μ = 10) can be assessed with a one-sample t-test. (b) When the reference value is itself a sample , the two-sample version of the test is used, in which the t-statistic is based on a combined spread of X and Y, which is estimated using the pooled variance, sp2.

  3. The paired t-test is appropriate for matched-sample experiments.
    Figure 3: The paired t-test is appropriate for matched-sample experiments.

    (a) When samples are independent, within-sample variability makes differences between sample means difficult to discern, and we cannot say that X and Y are different at α = 0.05. (b) If X and Y represent paired measurements, such as before and after treatment, differences between value pairs can be tested, thereby removing within-sample variability from consideration. (c) In a paired test, differences between values are used to construct a new sample, to which the one-sample test is applied .

References

  1. Krzywinski, M. & Altman, N. Nat. Methods 10, 10411042 (2013).
  2. Krzywinski, M. & Altman, N. Nat. Methods 10, 11391140 (2013).
  3. Ramsey, P.H. J. Educ. Stat. 5, 337349 (1980).
  4. Wiederman, W. & von Eye, A. Psychol. Test Assess. Model. 55, 3961 (2013).

Download references

Author information

Affiliations

  1. Martin Krzywinski is a staff scientist at Canada's Michael Smith Genome Sciences Centre.

  2. Naomi Altman is a Professor of Statistics at The Pennsylvania State University.

Competing financial interests

The authors declare no competing financial interests.

Author details

Supplementary information

Zip files

  1. Supplementary Table 1 (806 KB)

    Table can be used to explore two-sample comparisons.

Additional data