Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Data-driven hypothesis weighting increases detection power in genome-scale multiple testing

Abstract

Hypothesis weighting improves the power of large-scale multiple testing. We describe independent hypothesis weighting (IHW), a method that assigns weights using covariates independent of the P-values under the null hypothesis but informative of each test's power or prior probability of the null hypothesis (http://www.bioconductor.org/packages/IHW). IHW increases power while controlling the false discovery rate and is a practical approach to discovering associations in genomics, high-throughput biology and other large data sets.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Histograms stratified by the covariate as a diagnostic plot.
Figure 2: Performance evaluation.
Figure 3: Local true discovery rate and informative covariates.

References

  1. 1

    Benjamini, Y. & Hochberg, Y. J. R. Stat. Soc. Series B Stat. Methodol. 57, 289–300 (1995).

    Google Scholar 

  2. 2

    Benjamini, Y., Krieger, A.M. & Yekutieli, D. Biometrika 93, 491–507 (2006).

    Article  Google Scholar 

  3. 3

    Storey, J.D., Taylor, J.E. & Siegmund, D. J. R. Stat. Soc. Series B Stat. Methodol. 66, 187–205 (2004).

    Article  Google Scholar 

  4. 4

    Efron, B. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Cambridge University Press, 2010).

  5. 5

    Strimmer, K. BMC Bioinformatics 9, 303 (2008).

    Article  Google Scholar 

  6. 6

    Genovese, C.R., Roeder, K. & Wasserman, L. Biometrika 93, 509–524 (2006).

    Article  Google Scholar 

  7. 7

    Roeder, K., Devlin, B. & Wasserman, L. Genet. Epidemiol. 31, 741–747 (2007).

    Article  Google Scholar 

  8. 8

    Roquain, E. & van de Wiel, M. Electron. J. Stat. 3, 678–711 (2009).

    Article  Google Scholar 

  9. 9

    Bourgon, R., Gentleman, R. & Huber, W. Proc. Natl. Acad. Sci. USA 107, 9546–9551 (2010).

    CAS  Article  Google Scholar 

  10. 10

    Hu, J.X., Zhao, H. & Zhou, H.H. J. Am. Stat. Assoc. 105, 1215–1227 (2010).

    CAS  Article  Google Scholar 

  11. 11

    Dobriban, E., Fortney, K., Kim, S.K. & Owen, A.B. Biometrika 102, 753–766 (2015).

    Article  Google Scholar 

  12. 12

    Love, M.I., Huber, W. & Anders, S. Genome Biol. 15, 550 (2014).

    Article  Google Scholar 

  13. 13

    Bottomly, D. et al. PLoS One 6, e17820 (2011).

    CAS  Article  Google Scholar 

  14. 14

    Frazee, A.C., Langmead, B. & Leek, J.T. BMC Bioinformatics 12, 449 (2011).

    Article  Google Scholar 

  15. 15

    Dephoure, N. & Gygi, S.P. Sci. Signal. 5, rs2 (2012).

    Article  Google Scholar 

  16. 16

    Grubert, F. et al. Cell 162, 1051–1065 (2015).

    CAS  Article  Google Scholar 

  17. 17

    Peña, E.A., Habiger, J.D. & Wu, W. Ann. Stat. 39, 556–583 (2011).

    Article  Google Scholar 

  18. 18

    Sun, W. & Cai, T.T. J. Am. Stat. Assoc. 102, 901–912 (2007).

    CAS  Article  Google Scholar 

  19. 19

    Stephens, M. Preprint at http://biorxiv.org/content/early/2016/01/29/038216.article-info (2016).

  20. 20

    Cai, T.T. & Sun, W. J. Am. Stat. Assoc. 104, 1467–1481 (2009).

    Article  Google Scholar 

  21. 21

    Ochoa, A., Storey, J.D., Llinás, M. & Singh, M. PLoS Comput. Biol. 11, e1004509 (2015).

    Article  Google Scholar 

  22. 22

    Ploner, A., Calza, S., Gusnanto, A. & Pawitan, Y. Bioinformatics 22, 556–565 (2006).

    CAS  Article  Google Scholar 

  23. 23

    Scott, J.G., Kelly, R.C., Smith, M.A., Zhou, P. & Kass, R.E. J. Am. Stat. Assoc. 110, 459–471 (2015).

    CAS  Article  Google Scholar 

  24. 24

    Ferkingstad, E., Frigessi, A., Rue, H., Thorleifsson, G. & Kong, A. Ann. Appl. Stat. 2, 714–735 (2008).

    Article  Google Scholar 

  25. 25

    Efron, B. & Zhang, N.R. Biometrika 98, 251–271 (2011).

    Article  Google Scholar 

  26. 26

    Du, L. & Zhang, C. Ann. Stat. 42, 1262–1311 (2014).

    Article  Google Scholar 

  27. 27

    Yoo, Y.J., Bull, S.B., Paterson, A.D., Waggott, D. & Sun, L. Genet. Epidemiol. 34, 107–118 (2010).

    Article  Google Scholar 

  28. 28

    Benjamini, Y. & Yekutieli, D. Ann. Stat. 29, 1165–1188 (2001).

    Article  Google Scholar 

Download references

Acknowledgements

We thank B. Fischer, R. Gentleman, M.I. Love, M. Savitski, O. Stegle, and B. Velten for insightful discussions and comments on the manuscript; the COIN-OR project for the open-source SYMPHONY software and V. Kim for interfacing it to R through the lpsymphony package. We acknowledge support from the European Commission through the Horizon 2020 project SOUND.

Author information

Affiliations

Authors

Contributions

N.I. and W.H. developed the method and wrote the manuscript. N.I. implemented the method and performed the analyses. J.Z. analyzed the hQTL data set. B.K. contributed statistical concepts and ideas. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Wolfgang Huber.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Weights learned by IHW as a function of the covariate.

a) Learned weights of IHW at α = 0.1 applied to the RNA-seq dataset by Bottomly et al. [13] analyzed with DESeq2. The mean of the normalized counts for each gene is used as the covariate. Genes with low counts get assigned low weight. Note that for each value of the covariate, five potentially different weights were learned because of the splitting scheme (E2). The trend is consistent. b) Weights learned by IHW at α = 0.1 for the quantitative proteomics dataset [15]. As covariate we use the total number of peptides quantified for each protein.

Supplementary Figure 2 Companion to Figure 2 e-h, continued benchmark of different methods.

a-b) Results from applying the t-test to Normal samples (n = 2 × 5, σ =1) with either the same mean (true nulls, π0 = 0.95) or means differing by the effect size indicated on the x-axis (π1 = 0.05). The pooled variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). These plots are analogous to Figure 2 g-h, but with further methods.

a) The y-axis shows the actual FDR. All methods lose type I error control for low values of the effect size, though they regain it at higher effect sizes.

b) Power analysis for the t-test scenario.

Panels c-f highlight size-investing. Data were simulated as in [7]: p-values were computed from the one-sided z-test, with Zi N(0,1) under the null and Zi Ni,1) under the alternative, with the covariate ξi U [1, ξmax]. The fraction of alternatives was π1 = 0.1. The x-axis represents the simulation parameter ξmax.

c,e) The y-axis shows the actual FDR. In this case, naive IHW, Clfdr and FDRreg are slightly anticonservative.

d,f) The logarithm of the power increase compared to BH is shown as a function of ξmax. Large values of ξmax necessitate a size-investing strategy. Methods which cannot apply such a strategy, including SBH, TST-GBH, LSL-GBH, FDRreg and Greedy Independent Filtering, have power even lower than BH or at best equal to it.

Supplementary Figure 3 Size-investing and tdr

a) Cumulative distribution function of the p-value for tests of two hypotheses H1 and H2. The CDF in the case of H1 is stochastically smaller (more “powerful”) than that in the case of H2. This difference is due to different π0s and alternative distributions for the two hypothesis tests.

b) Situation similar to panel a, but here the difference in CDFs is due to different π0 solely, while the alternative distributions are the same.

c) Graphs of p-value as a function of tdr (this plot is analogous to Supplementary Fig. 4d-f with flipped axes) for the 2 hypotheses in panel a. Note that for a high fixed tdr value, the corresponding p-value for hypothesis H1 is greater than for H2. As the tdr decreases, eventually these two functions cross and the inequality changes direction. This explains the size-investing strategy.

d) Analogous to panel c, for the hypotheses in panel b. Because here only π0 is different among H1 and H2, no size-investing can occur.

Supplementary Figure 4 Some further intuitions on tdr

-c) These three density curves correspond to panels b-d in Figure 3 based on simulated scenarios. d-f) Note that the procedure of calculating tdr(t) based on Equation (1) can be applied for each t. Thus the tdr is a deterministic function of the p-value and these panels show the graph of this function for scenarios a-c. g-i) Histograms of simulated p-values based on the distribution in a-c. In practice we observe only one of these p-values for each hypothesis. Therefore, this distribution can only be estimated under simplifying assumptions and by sharing information across the hypotheses. j-l) Histogram of the tdr values, i.e., of the transformed p-values (by Equation (1) in the main text). The tdr-values are also random variables. Notice that for high signal situations the tdr histogram gives a much better split than the corresponding p-value histogram. But again, since we do not know the distribution, in practice we cannot observe these random variables (i. e., they are not statistics, in contrast to the p-values), and we can only approximate them by using assumptions as above.

Supplementary Figure 5 Simulation results for IHW-Bonferroni compared to Bonferroni.

a) This panel benchmarks type I error control if all null hypotheses are true. Shown is the true FWER against the nominal significance level α; the dashed line indicates the identity function. Both Bonferroni and IHW- Bonferroni control the FWER. Panels b-c look at implications of different effect sizes. The two-sample t-test was applied to Normal samples (n = 2 × 5, σ = 1) with either the same mean (true nulls) or means differing by the effect size indicated on the x-axis (true alternatives). The fraction of true alternatives was 0.05. The pooled empirical variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). b) The y-axis shows the actual FWER, which is controlled for both methods. c) A power analysis. IHW-Bonferroni shows dramatic power improvements compared to Bonferroni. Panels d-e highlight the size-investing simulation: p-values were computed from the one-sided z-test, with Zi N(0,1) under the null and Zi Ni,1) under the alternative, with the covariate ξi U [1, ξmax]. The fraction of alternatives was 0.1. The x- axis represents the simulation parameter ξmax. d) The y-axis shows the logarithm of the actual FWER, as well as the nominal level log10(α) = −1. FWER is again controlled by both methods. e) The power analysis again shows a power increase for IHW-Bonferroni.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–5, Supplementary Table 1 and Supplementary Notes 1–7 (PDF 675 kb)

Supplementary Software

IHW and IHWpaper R packages for implementing the methodology described in this paper and for reproducing the results. (ZIP 15107 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ignatiadis, N., Klaus, B., Zaugg, J. et al. Data-driven hypothesis weighting increases detection power in genome-scale multiple testing. Nat Methods 13, 577–580 (2016). https://doi.org/10.1038/nmeth.3885

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing