Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

Data-driven hypothesis weighting increases detection power in genome-scale multiple testing

Nature Methods volume 13, pages 577–580 (2016)Cite this article

Subjects

Abstract

Hypothesis weighting improves the power of large-scale multiple testing. We describe independent hypothesis weighting (IHW), a method that assigns weights using covariates independent of the P-values under the null hypothesis but informative of each test's power or prior probability of the null hypothesis (http://www.bioconductor.org/packages/IHW). IHW increases power while controlling the false discovery rate and is a practical approach to discovering associations in genomics, high-throughput biology and other large data sets.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Histograms stratified by the covariate as a diagnostic plot.
Figure 2: Performance evaluation.
Figure 3: Local true discovery rate and informative covariates.

Similar content being viewed by others

References

  1. Benjamini, Y. & Hochberg, Y. J. R. Stat. Soc. Series B Stat. Methodol. 57, 289–300 (1995).

    Google Scholar 

  2. Benjamini, Y., Krieger, A.M. & Yekutieli, D. Biometrika 93, 491–507 (2006).

    Article  Google Scholar 

  3. Storey, J.D., Taylor, J.E. & Siegmund, D. J. R. Stat. Soc. Series B Stat. Methodol. 66, 187–205 (2004).

    Article  Google Scholar 

  4. Efron, B. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Cambridge University Press, 2010).

  5. Strimmer, K. BMC Bioinformatics 9, 303 (2008).

    Article  Google Scholar 

  6. Genovese, C.R., Roeder, K. & Wasserman, L. Biometrika 93, 509–524 (2006).

    Article  Google Scholar 

  7. Roeder, K., Devlin, B. & Wasserman, L. Genet. Epidemiol. 31, 741–747 (2007).

    Article  Google Scholar 

  8. Roquain, E. & van de Wiel, M. Electron. J. Stat. 3, 678–711 (2009).

    Article  Google Scholar 

  9. Bourgon, R., Gentleman, R. & Huber, W. Proc. Natl. Acad. Sci. USA 107, 9546–9551 (2010).

    Article  CAS  Google Scholar 

  10. Hu, J.X., Zhao, H. & Zhou, H.H. J. Am. Stat. Assoc. 105, 1215–1227 (2010).

    Article  CAS  Google Scholar 

  11. Dobriban, E., Fortney, K., Kim, S.K. & Owen, A.B. Biometrika 102, 753–766 (2015).

    Article  Google Scholar 

  12. Love, M.I., Huber, W. & Anders, S. Genome Biol. 15, 550 (2014).

    Article  Google Scholar 

  13. Bottomly, D. et al. PLoS One 6, e17820 (2011).

    Article  CAS  Google Scholar 

  14. Frazee, A.C., Langmead, B. & Leek, J.T. BMC Bioinformatics 12, 449 (2011).

    Article  Google Scholar 

  15. Dephoure, N. & Gygi, S.P. Sci. Signal. 5, rs2 (2012).

    Article  Google Scholar 

  16. Grubert, F. et al. Cell 162, 1051–1065 (2015).

    Article  CAS  Google Scholar 

  17. Peña, E.A., Habiger, J.D. & Wu, W. Ann. Stat. 39, 556–583 (2011).

    Article  Google Scholar 

  18. Sun, W. & Cai, T.T. J. Am. Stat. Assoc. 102, 901–912 (2007).

    Article  CAS  Google Scholar 

  19. Stephens, M. Preprint at http://biorxiv.org/content/early/2016/01/29/038216.article-info (2016).

  20. Cai, T.T. & Sun, W. J. Am. Stat. Assoc. 104, 1467–1481 (2009).

    Article  Google Scholar 

  21. Ochoa, A., Storey, J.D., Llinás, M. & Singh, M. PLoS Comput. Biol. 11, e1004509 (2015).

    Article  Google Scholar 

  22. Ploner, A., Calza, S., Gusnanto, A. & Pawitan, Y. Bioinformatics 22, 556–565 (2006).

    Article  CAS  Google Scholar 

  23. Scott, J.G., Kelly, R.C., Smith, M.A., Zhou, P. & Kass, R.E. J. Am. Stat. Assoc. 110, 459–471 (2015).

    Article  CAS  Google Scholar 

  24. Ferkingstad, E., Frigessi, A., Rue, H., Thorleifsson, G. & Kong, A. Ann. Appl. Stat. 2, 714–735 (2008).

    Article  Google Scholar 

  25. Efron, B. & Zhang, N.R. Biometrika 98, 251–271 (2011).

    Article  Google Scholar 

  26. Du, L. & Zhang, C. Ann. Stat. 42, 1262–1311 (2014).

    Article  Google Scholar 

  27. Yoo, Y.J., Bull, S.B., Paterson, A.D., Waggott, D. & Sun, L. Genet. Epidemiol. 34, 107–118 (2010).

    Article  Google Scholar 

  28. Benjamini, Y. & Yekutieli, D. Ann. Stat. 29, 1165–1188 (2001).

    Article  Google Scholar 

Download references

Acknowledgements

We thank B. Fischer, R. Gentleman, M.I. Love, M. Savitski, O. Stegle, and B. Velten for insightful discussions and comments on the manuscript; the COIN-OR project for the open-source SYMPHONY software and V. Kim for interfacing it to R through the lpsymphony package. We acknowledge support from the European Commission through the Horizon 2020 project SOUND.

Author information

Authors and Affiliations

  1. European Molecular Biology Laboratory, Heidelberg, Germany

    Nikolaos Ignatiadis, Bernd Klaus, Judith B Zaugg & Wolfgang Huber

Authors
  1. Nikolaos Ignatiadis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bernd Klaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Judith B Zaugg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wolfgang Huber
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

N.I. and W.H. developed the method and wrote the manuscript. N.I. implemented the method and performed the analyses. J.Z. analyzed the hQTL data set. B.K. contributed statistical concepts and ideas. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Wolfgang Huber.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Weights learned by IHW as a function of the covariate.

a) Learned weights of IHW at α = 0.1 applied to the RNA-seq dataset by Bottomly et al. [13] analyzed with DESeq2. The mean of the normalized counts for each gene is used as the covariate. Genes with low counts get assigned low weight. Note that for each value of the covariate, five potentially different weights were learned because of the splitting scheme (E2). The trend is consistent. b) Weights learned by IHW at α = 0.1 for the quantitative proteomics dataset [15]. As covariate we use the total number of peptides quantified for each protein.

Supplementary Figure 2 Companion to Figure 2 e-h, continued benchmark of different methods.

a-b) Results from applying the t-test to Normal samples (n = 2 × 5, σ =1) with either the same mean (true nulls, π0 = 0.95) or means differing by the effect size indicated on the x-axis (π1 = 0.05). The pooled variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). These plots are analogous to Figure 2 g-h, but with further methods.

a) The y-axis shows the actual FDR. All methods lose type I error control for low values of the effect size, though they regain it at higher effect sizes.

b) Power analysis for the t-test scenario.

Panels c-f highlight size-investing. Data were simulated as in [7]: p-values were computed from the one-sided z-test, with Zi ∼ N(0,1) under the null and Zi ∼ N(ξi,1) under the alternative, with the covariate ξi ∼ U [1, ξmax]. The fraction of alternatives was π1 = 0.1. The x-axis represents the simulation parameter ξmax.

c,e) The y-axis shows the actual FDR. In this case, naive IHW, Clfdr and FDRreg are slightly anticonservative.

d,f) The logarithm of the power increase compared to BH is shown as a function of ξmax. Large values of ξmax necessitate a size-investing strategy. Methods which cannot apply such a strategy, including SBH, TST-GBH, LSL-GBH, FDRreg and Greedy Independent Filtering, have power even lower than BH or at best equal to it.

Supplementary Figure 3 Size-investing and tdr

a) Cumulative distribution function of the p-value for tests of two hypotheses H1 and H2. The CDF in the case of H1 is stochastically smaller (more “powerful”) than that in the case of H2. This difference is due to different π0s and alternative distributions for the two hypothesis tests.

b) Situation similar to panel a, but here the difference in CDFs is due to different π0 solely, while the alternative distributions are the same.

c) Graphs of p-value as a function of tdr (this plot is analogous to Supplementary Fig. 4d-f with flipped axes) for the 2 hypotheses in panel a. Note that for a high fixed tdr value, the corresponding p-value for hypothesis H1 is greater than for H2. As the tdr decreases, eventually these two functions cross and the inequality changes direction. This explains the size-investing strategy.

d) Analogous to panel c, for the hypotheses in panel b. Because here only π0 is different among H1 and H2, no size-investing can occur.

Supplementary Figure 4 Some further intuitions on tdr

-c) These three density curves correspond to panels b-d in Figure 3 based on simulated scenarios. d-f) Note that the procedure of calculating tdr(t) based on Equation (1) can be applied for each t. Thus the tdr is a deterministic function of the p-value and these panels show the graph of this function for scenarios a-c. g-i) Histograms of simulated p-values based on the distribution in a-c. In practice we observe only one of these p-values for each hypothesis. Therefore, this distribution can only be estimated under simplifying assumptions and by sharing information across the hypotheses. j-l) Histogram of the tdr values, i.e., of the transformed p-values (by Equation (1) in the main text). The tdr-values are also random variables. Notice that for high signal situations the tdr histogram gives a much better split than the corresponding p-value histogram. But again, since we do not know the distribution, in practice we cannot observe these random variables (i. e., they are not statistics, in contrast to the p-values), and we can only approximate them by using assumptions as above.

Supplementary Figure 5 Simulation results for IHW-Bonferroni compared to Bonferroni.

a) This panel benchmarks type I error control if all null hypotheses are true. Shown is the true FWER against the nominal significance level α; the dashed line indicates the identity function. Both Bonferroni and IHW- Bonferroni control the FWER. Panels b-c look at implications of different effect sizes. The two-sample t-test was applied to Normal samples (n = 2 × 5, σ = 1) with either the same mean (true nulls) or means differing by the effect size indicated on the x-axis (true alternatives). The fraction of true alternatives was 0.05. The pooled empirical variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). b) The y-axis shows the actual FWER, which is controlled for both methods. c) A power analysis. IHW-Bonferroni shows dramatic power improvements compared to Bonferroni. Panels d-e highlight the size-investing simulation: p-values were computed from the one-sided z-test, with Zi ∼ N(0,1) under the null and Zi ∼ N(ξi,1) under the alternative, with the covariate ξi ∼ U [1, ξmax]. The fraction of alternatives was 0.1. The x- axis represents the simulation parameter ξmax. d) The y-axis shows the logarithm of the actual FWER, as well as the nominal level log10(α) = −1. FWER is again controlled by both methods. e) The power analysis again shows a power increase for IHW-Bonferroni.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–5, Supplementary Table 1 and Supplementary Notes 1–7 (PDF 675 kb)

Supplementary Software

IHW and IHWpaper R packages for implementing the methodology described in this paper and for reproducing the results. (ZIP 15107 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ignatiadis, N., Klaus, B., Zaugg, J. et al. Data-driven hypothesis weighting increases detection power in genome-scale multiple testing. Nat Methods 13, 577–580 (2016). https://doi.org/10.1038/nmeth.3885

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.3885

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics