Abstract
Hypothesis weighting improves the power of large-scale multiple testing. We describe independent hypothesis weighting (IHW), a method that assigns weights using covariates independent of the P-values under the null hypothesis but informative of each test's power or prior probability of the null hypothesis (http://www.bioconductor.org/packages/IHW). IHW increases power while controlling the false discovery rate and is a practical approach to discovering associations in genomics, high-throughput biology and other large data sets.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Benjamini, Y. & Hochberg, Y. J. R. Stat. Soc. Series B Stat. Methodol. 57, 289–300 (1995).
Benjamini, Y., Krieger, A.M. & Yekutieli, D. Biometrika 93, 491–507 (2006).
Storey, J.D., Taylor, J.E. & Siegmund, D. J. R. Stat. Soc. Series B Stat. Methodol. 66, 187–205 (2004).
Efron, B. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Cambridge University Press, 2010).
Strimmer, K. BMC Bioinformatics 9, 303 (2008).
Genovese, C.R., Roeder, K. & Wasserman, L. Biometrika 93, 509–524 (2006).
Roeder, K., Devlin, B. & Wasserman, L. Genet. Epidemiol. 31, 741–747 (2007).
Roquain, E. & van de Wiel, M. Electron. J. Stat. 3, 678–711 (2009).
Bourgon, R., Gentleman, R. & Huber, W. Proc. Natl. Acad. Sci. USA 107, 9546–9551 (2010).
Hu, J.X., Zhao, H. & Zhou, H.H. J. Am. Stat. Assoc. 105, 1215–1227 (2010).
Dobriban, E., Fortney, K., Kim, S.K. & Owen, A.B. Biometrika 102, 753–766 (2015).
Love, M.I., Huber, W. & Anders, S. Genome Biol. 15, 550 (2014).
Bottomly, D. et al. PLoS One 6, e17820 (2011).
Frazee, A.C., Langmead, B. & Leek, J.T. BMC Bioinformatics 12, 449 (2011).
Dephoure, N. & Gygi, S.P. Sci. Signal. 5, rs2 (2012).
Grubert, F. et al. Cell 162, 1051–1065 (2015).
Peña, E.A., Habiger, J.D. & Wu, W. Ann. Stat. 39, 556–583 (2011).
Sun, W. & Cai, T.T. J. Am. Stat. Assoc. 102, 901–912 (2007).
Stephens, M. Preprint at http://biorxiv.org/content/early/2016/01/29/038216.article-info (2016).
Cai, T.T. & Sun, W. J. Am. Stat. Assoc. 104, 1467–1481 (2009).
Ochoa, A., Storey, J.D., Llinás, M. & Singh, M. PLoS Comput. Biol. 11, e1004509 (2015).
Ploner, A., Calza, S., Gusnanto, A. & Pawitan, Y. Bioinformatics 22, 556–565 (2006).
Scott, J.G., Kelly, R.C., Smith, M.A., Zhou, P. & Kass, R.E. J. Am. Stat. Assoc. 110, 459–471 (2015).
Ferkingstad, E., Frigessi, A., Rue, H., Thorleifsson, G. & Kong, A. Ann. Appl. Stat. 2, 714–735 (2008).
Efron, B. & Zhang, N.R. Biometrika 98, 251–271 (2011).
Du, L. & Zhang, C. Ann. Stat. 42, 1262–1311 (2014).
Yoo, Y.J., Bull, S.B., Paterson, A.D., Waggott, D. & Sun, L. Genet. Epidemiol. 34, 107–118 (2010).
Benjamini, Y. & Yekutieli, D. Ann. Stat. 29, 1165–1188 (2001).
Acknowledgements
We thank B. Fischer, R. Gentleman, M.I. Love, M. Savitski, O. Stegle, and B. Velten for insightful discussions and comments on the manuscript; the COIN-OR project for the open-source SYMPHONY software and V. Kim for interfacing it to R through the lpsymphony package. We acknowledge support from the European Commission through the Horizon 2020 project SOUND.
Author information
Authors and Affiliations
Contributions
N.I. and W.H. developed the method and wrote the manuscript. N.I. implemented the method and performed the analyses. J.Z. analyzed the hQTL data set. B.K. contributed statistical concepts and ideas. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Weights learned by IHW as a function of the covariate.
a) Learned weights of IHW at α = 0.1 applied to the RNA-seq dataset by Bottomly et al. [13] analyzed with DESeq2. The mean of the normalized counts for each gene is used as the covariate. Genes with low counts get assigned low weight. Note that for each value of the covariate, five potentially different weights were learned because of the splitting scheme (E2). The trend is consistent. b) Weights learned by IHW at α = 0.1 for the quantitative proteomics dataset [15]. As covariate we use the total number of peptides quantified for each protein.
Supplementary Figure 2 Companion to Figure 2 e-h, continued benchmark of different methods.
a-b) Results from applying the t-test to Normal samples (n = 2 × 5, σ =1) with either the same mean (true nulls, π0 = 0.95) or means differing by the effect size indicated on the x-axis (π1 = 0.05). The pooled variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). These plots are analogous to Figure 2 g-h, but with further methods.
a) The y-axis shows the actual FDR. All methods lose type I error control for low values of the effect size, though they regain it at higher effect sizes.
b) Power analysis for the t-test scenario.
Panels c-f highlight size-investing. Data were simulated as in [7]: p-values were computed from the one-sided z-test, with Zi ∼ N(0,1) under the null and Zi ∼ N(ξi,1) under the alternative, with the covariate ξi ∼ U [1, ξmax]. The fraction of alternatives was π1 = 0.1. The x-axis represents the simulation parameter ξmax.
c,e) The y-axis shows the actual FDR. In this case, naive IHW, Clfdr and FDRreg are slightly anticonservative.
d,f) The logarithm of the power increase compared to BH is shown as a function of ξmax. Large values of ξmax necessitate a size-investing strategy. Methods which cannot apply such a strategy, including SBH, TST-GBH, LSL-GBH, FDRreg and Greedy Independent Filtering, have power even lower than BH or at best equal to it.
Supplementary Figure 3 Size-investing and tdr
a) Cumulative distribution function of the p-value for tests of two hypotheses H1 and H2. The CDF in the case of H1 is stochastically smaller (more “powerful”) than that in the case of H2. This difference is due to different π0s and alternative distributions for the two hypothesis tests.
b) Situation similar to panel a, but here the difference in CDFs is due to different π0 solely, while the alternative distributions are the same.
c) Graphs of p-value as a function of tdr (this plot is analogous to Supplementary Fig. 4d-f with flipped axes) for the 2 hypotheses in panel a. Note that for a high fixed tdr value, the corresponding p-value for hypothesis H1 is greater than for H2. As the tdr decreases, eventually these two functions cross and the inequality changes direction. This explains the size-investing strategy.
d) Analogous to panel c, for the hypotheses in panel b. Because here only π0 is different among H1 and H2, no size-investing can occur.
Supplementary Figure 4 Some further intuitions on tdr
-c) These three density curves correspond to panels b-d in Figure 3 based on simulated scenarios. d-f) Note that the procedure of calculating tdr(t) based on Equation (1) can be applied for each t. Thus the tdr is a deterministic function of the p-value and these panels show the graph of this function for scenarios a-c. g-i) Histograms of simulated p-values based on the distribution in a-c. In practice we observe only one of these p-values for each hypothesis. Therefore, this distribution can only be estimated under simplifying assumptions and by sharing information across the hypotheses. j-l) Histogram of the tdr values, i.e., of the transformed p-values (by Equation (1) in the main text). The tdr-values are also random variables. Notice that for high signal situations the tdr histogram gives a much better split than the corresponding p-value histogram. But again, since we do not know the distribution, in practice we cannot observe these random variables (i. e., they are not statistics, in contrast to the p-values), and we can only approximate them by using assumptions as above.
Supplementary Figure 5 Simulation results for IHW-Bonferroni compared to Bonferroni.
a) This panel benchmarks type I error control if all null hypotheses are true. Shown is the true FWER against the nominal significance level α; the dashed line indicates the identity function. Both Bonferroni and IHW- Bonferroni control the FWER. Panels b-c look at implications of different effect sizes. The two-sample t-test was applied to Normal samples (n = 2 × 5, σ = 1) with either the same mean (true nulls) or means differing by the effect size indicated on the x-axis (true alternatives). The fraction of true alternatives was 0.05. The pooled empirical variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). b) The y-axis shows the actual FWER, which is controlled for both methods. c) A power analysis. IHW-Bonferroni shows dramatic power improvements compared to Bonferroni. Panels d-e highlight the size-investing simulation: p-values were computed from the one-sided z-test, with Zi ∼ N(0,1) under the null and Zi ∼ N(ξi,1) under the alternative, with the covariate ξi ∼ U [1, ξmax]. The fraction of alternatives was 0.1. The x- axis represents the simulation parameter ξmax. d) The y-axis shows the logarithm of the actual FWER, as well as the nominal level log10(α) = −1. FWER is again controlled by both methods. e) The power analysis again shows a power increase for IHW-Bonferroni.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–5, Supplementary Table 1 and Supplementary Notes 1–7 (PDF 675 kb)
Supplementary Software
IHW and IHWpaper R packages for implementing the methodology described in this paper and for reproducing the results. (ZIP 15107 kb)
Rights and permissions
About this article
Cite this article
Ignatiadis, N., Klaus, B., Zaugg, J. et al. Data-driven hypothesis weighting increases detection power in genome-scale multiple testing. Nat Methods 13, 577–580 (2016). https://doi.org/10.1038/nmeth.3885
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nmeth.3885
This article is cited by
-
Resistance of HNSCC cell models to pan-FGFR inhibition depends on the EMT phenotype associating with clinical outcome
Molecular Cancer (2024)
-
Aberrant non-canonical NF-κB signalling reprograms the epigenome landscape to drive oncogenic transcriptomes in multiple myeloma
Nature Communications (2024)
-
INPP5A phosphatase is a synthetic lethal target in GNAQ and GNA11-mutant melanomas
Nature Cancer (2024)
-
Transcriptome of GH-producing pituitary neuroendocrine tumours and models are significantly affected by somatostatin analogues
Cancer Cell International (2023)
-
PIK3CA mutation in endometriotic epithelial cells promotes viperin-dependent inflammatory response to insulin
Reproductive Biology and Endocrinology (2023)