Review Article | Published:

Statistical power and significance testing in large-scale genetic studies

Nature Reviews Genetics volume 15, pages 335346 (2014) | Download Citation

Abstract

Significance testing was developed as an objective method for summarizing statistical evidence for a hypothesis. It has been widely adopted in genetic studies, including genome-wide association studies and, more recently, exome sequencing studies. However, significance testing in both genome-wide and exome-wide studies must adopt stringent significance thresholds to allow multiple testing, and it is useful only when studies have adequate statistical power, which depends on the characteristics of the phenotype and the putative genetic variant, as well as the study design. Here, we review the principles and applications of significance testing and power calculation, including recently proposed gene-based tests for rare variants.

Key points

  • Significance testing, with appropriate multiple testing correction, is currently the most convenient method for summarizing the evidence for association between a disease and a genetic variant.

  • Inadequate statistical power increases not only the probability of missing genuine associations but also the probability that significant associations represent false-positive findings.

  • Statistical power declines rapidly with decreasing allele frequency and effect size, but it can be enhanced by increasing sample size and by selecting appropriate subjects (for example, family history positive cases and 'super normal' controls).

  • Exome sequencing studies can often identify the mutation responsible for a Mendelian disease by filtering out common variants, synonymous variants or variants that do not co-segregate with disease, and then assigning priority to the remaining variants using bioinformatic tools.

  • Adequate statistical power for rare-variant association analyses in complex diseases requires the aggregation of the effects of multiple rare variants within a defined portion of the genome (for example, a set of related genes).

  • Various computational tools are available for calculating the statistical power of genetic studies.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Statistical Methods for Research Workers (Oliver and Boyd, 1925).

  2. 2.

    & On the problem of the most efficient tests of statistical hypotheses. Phil. Trans. R. Soc. Lond. A 231, 289–337 (1933).

  3. 3.

    Null hypothesis significance testing: a review of an old and continuing controversy. Psychol. Methods 5, 241–301 (2000).

  4. 4.

    A tutorial on statistical methods for population association studies. Nature Rev. Genet. 7, 781–791 (2006).

  5. 5.

    & Bayesian statistical methods for genetic association studies. Nature Rev. Genet. 10, 681–690 (2009). This is a highly readable account of Bayesian approaches for the analysis of genetic association studies.

  6. 6.

    , , & A comprehensive review of genetic association studies. Genet. Med. 4, 45–61 (2002).

  7. 7.

    Genetic associations: false or true? Trends Mol. Med. 9, 135–138 (2003).

  8. 8.

    et al. Genome-wide association studies for complex traits: consensus, uncertainty and challenges. Nature Rev. Genet. 9, 356–369 (2008).

  9. 9.

    The International HapMap Consortium. A haplotype map of the human genome. Nature 437, 1299–1320 (2005).

  10. 10.

    & Genome-wide association studies for common diseases and complex traits. Nature Rev. Genet. 6, 95–108 (2005).

  11. 11.

    , , & Genome-wide association studies: theoretical and practical concerns. Nature Rev. Genet. 6, 109–118 (2005).

  12. 12.

    et al. PLINK: a tool set for whole-genome association and population-based linkage analyses. Am. J. Hum. Genet. 81, 559–575 (2007).

  13. 13.

    et al. Principal components analysis corrects for stratification in genome-wide association studies. Nature Genet. 38, 904–909 (2006).

  14. 14.

    , , & Estimation of the multiple testing burden for genomewide association studies of nearly all common variants. Genet. Epidemiol. 32, 381–385 (2008).

  15. 15.

    & Estimation of significance thresholds for genomewide association scans. Genet. Epidemiol. 32, 227–234 (2008).

  16. 16.

    , , , & Genome-wide significance for dense SNP and resequencing data. Genet. Epidemiol. 32, 179–185 (2008).

  17. 17.

    et al. The metabochip, a custom genotyping array for genetic studies of metabolic, cardiovascular, and anthropometric traits. PLoS Genet. 8, e1002793 (2012).

  18. 18.

    et al. Immunochip analyses identify a novel risk locus for primary biliary cirrhosis at 13q14, multiple independent associations at four established risk loci and epistasis between 1p31 and 7q32 risk variants. Hum. Mol. Genet. 21, 5209–5221 (2012).

  19. 19.

    , , & Establishing an adjusted p-value threshold to control the family-wide type 1 error in genome wide association studies. BMC Genomics 9, 516 (2008).

  20. 20.

    A simple correction for multiple testing for single-nucleotide polymorphisms in linkage disequilibrium with each other. Am. J. Hum. Genet. 74, 765–769 (2004).

  21. 21.

    A new measure of the effective number of tests, a practical tool for comparing families of non-independent significance tests. Genet. Epidemiol. 33, 559–568 (2009).

  22. 22.

    & Adjusting multiple testing in multilocus analyses using the eigenvalues of a correlation matrix. Heredity 95, 221–227 (2005).

  23. 23.

    & On multiple-testing correction in genome-wide association studies. Genet. Epidemiol. 32, 567–573 (2008).

  24. 24.

    , , & Evaluating the effective number of independent tests and significant p-value thresholds in commercial genotyping arrays and public imputation reference datasets. Hum. Genet. 131, 747–756 (2012).

  25. 25.

    , & A note on the calculation of empirical P values from Monte Carlo procedures. Am. J. Hum. Genet. 71, 439–441 (2002).

  26. 26.

    , & A note on calculation of empirical P values from Monte Carlo procedure. Am. J. Hum. Genet. 72, 498–499 (2003).

  27. 27.

    & Efficient computation of significance levels for multiple associations in large studies of correlated data, including genomewide association studies. Am. J. Hum. Genet. 75, 424–435 (2004).

  28. 28.

    & Müller-Myhsok, B. Rapid simulation of P values for product methods and multiple-testing adjustment in association studies. Am. J. Hum. Genet. 76, 399–408 (2005).

  29. 29.

    , , , & Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J. Natl Cancer Inst. 96, 434–442 (2004).

  30. 30.

    , & Genome-Wide Significance Project. What should the genome-wide significance threshold be? Empirical replication of borderline genetic associations. Int. J. Epidemiol. 41, 273–286 (2011).

  31. 31.

    & Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B Stat. Methodol. 57, 289–300 (1995).

  32. 32.

    Bayes factors for genome-wide association studies: comparison with P-values. Genet. Epidemiol. 33, 79–86 (2009).

  33. 33.

    , , & Five years of GWAS discovery. Am. J. Hum. Genet. 90, 7–24 (2012). This paper summarizes and interprets GWAS findings on common diseases and quantitative traits.

  34. 34.

    , & How many genetic variants remain to be discovered? PLoS ONE 4, e7969 (2009).

  35. 35.

    , & Genetic Power Calculator: design of linkage and association genetic mapping studies of complex traits. Bioinformatics 19, 149–150 (2003).

  36. 36.

    Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008).

  37. 37.

    & Bias-reduced estimators and confidence intervals for odds ratios in genome-wide association studies. Biostatistics 9, 621–634 (2008).

  38. 38.

    , & Estimating odds ratios in genome scans: an approximate conditional likelihood approach. Am. J. Hum. Genet. 82, 1064–1074 (2008).

  39. 39.

    & Overcoming the winner's curse: estimating penetrance parameters from case–control data. Am. J. Hum. Genet. 80, 605–615 (2007).

  40. 40.

    , , & Power of linkage versus association analysis of quantitative traits, by use of variance-components models, for sibship data. Am. J. Hum. Genet. 66, 1616–1630 (2000).

  41. 41.

    , & Including known covariates can reduce power to detect genetic effects in case–control studies. Nature Genet. 44, 848–851 (2012).

  42. 42.

    , , & Efficient approximation of P-value of the maximum of correlated tests, with applications to genome-wide association studies. Ann. Hum. Genet. 72, 397–406 (2008).

  43. 43.

    et al. Maximizing association statistics over genetic models. Genet. Epidemiol. 32, 246–254 (2008).

  44. 44.

    & Robust association tests under different genetic models, allowing for binary or quantitative traits and covariates. Behav. Genet. 41, 768–775 (2011).

  45. 45.

    et al. Exome sequencing as a tool for Mendelian disease gene discovery. Nature Rev. Genet. 12, 745–755 (2011).

  46. 46.

    et al. Exome sequencing and the genetic basis of complex traits. Nature Genet. 44, 623–630 (2012).

  47. 47.

    , & Most rare missense alleles are deleterious in humans: implications for complex disease and association studies. Am. J. Hum. Genet. 80, 727–739 (2007).

  48. 48.

    et al. An abundance of rare functional variants in 202 drug target genes sequenced in 14,002 people. Science 337, 100–104 (2012).

  49. 49.

    et al. Analysis of 6,515 exomes reveals the recent origin of most human protein-coding variants. Nature 493, 216–220 (2013).

  50. 50.

    & Discovery of rare variants via sequencing: implications for the design of complex trait association studies. PLoS Genet. 5, e1000481 (2009).

  51. 51.

    & Replication strategies for rare variant complex trait association studies via next-generation sequencing. Am. J. Hum. Genet. 87, 790–801 (2010).

  52. 52.

    , , , & A comprehensive framework for prioritizing variants in exome sequencing studies of Mendelian diseases. Nucleic Acids Res. 40, e53 (2012).

  53. 53.

    et al. Exome sequencing identifies MLL2 mutations as a cause of Kabuki syndrome. Nature Genet. 42, 790–793 (2010).

  54. 54.

    & Statistical guidance for experimental design and data analysis of mutation detection in rare monogenic mendelian diseases by exome sequencing. PLoS ONE 7, e31358 (2012).

  55. 55.

    , , & Design considerations for massively parallel sequencing studies of complex human disease. PLoS ONE 6, e23221 (2011).

  56. 56.

    & Methods for detecting associations with rare variants for common diseases: application to analysis of sequence data. Am. J. Hum. Genet. 83, 311–321 (2008). This is one of the first association tests for rare variants.

  57. 57.

    & A groupwise association test for rare mutations using a weighted sum statistic. PLoS Genet. 5, e1000384 (2009).

  58. 58.

    et al. Pooled association tests for rare variants in exon-resequencing studies. Am. J. Hum. Genet. 86, 982 (2010).

  59. 59.

    & A general framework for detecting disease associations with rare variants in sequencing studies. Am. J. Hum. Genet. 89, 354–367 (2011).

  60. 60.

    , , & Statistical analysis strategies for association studies involving rare variants. Nature Rev. Genet. 11, 773–785 (2010).

  61. 61.

    , & Computational and statistical approaches to analyzing variants identified by exome sequencing. Genome Biol. 12, 227 (2011).

  62. 62.

    & Comparison of statistical tests for disease association with rare variants. Genet. Epidemiol. 35, 606–619 (2011).

  63. 63.

    , , , & The empirical power of rare variant association methods: results from Sanger sequencing in 1,998 individuals. PLoS Genet. 8, e1002496 (2012).

  64. 64.

    , , & Empirical power of very rare variants for common traits and disease: results from Sanger sequencing 1998 individuals. Eur. J. Hum. Genet. 21, 1027–1030 (2013).

  65. 65.

    , , , & Comparative study of statistical methods for detecting association with rare variants in exome-resequencing data. BMC Proc. 5, S33 (2011).

  66. 66.

    et al. Testing for an unusual distribution of rare variants. PLoS Genet. 7, e1001322 (2011).

  67. 67.

    , et al. Rare-variant association testing for sequencing data with the sequence kernel association test. Am. J. Hum. Genet. 89, 82–93 (2011). This is the original paper that describes the SKAT for rare-variant association.

  68. 68.

    et al. Analysis of rare, exonic variation amongst subjects with autism spectrum disorders and population controls. PLoS Genet. 9, e1003443 (2013).

  69. 69.

    et al. Searching for missing heritability: Designing rare variant association studies. Proc. Natl Acad. Sci. USA 111, E455–E464 (2013). This paper presents a framework for power calculation and ways to improve power for rare-variant studies.

  70. 70.

    et al. A method and server for predicting damaging missense mutations. Nature Methods 7, 248–249 (2010).

  71. 71.

    , , , & Using extreme phenotype sampling to identify the rare causal variants of quantitative traits in association studies. Genet. Epidemiol. 35, 790–799 (2011).

  72. 72.

    , , , & Rare variants of IFIH1, a gene implicated in antiviral responses, protect against type 1 diabetes. Science 324, 387–389 (2009).

  73. 73.

    & Linkage analysis in the next-generation sequencing era. Hum. Hered. 72, 228–236 (2011).

  74. 74.

    , , , & Family-based association tests for sequence data, and comparisons with population-based association tests. Eur. J. Hum. Genet. 21, 1158–1162 (2013).

  75. 75.

    et al. Functional impact of global rare copy number variation in autism spectrum disorders. Nature 466, 368–372 (2010).

  76. 76.

    et al. De novo gene disruptions in children on the autistic spectrum. Neuron 74, 285–299 (2012).

  77. 77.

    , et al. Rare complete knockouts in humans: population distribution and significant role in autism spectrum disorders. Neuron 77, 235–242 (2013).

  78. 78.

    , , & Three ways of combining genotyping and resequencing in case–control association studies. PLoS ONE 5, e14318 (2010).

  79. 79.

    et al. Combining effects from rare and common genetic variants in an exome-wide association study of sequence data. BMC Proc. 5, S44 (2011).

  80. 80.

    et al. Integrated model of de novo and inherited genetic variants yields greater power to identify risk genes. PLoS Genet. 9, e1003671 (2013).

  81. 81.

    & Detecting multiple causal rare variants in exome sequence data. Genet. Epidemiol. 35, S18–S21 (2011).

  82. 82.

    , & SimRare: a program to generate and analyze sequence-based data for association studies of quantitative and qualitative traits. Bioinformatics 28, 2703–2704 (2012).

  83. 83.

    & Differential confounding of rare and common variants in spatially structured populations. Nature Genet. 44, 243–246 (2012).

  84. 84.

    , , , & General framework for meta-analysis of rare variants in sequencing association studies. Am. J. Hum. Genet. 93, 42–53 (2013).

  85. 85.

    et al. Meta-analysis of gene-level associations for rare variants based on single-variant statistics. Am. J. Hum. Genet. 93, 236–248 (2013). References 83 and 84 propose powerful and convenient score tests for meta-analyses of rare-variant association studies.

  86. 86.

    , & Optimal tests for rare variant effects in sequencing association studies. Biostatistics 13, 762–775 (2012). This paper describes the SKAT power calculation tool.

  87. 87.

    et al. Analysis of copy number variations at 15 schizophrenia-associated loci. Br. J. Psychiatry 204, 108–114 (2013).

  88. 88.

    The power function of the test for the difference between two proportions in a 2 × 2 table. Biometrika 35, 157 (1948).

  89. 89.

    Rectangular confidence regions for the means of multivariate normal distributions. J. Am. Statist. Associ. 62, 626 (1967).

  90. 90.

    & Bootstrap Methods and Their Application (Cambridge Univ. Press, 1997).

  91. 91.

    The non-central χ2 - and F-distribution and their applications. Biometrika 36, 202 (1949).

  92. 92.

    & Power comparisons of the transmission/disequilibrium test and sib–transmission/disequilibrium-test statistics. Am. J. Hum. Genet. 65, 578–580 (1999).

  93. 93.

    , , & Combined linkage and association sib-pair analysis for quantitative traits. Am. J. Hum. Genet. 64, 259–267 (1999).

  94. 94.

    , , & Novel sib pair selection strategy increases power in quantitative association analysis. Behav. Genet. 39, 571–579 (2009).

  95. 95.

    Sample size determination for studies of gene–environment interaction. Int. J. Epidemiol. 30, 1035–1040 (2001).

  96. 96.

    Sample size requirements for association studies of gene–gene interaction. Am. J. Epidemiol. 155, 478–484 (2002).

  97. 97.

    Sample size requirements for matched case–control studies of gene–environment interaction. Statist. Med. 21, 35–50 (2002).

Download references

Acknowledgements

This work was supported by The University of Hong Kong Strategic Research Theme on Genomics; Hong Kong Research Grants Council (HKRGC) General Research Funds 777511M, 776412M and 776513M; HKRGC Theme-Based Research Scheme T12-705/11 and T12-708/12-N; and the European Community Seventh Framework Programme Grant on European Network of National Schizophrenia Networks Studying Gene–Environment Interactions (EU-GEI); and the US National Institutes of Health grants R01 MH099126 and R01 HG005827 (to S.M.P.). The authors thank R. Porsch and S.-W. Choi for technical assistance with the manuscript.

Author information

Affiliations

  1. Centre for Genomic Sciences, Jockey Club Building for Interdisciplinary Research; State Key Laboratory of Brain and Cognitive Sciences, and Department of Psychiatry, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China.

    • Pak C. Sham
  2. Center for Statistical Genetics, Icahn School of Medicine at Mount Sinai, New York 10029–6574, USA.

    • Shaun M. Purcell
  3. Center for Human Genetic Research, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts 02114, USA.

    • Shaun M. Purcell

Authors

  1. Search for Pak C. Sham in:

  2. Search for Shaun M. Purcell in:

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Pak C. Sham.

Glossary

Likelihoods

Probabilities (or probability densities) of observed data under an assumed statistical model as a function of model parameters.

Family-wise error rate

(FWER). The probability of at least one false-positive significant finding from a family of multiple tests when the null hypothesis is true for all the tests.

C-alpha test

A rare-variant association test based on the distribution of variants in cases and controls (that is, whether such a distribution has inflated variance compared with a binomial distribution).

Sequence kernel association test

(SKAT). A test based on score statistics for testing the association of rare variants from sequence data with either a continuous or a discontinuous genetic trait.

About this article

Publication history

Published

DOI

https://doi.org/10.1038/nrg3706

Further reading