Genotype Query Tools (GQT) is an indexing strategy that expedites analyses of genome-variation data sets in Variant Call Format based on sample genotypes, phenotypes and relationships. GQT's compressed genotype index minimizes decompression for analysis, and its performance relative to that of existing methods improves with cohort size. We show substantial (up to 443-fold) gains in performance over existing methods and demonstrate GQT's utility for exploring massive data sets involving thousands to millions of genomes. GQT can be accessed at https://github.com/ryanlayer/gqt.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Genome Biology Open Access 19 August 2021
Human Genetics Open Access 06 February 2021
BMC Genomics Open Access 23 November 2020
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Zuk, O. et al. Proc. Natl. Acad. Sci. USA 111, E455–E464 (2014).
Stephens, Z.D. et al. PLoS Biol. 13, e1002195 (2015).
Danecek, P. et al. Bioinformatics 27, 2156–2158 (2011).
Keinan, A. & Clark, A.G. Science 336, 740–743 (2012).
1000 Genomes Project Consortium. et al. Nature 491, 56–65 (2012).
Wu, K., Otoo, E.J. & Shoshani, A. In Proc. 14th International Conference on Scientific and Statistical Database Management (Ed. Kennedy, J.) 99–108 (IEEE, 2002).
Siqueira, T.L.L., Ciferri, C.D.deA., Times, V.C., de Oliveira, A.G. & Ciferri, R.R. J. Braz. Comput. Soc. 15, 19–34 (2009).
Liu, Y.-B. et al. J. Korean Astron. Soc. 47, 115–122 (2014).
Quinlan, A.R. & Hall, I.M. Bioinformatics 26, 841–842 (2010).
Li, H. Bioinformatics 27, 718–719 (2011).
Ziv, J. & Lempel, A. IEEE Trans. Inf. Theory 23, 337–343 (1977).
Weir, B.S. & Cockerham, C.C. Evolution 38, 1358 (1984).
Chen, G.K., Marjoram, P. & Wall, J.D. Genome Res. 19, 136–142 (2009).
We are grateful to C. Chiang for conceptual discussions, I. Levicki for helpful advice on AVX2 operations, S. McCarthy and P. Danecek for their guidance with htslib, and Z. Kronenberg for advice on population genetics measures. We also thank the Exome Aggregation Consortium and the groups that provided exome variant data for comparison. A full list of contributing groups can be found at http://exac.broadinstitute.org/about. This research was supported by a US National Human Genome Research Institute award to A.R.Q. (NIH R01HG006693).
The authors declare no competing financial interests.
A list of members and affiliations appears in Supplementary Note 1.
Integrated supplementary information
(a) A graphical comparison of the genotype distribution of individuals (rows) and variants (columns) before and after sorting. These data represent genotypes from the 1,000 Genomes Project, phase 3, for a portion chromosome 20. (b) The run length distribution for unsorted and sorted genotypes. (c) The distribution of the number of runs for sorted and unsorted data. In both cases the second peak is composed predominantly of individuals from African decent (AFR); 604/661 AFR are in the second peak in the sorted case, and 640/661 AFR in the unsorted case.
(a) A bit array marks the existence of one genotype state (for example, homozygous reference; “0/0”) for all variants. Similarly, a bitmap index is composed of a distinct bit array for each possible genotype state. (b) Example genotypes in VCF format are presented for three individuals (I1, I2, I3) at 10 variant sites (V1-V10). A bitwise AND of the bit arrays corresponding to the heterozygous genotype yields the variant that is heterozygous in all individuals.
(a) When considering genotypes in ASCII format (e.g., VCF), an algorithm searching for the set of variants that are heterozygous in all individuals must operate on every genotype for each for every individual separately. (b) In contrast, when genotypes are represented with a bitmap index, where a set of genotypes are encoded into a single CPU word (for brevity, only the bit arrays associated with the heterozygous state are shown), bitwise logical operations can be used to operate on all of the genotypes in the word with a single operation. This example assumes a word size of 8, but modern CPU support up to a 64-bit word and the speedup of bitmaps will be linear to the word size. For the 24 genotypes given here (3 individuals, 8 genotypes each), the ASCII-base algorithm executes the “if” statement 24 times, while the bit-wise algorithm executes the logical AND (“&”) only three times, with both algorithms producing equivalent results.
Two bit arrays are given in three different binary encodings: the uncompressed bit array (in 7 bit words) on top, followed by the run-length encoding (RLE) and word-aligned hybrid encoding (WAH), which each add an additional bit as part of the encoding to make 8 bit words. RLE maps each set of consecutive bits to a new value that uses one bit for the run value, and the remaining bits for the number of bits in the run. WAH maps bits to one of two types of values: those that include runs and those that encode the raw binary. The first bit in a WAH value indicates if the value encodes a run (the fill bit). If that bit is set then the second bit gives the run value and the remaining bits give the run length in number of words (i.e., not number of bits as in RLE). If the fill bit is not set then the remaining bits are the uncompressed binary values. (b) The logical OR for the three encodings is given. For the uncompressed binary, the OR follows bit for bit across both values. For RLE, the logical OR is undefined (without inflation) because the two encoded values are no longer aligned and have different lengths. For WAH, the values are aligned based on their run length, then the logical OR is performed (in this case) between the value bit and the associated uncompressed values.
(a) The CEPH 1463 pedigree. Our analysis is focused on the discovery of de novo mutations in NA12878, the daughter of NA12891 and NA12892. (b) A GQT query for de novo mutations based on the expected genotypes (homozygous for the reference allele) in NA12878’s parents, as well as an expected heterozygous genotype in NA12878. (c) True de novo mutations in NA12878’s germline should be passed on to 50% of her offspring, on average. Allowing for genotyping error and binomial expectation, we filter for more confident de novo mutation candidates by requiring that the apparent mutation allele is passed on to at least 30% of NA12878’s children. (d) A GQT query that further filters candidate mutations by excluding those lying in low complexity regions of the genome.
VCF files are composed of a variant data section and a sample genotype section and each of these include both core information (e.g. variant position and alleles, and genotype, respectively) and extra metadata. BCF encodes both sections into a binary format, and then compresses the data using blocked LZ77 compression. GQT uses three files: a BIM file that contains all variant data compressed with LZ77, a GQT file with WAH-compressed genotypes (without metadata), and a VID file that stores the mapping between the allele-frequency variant order and the original source VCF variant order. PLINK (not shown) omits all metadata, storing uncompressed binary encoded for genotypes in a BED file and variant positions in a BIM file.
Supplementary Figure 7 GQT compression performance for 1,000 Genomes data (phase 3) and simulated genomes.
Compression ratios describe the fold reduction in file size relative to an uncompressed VCF file. GQT represents solely total size of the GQT genotype index. GQT* reflects the total size of the GQT genotype index plus the size of the BIM and VID indices. BCF* represents the size of the full BCF file including genotype metadata, whereas BCF omits genotype metadata. Lastly, PLINK reflects the size of PLINK v1.9 BED and BIM files. (a) File size reduction for 1,000 Genomes phase 3, which comprised 2,504 individuals and more than 84 million variants. (b) A comparison of BCFTOOLS, GQT, and PLINK for simulated genotypes on a 100-Mb genome with between 100 and 100,000 individuals.
FST for 1,000 Genomes phase 3 of Europeans versus East Asians and Europeans versus Africans on chromosome 12.
About this article
Cite this article
Layer, R., Kindlon, N., Karczewski, K. et al. Efficient genotype compression and analysis of large genetic-variation data sets. Nat Methods 13, 63–65 (2016). https://doi.org/10.1038/nmeth.3654
Genome Biology (2021)
Human Genetics (2021)
BMC Genomics (2020)
Translational Psychiatry (2020)
BMC Bioinformatics (2019)