Correcting for cell-type heterogeneity in DNA methylation: a comprehensive evaluation

This article has been updated

Rahmani et al. reply:

Zheng et al.1 discuss potential pitfalls in our evaluation of ReFACTor2, a reference-free method to account for cell-type heterogeneity. Below, we reproduce their analysis and demonstrate that conclusions cannot be drawn on the basis of their results owing to conceptual and technical flaws in their analysis. We show with our reanalysis and further evidence from experiments on a total of 10 data sets that ReFACTor has improved performance over alternative methods, including the reference-based method of Houseman et al.3.

Zheng et al.1 claim that more evidence needs to be provided to determine whether ReFACTor is applicable to tissues other than blood. They generated a “gold standard” set of “true positives” and “true negatives” for breast cancer differentially methylated CpGs (DMCs) and compared ReFACTor to Surrogate Variable Analysis (SVA)4 using EWAS data. There are multiple problems with this analysis. First, the list of 'true positives' is unreliable owing to the fact that only two control individuals were used for its construction (Supplementary Note 1). We show through a simple permutation analysis that using only two controls is likely to result in tens of thousands of spurious 'true positives' (Supplementary Fig. 1). Therefore, benchmarking on these 'true positives' is an invalid approach. Second, Zheng et al.1 report improved sensitivity for SVA; however, they do not report that a simple unadjusted analysis using a standard Bonferroni significance level achieves considerably better sensitivity and greater specificity than SVA (Supplementary Table 1). Thus, the metric used to evaluate performance is also invalid, as the naive method that does not adjust for cell-type heterogeneity outperforms a method that does. A detailed description of this experiment as well as additional flaws in their analysis is given in Supplementary Note 1.

The focus of Zheng et al.1 on the potential loss of power in the case of many true positives is of interest. Because a reliable gold standard is currently not available, we examined this scenario by splitting a large set of breast cancer samples (n = 305)5 into two groups on the basis of the reference-based cell-composition estimates provided by Zheng et al.1. One group was labeled as controls, and differential methylation effects were added to all samples in the other group in more than 20,000 sites. The results (Supplementary Note 2 and Supplementary Tables 2–4) show that ReFACTor and SVA obtain similar sensitivity, but ReFACTor captures the cell composition substantially better than SVA and thus adjusts well for false positives, whereas SVA suffers from thousands of false positives. In contrast to the argument of Zheng et al.1, when ReFACTor is correctly applied (Zheng et al.1 did not follow our guidelines), the ReFACTor components are dominated by information about cell-type composition rather than disease status (Supplementary Note 1 and Supplementary Fig. 2).

Zheng et al.1 next consider our original experiment in which FACS cell counts were available2. They argue that successively adding components may cause overfitting. However, our point in that section of Rahmani et al.2. was to evaluate the relative performance of different methods as a function of model dimension, and thus there is no issue of overfitting (Supplementary Note 3). They evaluated ReFACTor by measuring the correlation between each cell type and ReFACTor components selected via likelihood ratio test (LRT) and observed that ReFACTor only slightly improves upon the reference-based approach. However, LRT depends on sample size, hence we re-evaluated ReFACTor using LRT with all 560 samples in the data set (as opposed to a subset). Our analysis revealed more significant components, which leads to a substantial improvement, far outperforming the reference-based approach (Supplementary Note 3 and Supplementary Fig. 3).

Finally, Zheng et al.1 try to demonstrate the advantage of the reference-based method3 using a very small data set with known cell composition (n = 18)6. However, in their analysis, Zheng et al.1 did not correct for known batch effects, and we found that adjusting for batch information produces similar performance for ReFACTor and the reference-based method (Supplementary Fig. 4). Furthermore, such a small sample size cannot provide statistically significant evidence for the improvement of any method. Specifically, using multiple subsampled FACS data sets of 18 samples, we observed that the performance of both methods was highly variable (Supplementary Fig. 5). Moreover, Zheng et al.1 relied on a method for determining the dimension of the data (RMT)7. We found that the number of dimensions estimated by RMT is linearly determined by the sample size (R2 > 0.95), making it inapplicable (Supplementary Fig. 6 and Supplementary Note 3).

Given that firm conclusions cannot be drawn based on small data sets, we further evaluated the performance of ReFACTor and the reference-based method using five large whole-blood data sets (minimum n = 312). We divided the samples in each data set into two groups on the basis of cell-composition distribution (Supplementary Note 3). Then, we conducted an EWAS on the assignment into groups as the phenotype. In this scenario, the assignment into groups is expected to be correlated with the true underlying cell composition, and an insufficient correction will lead to spurious associations. We found that ReFACTor consistently outperformed the reference-based method; particularly, the reference-based method resulted in more than 1,000 false positives in some cases, whereas ReFACTor resulted in a few dozen at most (Supplementary Note 3 and Table 1). This is not surprising given the inherent limitations of reference-based methods (Supplementary Note 4).

Table 1 Numbers of false positives in whole-blood EWAS

Comprehensive guidelines for using ReFACTor are provided in Supplementary Note 5, and a more comprehensive implementation of ReFACTor is available at http://glint-epigenetics.readthedocs.io. We also provide recommendations for the choice of algorithms for specific tasks in methylation analysis in Supplementary Table 5. We agree with Zheng et al.1 that the community should constantly seek further evaluation of existing methods in more scenarios and using more data sets and tissues. Specifically, reasonable sample sizes should be used to draw significant conclusions. Our results provide multiple compelling evidence based on multiple large data sets that ReFACTor is currently the state-of-the-art approach for controlling for tissue heterogeneity in EWAS, even when compared to the reference-based method in whole blood.

Data availability statement. All data analyzed are publicly available. See Supplementary Methods for details.

Author contributions

E.H. and E.R. designed the experiments. E.R. performed the experiments. E.R., N.Z., Y.B., C.E., D.H., J.G., S.O., E.G.B., E.E., J.Z. and E.H. wrote the manuscript.

Change history

  • 14 March 2017

    In the version of this article initially published, some numbers in Table 1 did not appear in boldface. In the HTML originally posted online, the author affiliation for Elior Rahmani was incorrect; Rahmani is affiliated with only the Tel-Aviv University, Israel. The Supplementary Information file has been replaced to correct for additional callouts of Supplementary Notes in the Supplementary Figure legends. The errors have been corrected in the HTML and PDF files as of 14 March 2017.

References

  1. 1

    Zheng, S.C. et al. Nat. Methods 14, 216–217 (2017).

    CAS  Article  Google Scholar 

  2. 2

    Rahmani, E. et al. Nat. Methods 13, 443–445 (2016).

    CAS  Article  Google Scholar 

  3. 3

    Houseman, E.A. et al. BMC Bioinformatics 13, 86 (2012).

    Article  Google Scholar 

  4. 4

    Leek, J.T. & Storey, J.D. PLoS Genet. 3, e161 (2007).

    Article  Google Scholar 

  5. 5

    Teschendorff, A.E. et al. Nat. Commun. 7, 10478 (2016).

    CAS  Article  Google Scholar 

  6. 6

    Koestler, D.C. et al. BMC Bioinformatics 17, 120 (2016).

    Article  Google Scholar 

  7. 7

    Teschendorff, A.E., Zhuang, J. & Widschwendter, M. Bioinformatics 27, 1496–1505 (2011).

    CAS  Article  Google Scholar 

  8. 8

    Tan, Q. et al. BMC Genomics 15, 1062 (2014).

    Article  Google Scholar 

  9. 9

    Liu, Y. et al. Nat. Biotechnol. 31, 142–147 (2013).

    CAS  Article  Google Scholar 

  10. 10

    Hannon, E. et al. Genome Biol. 17, 176 (2016).

    Article  Google Scholar 

Download references

Acknowledgements

This research was partially supported by the Edmond J. Safra Center for Bioinformatics at Tel-Aviv University, the Israel Science Foundation (1425/13 to E.R. and E.H.), US National Science Foundation grant 1331176 and United States Israel Binational Science Foundation grant 2012304 (to E.R., Y.B. and E.H.). E.R. was supported by Len Blavatnik and the Blavatnik Research Foundation. N.Z. was supported in part by a US National Institutes of Health (NIH) career development award from the NHLBI (K25HL121295). C.E., S.H., D.H., J.G., S.O. and E.G.B. were supported by the Sandler Family Foundation, the American Asthma Foundation, Hind Distinguished Professorships and NIH grants 1P60MD006902, 1R01HL117004, R21ES24844, R01Hl128439 and TRDRP 24RT-0025. E.E. was supported by NSF grants 1065276, 1302448, 1320589 and 1331176 and NIH grants R01-GM083198, R01-ES021801, R01-MH101782, R01-ES022282 and U54EB020403.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Eran Halperin.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Evaluation of the number of expected spurious associations when using a small number of controls in EWAS.

The histogram of significant associations found across 100 EWAS experiments on the data used by Zheng et al. for constructing the gold standard list of “true positives”. The red line marks 23,258, the number of sites defined in the “true positives” list by Zheng et al.

Supplementary Figure 2 Capturing cell-type composition in breast cancer data using ReFACTor.

(a) A reconstruction of Figure S1 from Zheng et al., showing correlation of the cell-types and disease status (N/C) with each of the first 25 principal components of the data (n=355). Here, as well as in the following subfigures, the colors correspond to the logarithm of the P-values of the correlations. (b) The correlation of the first 25 ReFACTor components with the cell-types and disease status, as well as with the variation of the disease status that is independent of the cell composition (Adj. N/C). (c) The correlation of the first 25 SVA components (SVs) with the cell-types and with the unadjusted and adjusted disease status. (d) The mean R2 levels, across the nine estimated cell-types, of linear models fitted for each cell-type using an increasing number of ReFACTor component and using an increasing number of SVs. For any given number of components, ReFACTor has better R2 level than SVA.

Supplementary Figure 3 Capturing cell-type composition variation in the GALA II dataset.

(a)-(d) R2 values of the linear model predicting flow-cytometric estimates for blood cell-types, as a function of the number of ReFACTor components included in the model (blue data points and lines) for the GALA II dataset (n=84). Horizontal blue lines indicate the R2 values of the model using the ReFACTor components with significant likelihood ratio test (LRT) P-values (significant components are marked with squares). The reference-based estimates of the entire GALA II dataset (n=560) were used to determine the number of significant ReFACTor components. Horizontal orange lines indicate the performance of the reference-based method. (e) The mean R2 level over all cell-types.

Supplementary Figure 4 Capturing cell-type composition variation in the Koestler et al. dataset.

(a)-(f) R2 values of the linear model predicting flow-cytometric estimates for blood cell-types, as a function of the number of ReFACTor components included in the model (blue data points and lines) for the Koestler et al. dataset (n=18). Horizontal blue lines indicate the R2 values of the model using the ReFACTor components with significant likelihood ratio test (LRT) P-values (significant components are marked with squares). Horizontal orange lines indicate the performance of the reference-based method. (g) The mean R2 level over all cell-types.

Supplementary Figure 5 Performance for capturing cell-type composition in small data is highly variable.

(a) Sampling 100 subsets of 18 individuals with cell counts from the GALA II dataset (n=84) reveals that the performance (measured in mean R2 across all cell-types) of both ReFACTor and the reference-based method are highly variable due to the small number of samples used. (b) Distribution of the performance after sampling 100 subsets of 15 individuals from the Koestler et al. data (n=18).

Supplementary Figure 6 RMT estimates of the dimension in data as a function of the sample size.

The estimated dimensions by the RMT method (Teschendorff et al. 2011) as a function of the number of samples (each time adding a new randomly selected sample) using (a) the GALA II dataset (n=560), (b) a dataset by Liu et al. (n=686) and (c)-(d) two independent datasets by Hannon et al (n=675 and n=847). Linear regression lines (indicated in red) demonstrate a nearly perfect linear relation (P-value<10−93 in all plots).

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–6, Supplementary Tables 1–6, Supplementary Methods and Supplementary Notes 1–5 (PDF 720 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rahmani, E., Zaitlen, N., Baran, Y. et al. Correcting for cell-type heterogeneity in DNA methylation: a comprehensive evaluation. Nat Methods 14, 218–219 (2017). https://doi.org/10.1038/nmeth.4190

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing