Rahmani et al. reply:
Zheng et al.1 discuss potential pitfalls in our evaluation of ReFACTor2, a reference-free method to account for cell-type heterogeneity. Below, we reproduce their analysis and demonstrate that conclusions cannot be drawn on the basis of their results owing to conceptual and technical flaws in their analysis. We show with our reanalysis and further evidence from experiments on a total of 10 data sets that ReFACTor has improved performance over alternative methods, including the reference-based method of Houseman et al.3.
Zheng et al.1 claim that more evidence needs to be provided to determine whether ReFACTor is applicable to tissues other than blood. They generated a “gold standard” set of “true positives” and “true negatives” for breast cancer differentially methylated CpGs (DMCs) and compared ReFACTor to Surrogate Variable Analysis (SVA)4 using EWAS data. There are multiple problems with this analysis. First, the list of 'true positives' is unreliable owing to the fact that only two control individuals were used for its construction (Supplementary Note 1). We show through a simple permutation analysis that using only two controls is likely to result in tens of thousands of spurious 'true positives' (Supplementary Fig. 1). Therefore, benchmarking on these 'true positives' is an invalid approach. Second, Zheng et al.1 report improved sensitivity for SVA; however, they do not report that a simple unadjusted analysis using a standard Bonferroni significance level achieves considerably better sensitivity and greater specificity than SVA (Supplementary Table 1). Thus, the metric used to evaluate performance is also invalid, as the naive method that does not adjust for cell-type heterogeneity outperforms a method that does. A detailed description of this experiment as well as additional flaws in their analysis is given in Supplementary Note 1.
The focus of Zheng et al.1 on the potential loss of power in the case of many true positives is of interest. Because a reliable gold standard is currently not available, we examined this scenario by splitting a large set of breast cancer samples (n = 305)5 into two groups on the basis of the reference-based cell-composition estimates provided by Zheng et al.1. One group was labeled as controls, and differential methylation effects were added to all samples in the other group in more than 20,000 sites. The results (Supplementary Note 2 and Supplementary Tables 2–4) show that ReFACTor and SVA obtain similar sensitivity, but ReFACTor captures the cell composition substantially better than SVA and thus adjusts well for false positives, whereas SVA suffers from thousands of false positives. In contrast to the argument of Zheng et al.1, when ReFACTor is correctly applied (Zheng et al.1 did not follow our guidelines), the ReFACTor components are dominated by information about cell-type composition rather than disease status (Supplementary Note 1 and Supplementary Fig. 2).
Zheng et al.1 next consider our original experiment in which FACS cell counts were available2. They argue that successively adding components may cause overfitting. However, our point in that section of Rahmani et al.2. was to evaluate the relative performance of different methods as a function of model dimension, and thus there is no issue of overfitting (Supplementary Note 3). They evaluated ReFACTor by measuring the correlation between each cell type and ReFACTor components selected via likelihood ratio test (LRT) and observed that ReFACTor only slightly improves upon the reference-based approach. However, LRT depends on sample size, hence we re-evaluated ReFACTor using LRT with all 560 samples in the data set (as opposed to a subset). Our analysis revealed more significant components, which leads to a substantial improvement, far outperforming the reference-based approach (Supplementary Note 3 and Supplementary Fig. 3).
Finally, Zheng et al.1 try to demonstrate the advantage of the reference-based method3 using a very small data set with known cell composition (n = 18)6. However, in their analysis, Zheng et al.1 did not correct for known batch effects, and we found that adjusting for batch information produces similar performance for ReFACTor and the reference-based method (Supplementary Fig. 4). Furthermore, such a small sample size cannot provide statistically significant evidence for the improvement of any method. Specifically, using multiple subsampled FACS data sets of 18 samples, we observed that the performance of both methods was highly variable (Supplementary Fig. 5). Moreover, Zheng et al.1 relied on a method for determining the dimension of the data (RMT)7. We found that the number of dimensions estimated by RMT is linearly determined by the sample size (R2 > 0.95), making it inapplicable (Supplementary Fig. 6 and Supplementary Note 3).
Given that firm conclusions cannot be drawn based on small data sets, we further evaluated the performance of ReFACTor and the reference-based method using five large whole-blood data sets (minimum n = 312). We divided the samples in each data set into two groups on the basis of cell-composition distribution (Supplementary Note 3). Then, we conducted an EWAS on the assignment into groups as the phenotype. In this scenario, the assignment into groups is expected to be correlated with the true underlying cell composition, and an insufficient correction will lead to spurious associations. We found that ReFACTor consistently outperformed the reference-based method; particularly, the reference-based method resulted in more than 1,000 false positives in some cases, whereas ReFACTor resulted in a few dozen at most (Supplementary Note 3 and Table 1). This is not surprising given the inherent limitations of reference-based methods (Supplementary Note 4).
Comprehensive guidelines for using ReFACTor are provided in Supplementary Note 5, and a more comprehensive implementation of ReFACTor is available at http://glint-epigenetics.readthedocs.io. We also provide recommendations for the choice of algorithms for specific tasks in methylation analysis in Supplementary Table 5. We agree with Zheng et al.1 that the community should constantly seek further evaluation of existing methods in more scenarios and using more data sets and tissues. Specifically, reasonable sample sizes should be used to draw significant conclusions. Our results provide multiple compelling evidence based on multiple large data sets that ReFACTor is currently the state-of-the-art approach for controlling for tissue heterogeneity in EWAS, even when compared to the reference-based method in whole blood.
Data availability statement. All data analyzed are publicly available. See Supplementary Methods for details.
E.H. and E.R. designed the experiments. E.R. performed the experiments. E.R., N.Z., Y.B., C.E., D.H., J.G., S.O., E.G.B., E.E., J.Z. and E.H. wrote the manuscript.
Zheng, S.C. et al. Nat. Methods 14, 216–217 (2017).
Rahmani, E. et al. Nat. Methods 13, 443–445 (2016).
Houseman, E.A. et al. BMC Bioinformatics 13, 86 (2012).
Leek, J.T. & Storey, J.D. PLoS Genet. 3, e161 (2007).
Teschendorff, A.E. et al. Nat. Commun. 7, 10478 (2016).
Koestler, D.C. et al. BMC Bioinformatics 17, 120 (2016).
Teschendorff, A.E., Zhuang, J. & Widschwendter, M. Bioinformatics 27, 1496–1505 (2011).
Tan, Q. et al. BMC Genomics 15, 1062 (2014).
Liu, Y. et al. Nat. Biotechnol. 31, 142–147 (2013).
Hannon, E. et al. Genome Biol. 17, 176 (2016).
This research was partially supported by the Edmond J. Safra Center for Bioinformatics at Tel-Aviv University, the Israel Science Foundation (1425/13 to E.R. and E.H.), US National Science Foundation grant 1331176 and United States Israel Binational Science Foundation grant 2012304 (to E.R., Y.B. and E.H.). E.R. was supported by Len Blavatnik and the Blavatnik Research Foundation. N.Z. was supported in part by a US National Institutes of Health (NIH) career development award from the NHLBI (K25HL121295). C.E., S.H., D.H., J.G., S.O. and E.G.B. were supported by the Sandler Family Foundation, the American Asthma Foundation, Hind Distinguished Professorships and NIH grants 1P60MD006902, 1R01HL117004, R21ES24844, R01Hl128439 and TRDRP 24RT-0025. E.E. was supported by NSF grants 1065276, 1302448, 1320589 and 1331176 and NIH grants R01-GM083198, R01-ES021801, R01-MH101782, R01-ES022282 and U54EB020403.
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Evaluation of the number of expected spurious associations when using a small number of controls in EWAS.
The histogram of significant associations found across 100 EWAS experiments on the data used by Zheng et al. for constructing the gold standard list of “true positives”. The red line marks 23,258, the number of sites defined in the “true positives” list by Zheng et al.
(a) A reconstruction of Figure S1 from Zheng et al., showing correlation of the cell-types and disease status (N/C) with each of the first 25 principal components of the data (n=355). Here, as well as in the following subfigures, the colors correspond to the logarithm of the P-values of the correlations. (b) The correlation of the first 25 ReFACTor components with the cell-types and disease status, as well as with the variation of the disease status that is independent of the cell composition (Adj. N/C). (c) The correlation of the first 25 SVA components (SVs) with the cell-types and with the unadjusted and adjusted disease status. (d) The mean R2 levels, across the nine estimated cell-types, of linear models fitted for each cell-type using an increasing number of ReFACTor component and using an increasing number of SVs. For any given number of components, ReFACTor has better R2 level than SVA.
(a)-(d) R2 values of the linear model predicting flow-cytometric estimates for blood cell-types, as a function of the number of ReFACTor components included in the model (blue data points and lines) for the GALA II dataset (n=84). Horizontal blue lines indicate the R2 values of the model using the ReFACTor components with significant likelihood ratio test (LRT) P-values (significant components are marked with squares). The reference-based estimates of the entire GALA II dataset (n=560) were used to determine the number of significant ReFACTor components. Horizontal orange lines indicate the performance of the reference-based method. (e) The mean R2 level over all cell-types.
(a)-(f) R2 values of the linear model predicting flow-cytometric estimates for blood cell-types, as a function of the number of ReFACTor components included in the model (blue data points and lines) for the Koestler et al. dataset (n=18). Horizontal blue lines indicate the R2 values of the model using the ReFACTor components with significant likelihood ratio test (LRT) P-values (significant components are marked with squares). Horizontal orange lines indicate the performance of the reference-based method. (g) The mean R2 level over all cell-types.
Supplementary Figure 5 Performance for capturing cell-type composition in small data is highly variable.
(a) Sampling 100 subsets of 18 individuals with cell counts from the GALA II dataset (n=84) reveals that the performance (measured in mean R2 across all cell-types) of both ReFACTor and the reference-based method are highly variable due to the small number of samples used. (b) Distribution of the performance after sampling 100 subsets of 15 individuals from the Koestler et al. data (n=18).
The estimated dimensions by the RMT method (Teschendorff et al. 2011) as a function of the number of samples (each time adding a new randomly selected sample) using (a) the GALA II dataset (n=560), (b) a dataset by Liu et al. (n=686) and (c)-(d) two independent datasets by Hannon et al (n=675 and n=847). Linear regression lines (indicated in red) demonstrate a nearly perfect linear relation (P-value<10−93 in all plots).
About this article
Cite this article
Rahmani, E., Zaitlen, N., Baran, Y. et al. Correcting for cell-type heterogeneity in DNA methylation: a comprehensive evaluation. Nat Methods 14, 218–219 (2017). https://doi.org/10.1038/nmeth.4190
BMC Genomics (2020)
Comparing DNA methylation profiles across different tissues associated with the diagnosis of pediatric asthma
Scientific Reports (2020)
CONFINED: distinguishing biological from technical sources of variation by leveraging multiple methylation datasets
Genome Biology (2019)
Nature Communications (2019)
Nature Communications (2019)