In a recent paper published online in Molecular Psychiatry, Skafidas et al.1 report a classifier for identifying individuals at risk for autism spectrum disorders (ASDs). Their classifier is based on 267 single-nucleotide polymorphisms (SNPs) that were selected from the results of a pathway analysis using cases from the Autism Genetic Resource Exchange (AGRE).1 Using within-sample cross-validation, the authors claim a classification accuracy for ASDs of 85.6%. They subsequently applied their classifier to ASD cases from the Simons Foundation Autism Research Initiative (SFARI) and controls from the Wellcome Trust Birth Cohort (WTBC) and report ASD classification accuracy of 71.7%.
We believe that the claims made by Skafidas et al.1 are inconsistent with current knowledge of the genetics of ASDs,2 and inconsistent with the expected precision of risk predictions for complex psychiatric disorders. Further, as classification accuracy depends on the size of the discovery sample, the results are also inconsistent with the size of the sample they employed (only 123 controls were included in the discovery set).
To examine the validity of Skafidas et al.’s claims, we pursued a range of analyses to assess the evidence for association between ASDs and (1) the individual SNPs named in their paper as most predictive, (2) their genetic classifier, to the extent it was described and (3) the pathways identified in the report, from which the predictive SNPs were selected. For each analysis, where possible, we attempted to replicate the analytic approach of Skafidas et al.1 using data from the Psychiatric Genomics Consortium (PGC) autism group, which includes ∼5400 cases, more than three times the number used in the original report. The methodology of these analyses is described in detail in Supplementary Information.
First, we found no evidence for single SNP associations between any of the 30 most contributory SNPs listed by Skafidas et al.1 in their Table 2 and ASDs in the PGC (Table 1). In the current PGC meta-analysis, the mean P-value for these SNPs was 0.47 with a minimum 0.007, and none are notable or survive a 30 SNP correction for multiple testing. Further information on these associations can be found in Supplementary Information.
Second, we examined the classification ability of the 30 SNPs disclosed in Skafidas et al.1 (their Table 2) for ASDs in the PGC. We wrote to the authors, asking for the complete list of 237 SNPs and weights, but they declined to provide the complete list. We accordingly built a classifier using the data for 30 SNPs disclosed in Skafidas et al.,1 which the authors identify as the most influential (explaining approximately 58% of the total predictive power of the classifier). We constructed the classifier using two approaches. We initially used the weights provided by Skafidas et al.1 and examined the predictive ability of the 30 SNP classifier in the full PGC autism sample. As described in detail in Supplementary Information, the classifier did not differ from chance in its ability to predict ASDs (AUC=0.505, P=0.22).
We then built the score using the SNP weights estimated from the PGC data. We randomly selected a set of 732 trios to build a classifier and then tested the predictive ability of the classifier in a distinct set of 243 trios (these number mirror those used by Skafidas et al.1). For all trios, we created case pseudo–control pairs to perform model building and validation, but otherwise followed the methods proposed in Skafidas et al.1 (for example, using 0, 1, 3 scoring against minor allele count). We repeated this procedure across 100 random samples of the same size from the PGC autism data. Across these replicates, we tested for a difference between case and control risk scores using a t-test (mean risk score of cases—mean risk score of controls) and found an average t-statistic of 0.492 with an average P-value of 0.50 for the validation samples. We conclude that the classifier presented by Skafidas et al.,1 at least as constructed using the 30 top SNPs named in their report, does not generalize to predict ASDs in other samples. This result strongly suggests that the Skafidas et al.1 results cannot be used to predict ASDs.
We repeated the set of analyses above using a case–control design, to mirror the approach employed by Skafidas et al.1 We used 732 cases matched with 732 population controls for discovery, and 243 cases matched with 243 population controls for validation, much as the authors initially reported. In these comparisons, when principal components were included in the analysis to control for population ancestry, we observed nearly identical results to what we found in the family-based study described above (see Supplementary Information). However, without controlling for population ancestry, we observed a bias in estimates of the AUC for the curve, suggesting that such bias may have contributed to the results reported by Skafidas et al., as has already been suggested.3
Finally, we evaluated the significance of the pathways identified by Skafidas et al.1 (their Table 1), the analysis which provided the basis for their SNP selection. We did not observe significant evidence for a relationship between any of these pathways and ASDs using five different pathway analysis tools in the combined PGC ASD sample set (Table 2). This result strongly suggests that the pathway analyses do not generalize to external samples and therefore cannot be validly used in the development of a classifier.
To put the results reported in Skafidas et al.1 into perspective, consider the magnitude of effects implied by the results of the classifier. From the external validation experiment, the authors report an area under the receiver operating characteristic curve 0.747 (Skafidas et al., Supplementary Figure S2). This result implies that their SNP-set explains ∼11% of variation in liability to ASDs (assuming a prevalence of 1% and a liability threshold model).4 For complex traits, in particular psychiatric disorders, explaining so much variation with so few SNPs and such a small discovery sample size (732 cases and 123 controls) is unprecedented, and inconsistent with results from genome-wide association studies. For example, to achieve similar levels of variance explained in human height, sample sizes of ∼180 000 individuals were required.5
We find no evidence that the implicated SNPs, the classifier or the pathways named in Skafidas et al.1 are associated with ASDs. We therefore conclude that the classifier, as presented, cannot be used in a general way to predict ASDs, and consequently is unlikely to have any translational value.
The differences between the report of Skafidas et al.1 and our analyses are striking. We suspect that our failures to replicate their claims originate from several issues with the original analyses and data. In particular, the failure to control for potential population stratification in Skafidas et al.1 has likely led to biased estimates of allelic effects, as suggested in a recent letter.3 We detail other technical issues in Supplementary Information, which may also explain the differences in the results.
There are a great many challenges to the accurate interpretation of genomic data and multiple false-positive associations from technical or study design biases have been identified in the literature. We conclude that the classifier presented in Skafidas et al.1 will not usefully identify individuals at risk for ASDs in the population. Nevertheless, there are increasing numbers of robust and replicable finding emerging in psychiatric genetics. These findings hold great promise for understanding the biological basis of psychiatric disorders and for translation.
Skafidas E, Testa R, Zantomio D, Chana G, Everall IP, Pantelis C . Mol Psychiatry, advance online publication, 11 September 2012; doi:10.1038/mp.2012.126 (e-pub ahead of print).
Anney R, Klei L, Pinto D, Almeida J, Bacchelli E, Baird G et al. Hum Mol Genet 2012; 21: 4781–4792.
Belgard TG, Jankovic I, Lowe JK, Geschwind DH . Mol Psychiatry, advance online publication, 2 April 2013; doi:10.1038/mp.2013.34 (e-pub ahead of print).
Wray NR, Yang J, Goddard ME, Visscher PM . PLoS Genet 2010; 6: e1000864.
Lango Allen H, Estrada K, Lettre G, Berndt SI, Weedon MN, Rivadeneira F et al. Nature 2010; 467: 832–838.
Moskvina V, O’Dushlaine C, Purcell S, Craddock N, Holmans P, O’Donovan MC . Genet Epidemiol 2011; 35: 861–866.
Pedroso I, Lourdusamy A, Rietschel M, Nöthen MM, Cichon S, McGuffin P et al. Biol Psychiatry 2012; 72: 311–317.
Holmans P, Green EK, Pahwa JS, Ferreira MA, Purcell SM, Sklar P et al. Am J Hum Genet 2009; 85: 13–24.
Segrè AV, DIAGRAM Consortium, MAGIC investigators, Groop L, Mootha VK, Daly MJ, Altshuler D . PLoS Genet 2010; 6: e1001058.
Lee PH, O’Dushlaine C, Thomas B, Purcell SM . Bioinformatics 2012; 28: 1797–1799.
The authors declare no conflict of interest.
Supplementary Information accompanies the paper on the Molecular Psychiatry website
Rights and permissions
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/
About this article
Cite this article
Robinson, E., Howrigan, D., Yang, J. et al. Response to ‘Predicting the diagnosis of autism spectrum disorder using gene pathway analysis’. Mol Psychiatry 19, 860–861 (2014). https://doi.org/10.1038/mp.2013.125
This article is cited by
Navigating the pitfalls of applying machine learning in genomics
Nature Reviews Genetics (2022)
Bio-collections in autism research
Molecular Autism (2017)
Advancing the understanding of autism disease mechanisms through genetics
Nature Medicine (2016)
Response to Robinson et al.
Molecular Psychiatry (2015)