The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements

Abstract

Over the last decade, the introduction of microarray technology has had a profound impact on gene expression research. The publication of studies with dissimilar or altogether contradictory results, obtained using different microarray platforms to analyze identical RNA samples, has raised concerns about the reliability of this technology. The MicroArray Quality Control (MAQC) project was initiated to address these concerns, as well as other performance and data analysis issues. Expression data on four titration pools from two distinct reference RNA samples were generated at multiple test sites using a variety of microarray-based and alternative technology platforms. Here we describe the experimental design and probe mapping efforts behind the MAQC project. We show intraplatform consistency across test sites as well as a high level of interplatform concordance in terms of genes identified as differentially expressed. This study provides a resource that represents an important first step toward establishing a framework for the use of microarrays in clinical and regulatory settings.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Repeatability of expression signal within test sites.
Figure 2: Signal variation within and between test sites.
Figure 3: Concordance of detection calls within and between test sites.
Figure 4: Agreement of gene lists.
Figure 5: Agreement of log ratios across platforms and test sites.
Figure 6: Correlation between microarray and TaqMan data.

Accession codes

Accessions

Gene Expression Omnibus

References

  1. 1

    Lesko, L.J. & Woodcock, J. Translation of pharmacogenomics and pharmacogenetics: a regulatory perspective. Nat. Rev. Drug Discov. 3, 763–769 (2004).

  2. 2

    Frueh, F.W. Impact of microarray data quality on genomic data submissions to the FDA. Nat. Biotechnol. 24, 1105–1107 (2006).

  3. 3

    Dix, D.J. et al. A framework for the use of genomics data at the EPA. Nat. Biotechnol. 24, 1108–1111 (2006).

  4. 4

    Tan, P.K. et al. Evaluation of gene expression measurements from commercial microarray platforms. Nucleic Acids Res. 31, 5676–5684 (2003).

  5. 5

    Ramalho-Santos, M., Yoon, S., Matsuzaki, Y., Mulligan, R.C. & Melton, D.A. “Stemness”: transcriptional profiling of embryonic and adult stem cells. Science 298, 597–600 (2002).

  6. 6

    Ivanova, N.B. et al. A stem cell molecular signature. Science 298, 601–604 (2002).

  7. 7

    Miller, R.M. et al. Dysregulation of gene expression in the 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine-lesioned mouse substantia nigra. J. Neurosci. 24, 7445–7454 (2004).

  8. 8

    Fortunel, N.O. et al. Comment on “'Stemness': transcriptional profiling of embryonic and adult stem cells” and “a stem cell molecular signature”. Science 302, 393 author reply 393 (2003).

  9. 9

    Miklos, G.L. & Maleszka, R. Microarray reality checks in the context of a complex disease. Nat. Biotechnol. 22, 615–621 (2004).

  10. 10

    Frantz, S. An array of problems. Nat. Rev. Drug Discov. 4, 362–363 (2005).

  11. 11

    Marshall, E. Getting the noise out of gene arrays. Science 306, 630–631 (2004).

  12. 12

    Michiels, S., Koscielny, S. & Hill, C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 365, 488–492 (2005).

  13. 13

    Ein-Dor, L., Zuk, O. & Domany, E. Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer. Proc. Natl. Acad. Sci. USA 103, 5923–5928 (2006).

  14. 14

    Petersen, D. et al. Three microarray platforms: an analysis of their concordance in profiling gene expression. BMC Genomics 6, 63 (2005).

  15. 15

    Dobbin, K.K. et al. Interlaboratory comparability study of cancer gene expression analysis using oligonucleotide microarrays. Clin. Cancer Res. 11, 565–572 (2005).

  16. 16

    Irizarry, R.A. et al. Multiple-laboratory comparison of microarray platforms. Nat. Methods 2, 345–350 (2005).

  17. 17

    Larkin, J.E., Frank, B.C., Gavras, H., Sultana, R. & Quackenbush, J. Independence and reproducibility across microarray platforms. Nat. Methods 2, 337–344 (2005).

  18. 18

    Kuo, W.P. et al. A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies. Nat. Biotechnol. 24, 832–840 (2006).

  19. 19

    Shi, L. et al. QA/QC: challenges and pitfalls facing the microarray community and regulatory agencies. Expert Rev. Mol. Diagn. 4, 761–777 (2004).

  20. 20

    Shi, L. et al. Cross-platform comparability of microarray technology: intra-platform consistency and appropriate data analysis procedures are essential. BMC Bioinformatics 6 Suppl. 2, S12 (2005).

  21. 21

    Ji, H. & Davis, R.W. Data quality in genomics and microarrays. Nat. Biotechnol. 24, 1112–1113 (2006).

  22. 22

    Canales, R.D. et al. Evaluation of DNA microarray results with quantitative gene expression platforms. Nat. Biotechnol. 24, 1115–1122 (2006).

  23. 23

    Shippy, R. et al. Using RNA sample titrations to assess microarray platform performance and normalization techniques. Nat. Biotechnol. 24, 1123–1131 (2006).

  24. 24

    Patterson, T.A. et al. Performance comparison of one-color and two-color platforms within the MicroArray Quality Control (MAQC) project. Nat. Biotechnol. 24, 1140–1150 (2006).

  25. 25

    Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006).

  26. 26

    Guo, L. et al. Rat toxicogenomic study reveals analytical consistency across microarray platforms. Nat. Biotechnol. 24, 1162–1169 (2006).

  27. 27

    Mecham, B.H. et al. Sequence-matched probes produce increased cross-platform consistency and more reproducible biological results in microarray-based gene expression measurements. Nucleic Acids Res. 32, e74 (2004).

  28. 28

    Carter, S.L., Eklund, A.C., Mecham, B.H., Kohane, I.S. & Szallasi, Z. Redefinition of Affymetrix probe sets by sequence overlap with cDNA microarray probes reduces cross-platform inconsistencies in cancer-associated gene expression measurements. BMC Bioinformatics 6, 107 (2005).

  29. 29

    Draghici, S., Khatri, P., Eklund, A.C. & Szallasi, Z. Reliability and reproducibility issues in DNA microarray measurements. Trends Genet. 22, 101–109 (2006).

  30. 30

    Irizarry, R.A., Wu, Z. & Jaffee, H.A. Comparison of Affymetrix GeneChip expression measures. Bioinformatics 22, 789–794 (2006).

  31. 31

    Pruitt, K.D., Tatusova, T. & Maglott, D.R. NCBI Reference Sequence (RefSeq): a curated non-redundant sequence database of genomes, transcripts and proteins. Nucleic Acids Res. 33, D501–D504 (2005).

  32. 32

    Thierry-Mieg, D. & J, T.-M. AceView: a comprehensive cDNA-supported gene and transcripts annotation. Genome Biology 7, Suppl. 1, S12 (2006).

  33. 33

    Bammler, T. et al. Standardizing global gene expression analysis between laboratories and across platforms. Nat. Methods 2, 351–356 (2005).

  34. 34

    Harr, B. & Schlotterer, C. Comparison of algorithms for the analysis of Affymetrix microarray data as evaluated by co-expression of genes in known operons. Nucleic Acids Res. 34, e8 (2006).

  35. 35

    Tusher, V.G., Tibshirani, R. & Chu, G. Significance analysis of microarrays applied to the ionizing radiation response. Proc. Natl. Acad. Sci. USA 98, 5116–5121 (2001).

  36. 36

    Thompson, K.L. et al. Use of a mixed tissue RNA design for performance assessments on multiple microarray formats. Nucleic Acids Res. 33, e187 (2005).

  37. 37

    Baker, S.C. et al. The External RNA Controls Consortium: a progress report. Nat. Methods 2, 731–734 (2005).

  38. 38

    Reid, L.H. The value of a proficiency testing program to monitor performance in microarray laboratories. Pharm. Discov. 5, 20–25 (2005).

  39. 39

    Ball, C.A. et al. Standards for microarray data. Science 298, 539 (2002).

  40. 40

    Tong, W. et al. ArrayTrack–supporting toxicogenomic research at the U.S. Food and Drug Administration National Center for Toxicological Research. Environ. Health Perspect. 111, 1819–1826 (2003).

  41. 41

    Tong, W. et al. Development of public toxicogenomics software for microarray data management and analysis. Mutat. Res. 549, 241–253 (2004).

Download references

Acknowledgements

All MAQC participants freely donated their time and reagents for the completion and analysis of the MAQC project. Participants from the National Institutes of Health (NIH) were supported by the Intramural Research Program of NIH, Bethesda, Maryland. D.H. thanks Ian Korf for BLAST discussions. This study utilized a number of computing resources, including the high-performance computational capabilities of the Biowulf PC/Linux cluster at the NIH (http://biowulf.nih.gov/) as well as resources at the analysis sites.

Author information

Correspondence to Leming Shi.

Ethics declarations

Competing interests

Many of the MAQC participants are employed by companies that manufacture gene expression products and/or perform testing services. J.C.W. is a consultant for and has significant financial interest in Gene Express, Inc.

Supplementary information

Supplementary Table 1

cRNA Size and Yields Collected from Microarray Platforms (XLS 45 kb)

Supplementary Table 2

Genes-to-NM Accession List for All Probes (TXT 2629 kb)

Supplementary Table 3

Most 3′ Probe-to-NM Accession List for 23,971 Common Probes (TXT 3272 kb)

Supplementary Table 4

One Probe-to-One Gene List for All Genes (TXT 1718 kb)

Supplementary Table 5

One Probe-to-One Gene List for 12,091 Common Genes (TXT 1211 kb)

Supplementary Methods

BLAST Analysis for the MAQC Project (PDF 67 kb)

Supplementary Notes

README for Probe Mapping Files (PDF 11 kb)

Supplementary Data

Supplementary Information on MAQC Study (PDF 663 kb)

Rights and permissions

Reprints and Permissions

About this article

Further reading