Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Bias as a threat to the validity of cancer molecular-marker research

Abstract

Claims that molecular markers can accurately diagnose cancer have recently been disputed; some prominent results have not been reproduced and bias has been proposed to explain the original observations. As new '-omics' fields are explored to assess molecular markers for cancer, bias will increasingly be recognized as the most important 'threat to validity' that must be addressed in the design, conduct and interpretation of such research.

This is a preview of subscription content

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Figure 1: Research structure of an experiment.
Figure 2: Checking to see whether randomization was successful.
Figure 3: Guidelines for reporting studies of diagnostic accuracy.

References

  1. Ransohoff, D. F. Developing molecular biomarkers for cancer. Science 299, 1679–1680 (2003).

    CAS  Article  PubMed  Google Scholar 

  2. Stears, R. L., Martinsky, T. & Schena, M. Trends in microarray analysis. Nature Med. 9, 140–145 (2003).

    CAS  Article  PubMed  Google Scholar 

  3. Petricoin, E. F. et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet 359, 572–577 (2002).

    CAS  Article  PubMed  Google Scholar 

  4. Zhu, W. et al. Detection of cancer-specific markers amid massive mass spectral data. Proc. Natl Acad. Sci. USA 100, 14666–14671 (2003).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  5. Li, J., Zhang, Z., Rosenzweig, J., Wang, Y. Y. & Chan, D. W. Proteomics and bioinformatics approaches for identification of serum biomarkers to detect breast cancer. Clin. Chem. 48, 1296–1304 (2002).

    CAS  PubMed  Google Scholar 

  6. Adam, B. L. et al. Serum protein fingerprinting coupled with a pattern-matching algorithm distinguishes prostate cancer from benign prostate hyperplasia and healthy men. Cancer Res. 62, 3609–3614 (2002).

    CAS  PubMed  Google Scholar 

  7. Petricoin, E. F. et al. Serum proteomic patterns for detection of prostate cancer. J. Natl Cancer Inst. 94, 1576–1578 (2002).

    CAS  Article  PubMed  Google Scholar 

  8. FDA Press Office. Protein patterns may identify ovarian cancer [online], <http://www.fda.gov/bbs/topics/NEWS/2002/NEW00797.html> (2002).

  9. Marcus, A. Testing for ovarian cancer is on the way. Wall Street Journal (New York) D1, D2 (1 Oct 2002).

    Google Scholar 

  10. Pollack, A. New cancer test stirs hope and concern. New York Times (New York) D1, D6 (3 Feb 2004).

    Google Scholar 

  11. Food and Drug Administration. Letter to correlogic systems, Inc. [online], <http://www.fda.gov/cdrh/oivd/letters/021804-correlogic.html> (2004).

  12. Food and Drug Administration. Letter to laboratory corporation of America [online], <http://www.fda.gov/cdrh/oivd/letters/030204-labcorp.html> (2004).

  13. Food and Drug Administration. Letter to quest diagnostics [online], <http://www.fda.gov/cdrh/oivd/letters/030204-quest.html> (2004).

  14. Society of Gynecologic Oncologists. Society of gynecologic oncologists statement regarding OvaCheck [online], <http://www.sgo.org/images/pdfs/policy/OvaCheck_statement.pdf> (2004).

  15. Wagner, L. A test before its time? FDA stalls distribution process of proteomic test. J. Natl Cancer Inst. 96, 500–501 (2004).

    Article  PubMed  Google Scholar 

  16. Garber, K. Debate rages over proteomic patterns. J. Natl Cancer Inst. 96, 816–818 (2004).

    Article  PubMed  Google Scholar 

  17. Check, E. Running before we can walk? Nature 429, 496–497 (2004).

    CAS  Article  PubMed  Google Scholar 

  18. Diamandis, E. Analysis of serum proteomic patterns for early cancer diagnosis: drawing attention to potential problems. J. Natl Cancer Inst. 96, 353–356 (2004).

    Article  PubMed  Google Scholar 

  19. Baggerly, K. A., Morris, J. S. & Coombes, K. R. Reproducibility of SELDI-TOF protein patterns in serum: comparing datasets from different experiments. Bioinformatics 20, 777–785 (2004).

    CAS  Article  PubMed  Google Scholar 

  20. Sorace, J. M. & Zhan, M. A data review and re-assessment of ovarian cancer serum proteomic profiling. BMC Bioinformatics 4, 24 (2003).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Baggerly, K. A., Coombes, K. R. & Morris, J. S. Are the NCI/FDA ovarian proteomic data biased? A reply to 'producers and consumers'. Cancer Informatics (in the press).

  22. Baggerly, K. A., Edmonson, S. R., Morris, J. S. & Coombes, K. R. High-resolution serum proteomic patterns for ovarian cancer detection. Endo. Relat. Cancer 11, 583–584 (2004).

    CAS  Article  Google Scholar 

  23. van de Vijver, M. J. et al. A gene-expression signature as a predictor of survival in breast cancer. N. Engl. J. Med. 347, 1999–2009 (2002).

    CAS  Article  PubMed  Google Scholar 

  24. Huang, E. et al. Gene expression predictors of breast cancer outcomes. Lancet 361, 1590–1596 (2003).

    CAS  Article  PubMed  Google Scholar 

  25. Ramaswamy, S. & Perou, C. M. DNA microarrays in breast cancer: the promise of personalised medicine. Lancet 361, 1576–1577 (2003).

    Article  PubMed  Google Scholar 

  26. Ransohoff, D. F. Gene-expression signatures in breast cancer. N. Engl. J. Med. 348, 1715–1717; author reply 1715–1717 (2003).

    Article  PubMed  Google Scholar 

  27. Marshall, E. Getting the noise out of gene arrays. Science 306, 630–631 (2004).

    CAS  Article  PubMed  Google Scholar 

  28. Ambroise, C. & McLachlan, G. J. Selection bias in gene extraction on the basis of microarray gene-expression data. Proc. Natl Acad. Sci. USA 99, 6562–6566 (2002).

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  29. Baggerly, K. A. et al. A comprehensive approach to the analysis of matrix-assisted laser desorption/ionization-time of flight proteomics spectra from serum samples. Proteomics 3, 1667–1672 (2003).

    CAS  Article  PubMed  Google Scholar 

  30. Mehta, T., Tanik, M. & Allison, D. B. Towards sound epistemological foundations of statistical methods for high-dimensional biology. Nature Genet. 36, 943–947 (2004).

    CAS  Article  PubMed  Google Scholar 

  31. Ransohoff, D. F. Rules of evidence for cancer molecular-marker discovery and validation. Nature Rev. Cancer 4, 309–314 (2004).

    CAS  Article  Google Scholar 

  32. Ransohoff, D. F. Discovery-based research and fishing. Gastroenterology 125, 290 (2003).

    Article  PubMed  Google Scholar 

  33. Hulley, S. B. et al. Designing Clinical Research: An Epidemiologic Approach (Lippincott Williams & Wilkins, Philadelphia, 2001).

    Google Scholar 

  34. Rothman, K. J. Modern Epidemiology (Little, Brown and Company, Boston/Toronto, 1986).

    Google Scholar 

  35. Hennekens, C. H. & Buring, J. E. Epidemiology in Medicine (Little, Brown and Company, Boston, 1987).

    Google Scholar 

  36. Taubes, G. Epidemiology faces its limits. Science 269, 164–169 (1995).

    CAS  Article  PubMed  Google Scholar 

  37. Ransohoff, D. F. Research opportunity at the interface of molecular biology and clinical epidemiology. Gastroenterology 122, 1199 (2002).

    Article  Google Scholar 

  38. Ransohoff, D. F. Evaluating discovery-based research: when biologic reasoning cannot work. Gastroenterology 127, 1028 (2004).

    Article  PubMed  Google Scholar 

  39. Coombes, R. C. et al. A randomized trial of exemestane after two to three years of tamoxifen therapy in postmenopausal women with primary breast cancer. N. Engl. J. Med. 350, 1081–1092 (2004).

    CAS  Article  PubMed  Google Scholar 

  40. Altman, D. G. et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann. Intern. Med. 134, 663–694 (2001).

    CAS  Article  PubMed  Google Scholar 

  41. The case-control study: consensus and controversy. J. Chronic. Dis. 32, 1–144 (1979).

  42. Sackett, D. L. Bias in analytic research. J. Chronic. Dis. 32, 51–63 (1979).

    CAS  Article  PubMed  Google Scholar 

  43. Roos, N. P. et al. Mortality and reoperation after open and transurethral resection of the prostate for benign prostatic hyperplasia. N. Engl. J. Med. 320, 1120–1124 (1989).

    CAS  Article  PubMed  Google Scholar 

  44. Greenfield, S. The state of outcome research: are we on target? N. Engl. J. Med. 320, 1142–1143 (1989).

    CAS  Article  PubMed  Google Scholar 

  45. Concato, J., Horwitz, R. I., Feinstein, A. R., Elmore, J. G. & Schiff, S. F. Problems of comorbidity in mortality after prostatectomy. JAMA 267, 1077–1082 (1992).

    CAS  Article  PubMed  Google Scholar 

  46. Iezzoni, L. I. et al. Comorbidities, complications, and coding bias. Does the number of diagnosis codes matter in predicting in-hospital mortality? JAMA 267, 2197–2203 (1992).

    CAS  Article  PubMed  Google Scholar 

  47. Iezzoni, L. I. The risks of risk adjustment. JAMA 278, 1600–1607 (1997).

    CAS  Article  PubMed  Google Scholar 

  48. Hulley, S. et al. Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. Heart and Estrogen/progestin Replacement Study (HERS) Research Group. JAMA 280, 605–613 (1998).

    CAS  Article  PubMed  Google Scholar 

  49. Manson, J. E. et al. Estrogen plus progestin and the risk of coronary heart disease. N. Engl. J. Med. 349, 523–534 (2003).

    CAS  Article  PubMed  Google Scholar 

  50. Col, N. F. & Pauker, S. G. The discrepancy between observational studies and randomized trials of menopausal hormone therapy: did expectations shape experience? Ann. Intern. Med. 139, 923–929 (2003).

    Article  PubMed  Google Scholar 

  51. Rennie, D. How to report randomized controlled trials. The CONSORT statement. JAMA 276, 649 (1996).

    CAS  Article  PubMed  Google Scholar 

  52. Bossuyt, P. M. et al. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann. Intern. Med. 138, 40–44 (2003).

    Article  PubMed  Google Scholar 

  53. Bossuyt, P. M. et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann. Intern. Med. 138, W1–12 (2003).

    Article  PubMed  Google Scholar 

  54. Brazma, A. et al. Minimum information about a microarray experiment (MIAME)—toward standards for microarray data. Nature Genet. 29, 365–371 (2001).

    CAS  Article  PubMed  Google Scholar 

  55. Ransohoff, D. F. Lessons from controversy: ovarian cancer screening and serum proteomics. J. Natl Cancer Inst. (in the press).

  56. Imperiale, T. F. et al. Fecal DNA versus fecal occult blood for colorectal-cancer screening in an average-risk population. N. Engl. J. Med. 351, 2704–2714 (2004).

    CAS  Article  PubMed  Google Scholar 

  57. Prorok, P. C. et al. Design of the Prostate, Lung, Colorectal and Ovarian (PLCO) cancer screening trial. Control Clin. Trials 21, 273S–309S (2000).

    CAS  Article  PubMed  Google Scholar 

  58. Simon, R., Radmacher, M. D., Dobbin, K. & McShane, L. M. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J. Natl Cancer Inst. 95, 14–18 (2003).

    CAS  Article  PubMed  Google Scholar 

  59. Ntzani, E. E. & Ioannidis, J. P. Predictive ability of DNA microarrays for cancer outcomes and correlates: an empirical assessment. Lancet 362, 1439–1444 (2003).

    CAS  Article  PubMed  Google Scholar 

  60. Friedman, L. M., Furberg, C. D. & De Mets, D. L. Fundamentals of Clinical Trials, 3rd edn (Springer, New York, 1998).

    Book  Google Scholar 

  61. Sullivan Pepe, M. et al. Phases of biomarker development for early detection of cancer. J. Natl Cancer Inst. 93, 1054–1061 (2001).

    Article  Google Scholar 

Download references

Acknowledgements

Thanks to colleagues at the University of North Carolina at Chapel Hill, the National Cancer Institute and elsewhere for reviewing and commenting on earlier versions of the manuscript. Many ideas were developed through participation in activities of the Early Detection Research Network.

Author information

Authors and Affiliations

Authors

Ethics declarations

Competing interests

The author declares no competing financial interests.

Related links

Related links

DATABASES

National Cancer Institute

breast cancer

colorectal cancer

ovarian cancer

prostate cancer

FURTHER INFORMATION

Early Detection Research Network

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ransohoff, D. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer 5, 142–149 (2005). https://doi.org/10.1038/nrc1550

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrc1550

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing