Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Opinion
  • Published:

Rules of evidence for cancer molecular-marker discovery and validation

Nature Reviews Cancer volume 4, pages 309–314 (2004)Cite this article

Abstract

According to some claims, molecular markers are set to revolutionize the process of evaluating prognosis and diagnosis for cancer. Research about cancer markers has, however, been characterized by inflated expectations, followed by disappointment when original results can not be reproduced. Even now, disappointment might be expected, in part because rules of evidence to assess the validity of studies about diagnosis and prognosis are both underdeveloped and not routinely applied. What challenges are involved in assessing studies and how might problems be avoided so as to realize the full potential of this emerging technology?

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Method of dividing original sample to assess reproducibility and overfitting.

Similar content being viewed by others

References

  1. Ramaswamy, S. & Perou, C. M. DNA microarrays in breast cancer: the promise of personalised medicine. Lancet 361, 1576–1577 (2003).

    PubMed  Google Scholar 

  2. Kolata, G. Breast cancer: genes are tied to death rates. New York Times A1 (December 19, 2002).

  3. Zhu, W. et al. Detection of cancer-specific markers amid massive mass spectral data. Proc. Natl Acad. Sci. USA 100, 14666–14671 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. US Preventive Services Task Force. Guide to clinical preventive services 2nd edn Ch. 2 (US Government Prinitng Office, 1996).

  5. Woolf, S. H. Practice guidelines, a new reality in medicine. II. Methods of developing guidelines. Arch. Intern. Med. 152, 946–952 (1992).

    Article  CAS  PubMed  Google Scholar 

  6. Tannock, I. F. & Warr, D. G. Unconventional therapies for cancer: a refuge from the rules of evidence? CMAJ 159, 801–802 (1998).

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Vogelstein, B. et al. Genetic alterations during colorectal-tumor development. N. Engl J. Med. 319, 525–532 (1988).

    Article  CAS  PubMed  Google Scholar 

  8. Ahlquist, D. A. et al. Colorectal cancer screening by detection of altered human DNA in stool: feasibility of a multitarget assay panel. Gastroenterology 119, 1219–1227 (2000).

    Article  CAS  PubMed  Google Scholar 

  9. Stears, R. L., Martinsky, T. & Schena, M. Trends in microarray analysis. Nature Med. 9, 140–145 (2003).

    Article  CAS  PubMed  Google Scholar 

  10. Ransohoff, D. F. Developing molecular biomarkers for cancer. Science 299, 1679–1680 (2003).

    Article  CAS  PubMed  Google Scholar 

  11. Thomson, D. M., Krupey, J., Freedman, S. O. & Gold, P. The radioimmunoassay of circulating carcinoembryonic antigen of the human digestive system. Proc. Natl Acad. Sci. USA 64, 161–167 (1969).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Reid, M. C., Lachs, M. S. & Feinstein, A. R. Use of methodological standards in diagnostic test research. Getting better but still not good. JAMA 274, 645–651 (1995).

    Article  CAS  PubMed  Google Scholar 

  13. Ransohoff, D. F. & Feinstein, A. R. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N. Engl. J. Med. 299, 926–930 (1978).

    Article  CAS  PubMed  Google Scholar 

  14. Sackett, D. L. Zlinkoff honor lecture: basic research, clinical research, clinical epidemiology, and general internal medicine. J. Gen. Intern. Med. 2, 40–47 (1987).

    Article  CAS  PubMed  Google Scholar 

  15. Feinstein, A. R. Clinical biostatistics XXXI. On the sensitivity, specificity, and discrimination of diagnostic tests. Clin. Pharmacol. Ther. 17, 104–116 (1975).

    Article  CAS  PubMed  Google Scholar 

  16. Ransohoff, D. F. Challenges and opportunities in evaluating diagnostic tests. J. Clin. Epid. 55, 1178–1182 (2002).

    Article  Google Scholar 

  17. Sullivan Pepe, M. et al. Phases of biomarker development for early detection of cancer. J. Natl Cancer Inst. 93, 1054–1061 (2001).

    Article  Google Scholar 

  18. Sackett, D. L., Haynes, R. B., Tugwell, P. & Guyatt, G. H. Clinical Epidemiology: a Basic Science for Clinical Medicine (Little, Brown and Company, Boston, 1991).

    Google Scholar 

  19. Bogardus, S. T., Concato, J. & Feinstein, A. R. Clinical epidemiological quality in molecular genetic research: the need for methodological standards. JAMA 281, 1919–1926 (1999).

    Article  PubMed  Google Scholar 

  20. Deyo, R. A. & Jarvik, J. J. New diagnostic tests: breakthrough approaches or expensive add-ons? Ann. Intern. Med. 139, 950–951 (2003).

    Article  PubMed  Google Scholar 

  21. Simon, R. & Altman, D. G. Statistical aspects of prognostic factor studies in oncology. Br. J. Cancer 69, 979–985 (1994).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Wasson, J. H., Sox, H. C., Neff, R. K. & Goldman, L. Clinical prediction rules. Applications and methodological standards. N. Engl. J. Med. 313, 793–799 (1985).

    Article  CAS  PubMed  Google Scholar 

  23. Lachs, M. S. et al. Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection. Ann. Intern. Med. 117, 135–140 (1992).

    Article  CAS  PubMed  Google Scholar 

  24. Jaeschke, R., Guyatt, G. & Sackett, D. L. Users' guides to the medical literature. III. How to use an article about a diagnostic test. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA 271, 389–391 (1994).

    Article  CAS  PubMed  Google Scholar 

  25. Sackett, D. L. & Haynes, R. B. The architecture of diagnostic research. BMJ 324, 539–541 (2002).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Bossuyt, P. M. et al. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann. Intern. Med. 138, 40–44 (2003).

    Article  PubMed  Google Scholar 

  27. Bossuyt, P. M. et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann. Intern. Med. 138, W1–W12 (2003).

    Article  PubMed  Google Scholar 

  28. Potter, J. D. At the interfaces of epidemiology, genetics and genomics. Nature Rev. Genet. 2, 142–147 (2001).

    Article  CAS  PubMed  Google Scholar 

  29. Ambroise, C. & McLachlan, G. J. Selection bias in gene extraction on the basis of microarray gene-expression data. Proc. Natl Acad. Sci. USA 99, 6562–6566 (2002).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Simon, R., Radmacher, M. D., Dobbin, K. & McShane, L. M. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J. Natl Cancer Inst. 95, 14–18 (2003).

    Article  CAS  PubMed  Google Scholar 

  31. Katz, M. H. Multivariable analysis: a primer for readers of medical research. Ann. Intern. Med. 138, 644–650 (2003).

    Article  PubMed  Google Scholar 

  32. Selaru, F. M. et al. Artificial neural networks distinguish among subtypes of neoplastic colorectal lesions. Gastroenterology 122, 606–613 (2002).

    Article  PubMed  Google Scholar 

  33. Petricoin, E. F. et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet 359, 572–577 (2002).

    Article  CAS  PubMed  Google Scholar 

  34. Qu, Y. et al. Boosted decision tree analysis of surface-enhanced laser desorption/ionization mass spectral serum profiles discriminates prostate cancer from noncancer patients. Clin. Chem. 48, 1835–1843 (2002).

    CAS  PubMed  Google Scholar 

  35. Huang, E. et al. Gene expression predictors of breast cancer outcomes. Lancet 361, 1590–1596 (2003).

    Article  CAS  PubMed  Google Scholar 

  36. Harrell, F. E. Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, And Survival Analysis (Springer–Verlag, New York, 2001).

    Book  Google Scholar 

  37. Ntzani, E. E. & Ioannidis, J. P. Predictive ability of DNA microarrays for cancer outcomes and correlates: an empirical assessment. Lancet 362, 1439–1444 (2003).

    Article  CAS  PubMed  Google Scholar 

  38. van de Vijver, M. J. et al. A gene-expression signature as a predictor of survival in breast cancer. N. Engl. J. Med. 347, 1999–2009 (2002).

    Article  CAS  PubMed  Google Scholar 

  39. van 't Veer, L. J. et al. Gene expression profiling predicts clinical outcome of breast cancer. Nature 415, 530–536 (2002).

    Article  CAS  PubMed  Google Scholar 

  40. Ransohoff, D. F. Gene-expression signatures in breast cancer. N. Engl. J. Med. 348, 1715–1717 (2003).

    Article  PubMed  Google Scholar 

  41. Baker, S. G., Kramer, B. S. & Srivastava, S. Markers for early detection of cancer: statistical guidelines for nested case–control studies. BMC Med. Res. Methodol. 2, 4 (2002).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Chang, J. C. et al. Gene expression profiling for the prediction of therapeutic response to docetaxel in patients with breast cancer. Lancet 362, 362–369 (2003).

    Article  CAS  PubMed  Google Scholar 

  43. Brenton, J. D. & Caldas, C. Predictive cancer genomics: what do we need? Lancet 362, 340–341 (2003).

    Article  PubMed  Google Scholar 

  44. Rosenwald, A. et al. The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma. N. Engl. J. Med. 346, 1937–1947 (2002).

    Article  PubMed  Google Scholar 

  45. Hunter, K. W. Allelic diversity in the host genetic background may be an important determinant in tumor metastatic dissemination. Cancer Lett. 200, 97–105 (2003).

    Article  CAS  PubMed  Google Scholar 

  46. Masters, J. R. & Lakhani, S. R. How diagnosis with microarrays can help cancer patients. Nature 404, 921 (2000).

    Article  CAS  PubMed  Google Scholar 

  47. Pharmalicensing. Agenda to develop microarray-based breast cancer test using Agilent Technologies' gene expression platform [online], (cited 22 Sept. 2003) (2003).

  48. Wooster, R. & Weber, B. L. Breast and ovarian cancer. N. Engl. J. Med. 348, 2339–2347 (2003).

    Article  CAS  PubMed  Google Scholar 

  49. Feinstein, A. R. Multivariable Analysis: An Introduction (Yale University Press, New Haven, 1996).

    Book  Google Scholar 

  50. Fletcher, R. H., Fletcher, S. W. & Wagner, E. H. Clinical Epidemiology: The Essentials 3rd edn (Williams & Wilkins, Baltimore, 1996).

    Google Scholar 

  51. Bleeker, S. E. et al. External validation is necessary in prediction research: a clinical example. J. Clin. Epidemiol. 56, 826–832 (2003).

    Article  CAS  PubMed  Google Scholar 

  52. Sorace, J. M. & Zhan, M. A data review and re-assessment of ovarian cancer serum proteomic profiling. BMC Bioinformatics 4, 24 (2003).

    Article  PubMed  PubMed Central  Google Scholar 

  53. Baggerly, K. A., Morris, J. S. & Coombes, K. R. Reproducibility of SELDI–TOF protein patterns in serum: comparing data sets from different experiments. Bioinformatics. 29 Jan 2004. (doi:10.1093/bioinformatics/btg484)

  54. Hingorani, S. R. et al. Preinvasive and invasive ductal pancreatic cancer and its early detection in the mouse. Cancer Cell 4, 437–450 (2003).

    Article  CAS  PubMed  Google Scholar 

  55. Feinstein, A. R. Clinical Epidemiology: The Architecture of Clinical Research (WB Saunders, Philadelphia, 1985).

    Google Scholar 

  56. Hennekens, C. H. & Buring, J. E. Epidemiology in Medicine (Little, Brown and Company, Boston, 1987).

    Google Scholar 

  57. Freiman, J. A., Chalmers, T. C., Smith, H. Jr & Kuebler, R. R. The importance of β, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 'negative' trials. N. Engl. J. Med. 299, 690–694 (1978).

    Article  CAS  PubMed  Google Scholar 

  58. Ransohoff, D. F. Discovery-based research and fishing. Gastroenterology 125, 290 (2003).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Thanks to many colleagues at the National Cancer Institute, The University of North Carolina at Chapel Hill and elsewhere for reviewing and commenting on earlier versions of the manuscript.

Author information

Authors and Affiliations

  1. Departments of Medicine and Epidemiology, University of North Carolina at Chapel Hill, CB# 7080, Bioinformatics Bldg. 4103, Chapel Hill, 27599-7080, North Carolina, USA

    David F. Ransohoff

  2. Division of Cancer Prevention, National Cancer Institute, National Institutes of Health, Bethesda, 20892–7354, Maryland, USA

    David F. Ransohoff

Authors
  1. David F. Ransohoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Ethics declarations

Competing interests

The author declares no competing financial interests.

Related links

Related links

DATABASES

Cancer.gov

breast cancer

colorectal cancer

ovarian cancer

pancreatic cancer

FURTHER INFORMATION

Research about cancer molecular markers

Glossary

CROSS-VALIDATION

A technique used in multivariable analysis that is intended to reduce the possibility of overfitting and of non-reproducible results. The method involves sequentially leaving out parts of the original sample ('split-sample') and conducting a multivariable analysis; the process is repeated until the entire sample has been assessed. The results are combined into a final model that is the product of the training step.

DISCOVERY-BASED RESEARCH

Research in which large amounts of data are examined, without prior hypothesis, to discover markers or patterns that might discriminate among groups of individuals.

HIGH-THROUGHPUT ANALYSIS

Research in which large numbers of variables are analysed simultaneously. RNA expression analysis using microarrays simultaneously examines expression levels of tens of thousands of genes. Proteomic analysis of serum using mass spectroscopy simultaneously examines thousands of peaks related to proteins and peptides.

MICROARRAY

A solid surface on which thousands of specimens, such as synthetic oligonucleotides representing different genes, can be placed in separate locations and used to assess the status of genotype or gene expression for one individual.

MULTIVARIABLE MODELS

Models that simultaneously consider how multiple variables — such as age, gender, co-morbidity, symptoms and gene expression — relate to an outcome such as diagnosis or prognosis.

OVERFITTING

Finding a discriminatory pattern by chance, which can happen when large numbers of variables are assessed for a small number of outcomes.

POLYMERASE CHAIN REACTION

(PCR). A method to replicate or amplify small amounts of DNA into larger amounts that can be used in chemical analysis.

RULES OF EVIDENCE

Rules that are used to evaluate the strength or validity of research results by considering problems such as heterogeneity, complexity, bias and 'generalizeability'. Rules vary depending on the subject or purpose of the study: diagnosis, prognosis, therapy or aetiology.

SERIAL ANALYSIS OF GENE EXPRESSION

(SAGE). A method to estimate numbers of copies of genes.

SINGLE-NULEOTIDE POLYMORPHISM

(SNP). Variations involving a single base.

SPLIT-SAMPLE VALIDATION

Split sample validation is used, confusingly, to mean two different things. It can refer to the method in the training step by which the sample is divided during the process of cross-validation. It can also refer to the method used to divide the original sample of subjects into two groups for use in training and then in independent validation.

VALIDITY

Refers in general to efforts that are made to confirm the accuracy, precision or effectiveness of results, including reproducibility.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ransohoff, D. Rules of evidence for cancer molecular-marker discovery and validation. Nat Rev Cancer 4, 309–314 (2004). https://doi.org/10.1038/nrc1322

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrc1322

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing