Common pitfalls in preclinical cancer target validation

Journal name:
Nature Reviews Cancer
Volume:
17,
Pages:
425–440
Year published:
DOI:
doi:10.1038/nrc.2017.32
Published online

Abstract

An alarming number of papers from laboratories nominating new cancer drug targets contain findings that cannot be reproduced by others or are simply not robust enough to justify drug discovery efforts. This problem probably has many causes, including an underappreciation of the danger of being misled by off-target effects when using pharmacological or genetic perturbants in complex biological assays. This danger is particularly acute when, as is often the case in cancer pharmacology, the biological phenotype being measured is a 'down' readout (such as decreased proliferation, decreased viability or decreased tumour growth) that could simply reflect a nonspecific loss of cellular fitness. These problems are compounded by multiple hypothesis testing, such as when candidate targets emerge from high-throughput screens that interrogate multiple targets in parallel, and by a publication and promotion system that preferentially rewards positive findings. In this Perspective, I outline some of the common pitfalls in preclinical cancer target identification and some potential approaches to mitigate them.

At a glance

Figures

  1. Distinguishing correlation from causation.
    Figure 1: Distinguishing correlation from causation.

    a | Possible reasons that two variables, A and B, correlate with one another. b | Definitions of necessity and sufficiency.

  2. Drawing erroneous causal inferences from Kaplan-Meier curves.
    Figure 2: Drawing erroneous causal inferences from Kaplan–Meier curves.

    a | Hypothetical Kaplan–Meier curve for patients with chronic lung disease. b | Hypothetical Kaplan–Meier curve for patients with cancer. c | The association between hypoxia and hypoxia-inducible factor (HIF) and poor survival depicted in panel b could, in a non-mutually exclusive manner, reflect the ability of hypoxia and HIF to cause aggressive tumour growth or the fact that aggressive tumours are likely to outgrow their blood supplies, become hypoxic and induce HIF. It is also possible that hypoxia causes aggressive tumour growth, but HIF does not. d | Hypothetical Kaplan–Meier curve for patients with breast cancer. ER, oestrogen receptor.

  3. Up assays outperform down assays.
    Figure 3: Up assays outperform down assays.

    Up assays typically outperform down assays for two reasons: a better signal to noise ratio and fewer false positives. The former relates to the ability to see a positive or increased signal (asterisk) in a field of negative or decreased signals, respectively. The latter stems from the fact that there are more uninteresting or trivial ways to make a complex system perform worse than there are ways to make it perform better. Many pharmacodynamic assays in cancer pharmacology are based on loss of an analyte (for example, a phosphoepitope) and hence are down assays. An analyte that is rapidly induced upon target engagement could form the basis for a pharmacodynamic up assay. Likewise, most chemical and genetic screens in cancer biology are based on down assays, but one can envision screens based on up assays. For example, one could screen for inhibitors of a genetically validated oncogenic pathway in reporter cells in which that pathway actually suppresses cell fitness, owing to naturally occurring or engineered differences between those reporter cells and cells in which that pathway normally promotes tumorigenesis. The former approach might exploit the fact that some oncogenic pathways actually suppress cell growth in certain cellular contexts and the latter might entail cells that have been molecularly engineered to die in response to specific oncogenic signals.

  4. Commonly used down assays in cancer biology and cancer pharmacology.
    Figure 4: Commonly used down assays in cancer biology and cancer pharmacology.

    a | Pharmacodynamic assay based on loss of an analyte (for example, a protein or phosphoepitope). Note that if the analyte has a shorter half-life than the normalization control (as is often the case), then the analyte will seem to specifically disappear in response to agents that are nonspecifically toxic and cause a global decrease in transcription or translation. b | Proliferation assays. c | Cell viability assays. d |Ex vivo angiogenesis assays. e |Ex vivo invasion assays. f |In vivo tumour assays such as subcutaneous xenograft assays. g |In vivo metastasis assays, such as those carried out after tail vein injection of tumour cells.

  5. On-target versus off-target effects.
    Figure 5: On-target versus off-target effects.

    a | Perturbant (for example, drug, siRNA, shRNA or single guide RNA (sgRNA))-induced phenotype is on-target when it is caused by engagement of its intended target. b | Perturbant-induced phenotype is off-target when it is caused by engagement of an unintended target. c | Sometimes the phenotype induced by a perturbant reflects and requires engagement of its intended target and at least one unintended target. Note that in this scenario the phenotype can be reversed (rescued) by mitigating the effect of the perturbant on its unintended target. d | Sometimes a phenotype induced by a perturbant reflects and requires engagement of its intended target or at least one unintended target. Note that in this scenario the phenotype cannot necessarily be reversed (rescued) by mitigating the effect of the perturbant on its unintended target.

  6. Distinguishing on-target versus off-target effects with rescue experiments.
    Figure 6: Distinguishing on-target versus off-target effects with rescue experiments.

    a | The challenge with a phenotype caused by a perturbant (for example, drug, siRNA, shRNA or single guide RNA (sgRNA)) is to determine whether that phenotype is on-target, off-target or both. This is especially important when the phenotype in question could simply reflect a loss of fitness stemming from a nonspecific toxic effect. b | The classic way to address this is to ask whether the perturbant still induces that phenotype in the face of a target that is resistant to that perturbant. c | An alternative approach, which requires prior knowledge of the target's function and downstream activities, is to ask whether the perturbant still induces the phenotype if the downstream effector function of the target is maintained.

  7. Designing perturbant-resistant targets for rescue experiments.
    Figure 7: Designing perturbant-resistant targets for rescue experiments.

    For a drug, a version of its target (usually a protein) is discovered or designed that retains its biochemical activity (such as its enzymatic activity in the case of an enzyme) but is drug resistant because of a mutation that decreases drug binding. For siRNAs and shRNAs an expression vector for the target is introduced that encodes an mRNA that lacks the siRNA- or shRNA-binding site entirely, as can be done for siRNA- or shRNA-binding sites located in untranslated regions (UTRs), or the siRNA- or shRNA-binding site is crippled by the introduction of three or more synonymous mutations. For single guide RNAs (sgRNAs) one introduces an expression vector for the target that encodes an mRNA that partially or completely lacks the sgRNA-binding site (including the protospacer adjacent motif (PAM) site), as can be done for sgRNA-binding sites located at intron–exon boundaries, or by crippling the sgRNA binding, such as by introducing a synonymous mutation into the PAM site. ORF, open reading frame.

  8. Generating drug-resistant protein targets.
    Figure 8: Generating drug-resistant protein targets.

    a | A mammalian expression vector encoding the protein target of interest is first propagated in error-prone Escherichia coli, such as the strain XL1-red, and then introduced into a cell line that is sensitive to the drug. After drug exposure, drug-resistant clones are pooled, genomic DNA is isolated and the cDNA is amplified by PCR and sequenced using next-generation sequencing methods. Putative drug-resistance mutations are reintroduced into a wild-type target cDNA and tested for their ability to confer drug resistance to the target in biochemical assays and to confer drug resistance to cells expressing the mutant target. b | A mismatch repair-deficient cell line that is sensitive to the drug of interest is treated with that drug. Surviving cells are cloned, which are then analysed by RNA-seq and whole-exome sequencing for somatic mutations that recur among multiple independent clones. Performing multiple biological replicates or introducing a DNA bar code library before drug exposure can assist in identifying mutations that recur because of founder effects in the original population. AmpR, ampicillin resistance gene; ORF, open reading frame.

References

  1. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712 (2011).
  2. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531533 (2012).
  3. Frye, S. V. et al. Tackling reproducibility in academic preclinical drug discovery. Nat. Rev. Drug Discov. 14, 733734 (2015).
  4. Nosek, B. A. & Errington, T. M. Making sense of replications. eLife 6, e23383 (2017).
  5. Errington, T. M. et al. An open investigation of the reproducibility of cancer biology research. eLife 3, e04333 (2014).
  6. Flaherty, K. T. et al. Inhibition of mutated, activated BRAF in metastatic melanoma. N. Engl. J. Med. 363, 809819 (2010).
  7. Harris, C. C. p53 tumor suppressor gene: from the basis research laboratory to the clinic — an abridged historical perspective. Carcinogenesis 17, 11871198 (1996).
  8. Semenza, G. L. Defining the role of hypoxia-inducible factor 1 in cancer biology and therapeutics. Oncogene 29, 625634 (2010).
  9. Mellor, H. R. & Harris, A. L. The role of the hypoxia-inducible BH3-only proteins BNIP3 and BNIP3L in cancer. Cancer Metastasis Rev. 26, 553566 (2007).
  10. Reiling, J. H. & Hafen, E. The hypoxia-induced paralogs Scylla and Charybdis inhibit growth by down-regulating S6K activity upstream of TSC in Drosophila. Genes Dev. 18, 28792892 (2004).
  11. Brugarolas, J. et al. Regulation of mTOR function in response to hypoxia by REDD1 and the TSC1/TSC2 tumor suppressor complex. Genes Dev. 18, 28932904 (2004).
  12. Keith, B., Johnson, R. S. & Simon, M. C. HIF1alpha and HIF2alpha: sibling rivalry in hypoxic tumour growth and progression. Nat. Rev. Cancer 12, 922 (2012).
  13. Lin, A., Giuliano, C. J., Sayles, N. M. & Sheltzer, J. M. CRISPR/Cas9 mutagenesis invalidates a putative cancer dependency targeted in on-going clinical trials. eLife 6, e24179 (2017).
  14. Gorre, M. et al. Clinical resistance to STI-571 cancer therapy caused by BCR–ABL gene mutation or amplification. Science 293, 876880 (2001).
  15. Azam, M., Latek, R. R. & Daley, G. Q. Mechanisms of autoinhibition and STI-571/imatinib resistance revealed by mutagenesis of BCR–ABL. Cell 112, 831843 (2003).
  16. Burgess, M. R., Skaggs, B. J., Shah, N. P., Lee, F. Y. & Sawyers, C. L. Comparative analysis of two clinically active BCR–ABL kinase inhibitors reveals the role of conformation-specific binding in resistance. Proc. Natl Acad. Sci. USA 102, 33953400 (2005).
  17. Balbas, M. D. et al. Overcoming mutation-based resistance to antiandrogens with rational drug design. eLife 2, e00499 (2013).
  18. Wacker, S. A., Houghtaling, B. R., Elemento, O. & Kapoor, T. M. Using transcriptome sequencing to identify mechanisms of drug action and resistance. Nat. Chem. Biol. 8, 235237 (2012).
  19. Kasap, C., Elemento, O. & Kapoor, T. M. DrugTargetSeqR: a genomics- and CRISPR–Cas9-based method to analyze drug targets. Nat. Chem. Biol. 10, 626628 (2014).
  20. Han, T. et al. The antitumor toxin CD437 is a direct inhibitor of DNA polymerase α. Nat. Chem. Biol. 12, 511515 (2016).
  21. Forment, J. V. et al. Genome-wide genetic screening with chemically mutagenized haploid embryonic stem cells. Nat. Chem. Biol. 13, 1214 (2017).
  22. Emery, C. M. et al. MEK1 mutations confer resistance to MEK and B-RAF inhibition. Proc. Natl Acad. Sci. USA 106, 2041120416 (2009).
  23. Nazarian, R. et al. Melanomas acquire resistance to B-RAF(V600E) inhibition by RTK or N-RAS upregulation. Nature 468, 973977 (2010).
  24. Losman, J. A. et al. (R)-2-hydroxyglutarate is sufficient to promote leukemogenesis and its effects are reversible. Science 339, 16211625 (2013).
  25. Kaelin, W. G. Jr. Molecular biology. Use and abuse of RNAi to study mammalian gene function. Science 337, 421422 (2012).
  26. Kittler, R. et al. RNA interference rescue by bacterial artificial chromosome transgenesis in mammalian tissue culture cells. Proc. Natl Acad. Sci. USA 102, 23962401 (2005).
  27. Brignole, C. et al. Immune cell-mediated antitumor activities of GD2-targeted liposomal c-myb antisense oligonucleotides containing CpG motifs. J. Natl Cancer Inst. 96, 11711180 (2004).
  28. Munoz, D. M. et al. CRISPR screens provide a comprehensive assessment of cancer vulnerabilities but generate false-positive hits for highly amplified genomic regions. Cancer Discov. 6, 900913 (2016).
  29. Aguirre, A. J. et al. Genomic copy number dictates a gene-independent cell response to CRISPR/Cas9 targeting. Cancer Discov. 6, 914929 (2016).
  30. Blouw, B. et al. The hypoxic response of tumors is dependent on their microenvironment. Cancer Cell 4, 133146 (2003).
  31. Bridgeman, V. L. et al. Vessel co-option is common in human lung metastases and mediates resistance to anti-angiogenic therapy in preclinical lung metastasis models. J. Pathol. 241, 362374 (2017).
  32. Donnem, T. et al. Vessel co-option in primary human tumors and metastases: an obstacle to effective anti-angiogenic treatment? Cancer Med. 2, 427436 (2013).
  33. Tybulewicz, V. L., Crawford, C. E., Jackson, P. K., Bronson, R. T. & Mulligan, R. C. Neonatal lethality and lymphopenia in mice with a homozygous disruption of the c-abl proto-oncogene. Cell 65, 11531163 (1991).
  34. Hamilton, A. L. et al. Proteasome inhibition with bortezomib (PS-341): a phase I study with pharmacodynamic end points using a day 1 and day 4 schedule in a 14-day cycle. J. Clin. Oncol. 23, 61076116 (2005).
  35. Kaelin, W. J. Choosing anticancer drug targets in the postgenomic era. J. Clin. Invest. 104, 15031506 (1999).
  36. Kaelin, W. G. Jr. The concept of synthetic lethality in the context of anticancer therapy. Nat. Rev. Cancer 5, 689698 (2005).

Download references

Author information

Affiliations

  1. Howard Hughes Medical Institute, Dana-Farber Cancer Institute and Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts 02215, USA.

    • William G. Kaelin Jr

Competing interests statement

The author declares no competing interests.

Corresponding author

Correspondence to:

Author details

  • William G. Kaelin Jr

    William G. Kaelin Jr obtained his undergraduate degree and M.D. from Duke University (Durham, North Carolina, USA) and did Internal Medicine training at the Johns Hopkins Hospital, Baltimore, Maryland, USA. He completed a medical oncology fellowship at the Dana-Farber Cancer Institute, Boston, Massuchusetts, USA, where he later trained in David Livingston's laboratory. His laboratory studies the biochemical functions of specific tumour suppressor proteins, including the von Hippel–Lindau (VHL) protein, in the hopes of identifying new cancer targets.

Additional data