Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Reusability report: Evaluating reproducibility and reusability of a fine-tuned model to predict drug response in cancer patient samples


Machine learning and artificial intelligence methods are increasingly being used in personalized medicine, including precision oncology. Ma et al. (Nature Cancer 2021) have developed a new method called ‘transfer of cell line response prediction’ (TCRP) to train predictors of drug response in cancer cell lines and optimize their performance in higher complex cancer model systems via few-shot learning. TCRP has been presented as a successful modelling approach in multiple case studies. Given the importance of this approach for assisting clinicians in their treatment decision processes, we sought to independently reproduce the authors’ findings and improve the reusability of TCRP in new case studies, including validation in clinical-trial datasets—a high bar for drug-response prediction. Our reproducibility results, while not reaching the same level of superiority as those of the original authors, were able to confirm the superiority of TCRP in the original clinical context. Our reusability results indicate that, in the majority of novel clinical contexts, TCRP remains the superior method for predicting response for both preclinical and clinical settings. Our results thus support the superiority of TCRP over established statistical and machine learning approaches in preclinical and clinical settings. We also developed new resources to increase the reusability of the TCRP model for future improvements and validation studies.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Reproducibility attempt for Challenge #2 from the original paper by Ma et al.
Fig. 2: Equivalent performance calculations on TCRP and baseline methods, using Spearman’s correlation as previously done in the original paper by Ma et al.
Fig. 3: Performance of TCRP versus common baselines on new reference datasets.
Fig. 4: Performance of TCRP versus other baselines in three new validation contexts.

Similar content being viewed by others

Data availability

All datasets used in our study are available on the data platform ORCESTRA, under dataset names GDSC_2020(v1-8.2), GDSC_2020(v2-8.2), gCSI_2019, CTRPv2_2015, PDTX_2019, PDXE, UHNBreast_2019, scRNA_GSE25066_Breast and scRNA_GSE41998_Breast.

Code availability

The code to run and visualize the exact results of our reproducibility attempt as well as all our novel analyses for reusability are available in the corresponding Code Ocean capsule (Version 2.5)22. Code used for the analysis is available at


  1. Trastulla, L., Noorbakhsh, J., Vazquez, F., McFarland, J. & Iorio, F. Computational estimation of quality and clinical relevance of cancer cell lines. Mol. Syst. Biol. 18, e11017 (2022).

    Article  Google Scholar 

  2. Ma, J. et al. Few-shot learning creates predictive models of drug response that translate from high-throughput screens to individual patients. Nat. Cancer 2, 233–244 (2021).

    Article  Google Scholar 

  3. Yang, W. et al. Genomics of Drug Sensitivity in Cancer (GDSC): a resource for therapeutic biomarker discovery in cancer cells. Nucleic Acids Res. 41, D955–D961 (2013).

    Article  Google Scholar 

  4. Bruna, A. et al. A biobank of breast cancer explants with preserved intra-tumor heterogeneity to screen anticancer compounds. Cell 167, 260–274 (2016).

    Article  Google Scholar 

  5. Smirnov, P. et al. Evaluation of statistical approaches for association testing in noisy drug screening data. BMC Bioinf. 23, 188 (2022).

    Article  Google Scholar 

  6. Mammoliti, A. et al. Orchestrating and sharing large multimodal data for transparent and reproducible research. Nat. Commun. 12, 5797 (2021).

    Article  Google Scholar 

  7. Seashore-Ludlow, B. et al. Harnessing coÿnnectivity in a large-scale small-molecule sensitivity dataset. Cancer Discov 5, 1210–1223 (2015).

    Article  Google Scholar 

  8. Haverty, P. M. et al. Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 533, 333–337 (2016).

    Article  Google Scholar 

  9. Iorio, F. et al. A landscape of pharmacogenomic interactions in cancer. Cell 166, 740–754 (2016).

    Article  Google Scholar 

  10. Safikhani, Z. et al. Gene isoforms as expression-based biomarkers predictive of drug response in vitro. Nat. Commun. 8, 1126 (2017).

    Article  Google Scholar 

  11. Thu, K. L. et al. Disruption of the anaphase-promoting complex confers resistance to TTK inhibitors in triple-negative breast cancer. Proc. Natl Acad. Sci. USA 115, E1570–E1577 (2018).

    Article  Google Scholar 

  12. Gao, H. et al. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nat. Med. 21, 1318–1325 (2015).

    Article  Google Scholar 

  13. Hatzis, C. et al. A genomic predictor of response and survival following taxane-anthracycline chemotherapy for invasive breast cancer. JAMA 305, 1873–1881 (2011).

    Article  Google Scholar 

  14. Itoh, M. et al. Estrogen receptor (ER) mRNA expression and molecular subtype distribution in ER-negative/progesterone receptor-positive breast cancers. Breast Cancer Res. Treat. 143, 403–409 (2014).

    Article  Google Scholar 

  15. Baldasici, O. et al. Circulating small EVs miRNAs as predictors of pathological response to neo-adjuvant therapy in breast cancer patients. Int. J. Mol. Sci. 23, 12625 (2022).

    Article  Google Scholar 

  16. Horak, C. E. et al. Biomarker analysis of neoadjuvant doxorubicin/cyclophosphamide followed by ixabepilone or Paclitaxel in early-stage breast cancer. Clin. Cancer Res. 19, 1587–1595 (2013).

    Article  Google Scholar 

  17. RECIST 1.1 (EORTC);

  18. Safikhani, Z. et al. Revisiting inconsistency in large pharmacogenomic studies. F1000Res. 5, 2333 (2016).

    Article  Google Scholar 

  19. Barretina, J. et al. The Cancer Cell Line Encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature 483, 603–607 (2012).

    Article  Google Scholar 

  20. Raff, E. Research reproducibility as a survival analysis. In Proc. AAAI Conference on Artificial Intelligence Vol. 35, 469–478 (AAAI, 2021).

  21. Clyburne-Sherin, A., Fei, X. & Green, S. A. Computational reproducibility via containers in psychology. Meta-Psychology (2019).

  22. Reusability Report: Few-shot learning creates predictive models of drug response that translate from high-throughput screens to individual patients (Version 2.5) (Code Ocean);

Download references


This work was supported by Canadian Cancer Society Data Transformations (707609) and by a Natural Sciences and Engineering Research Council of Canada Discovery Grant (RGPIN-2021-02680).

Author information

Authors and Affiliations



E.S. and F.Y. conducted the reproducibility and reusability experiments, curated datasets, drafted the first version of the manuscript and integrated all the edits. B.W. supervised the manuscript. B.H.-K. contributed ideas and supervised the manuscript writing and final edits.

Corresponding author

Correspondence to Benjamin Haibe-Kains.

Ethics declarations

Competing interests

B.H.K. is a shareholder and paid consultant for Code Ocean Inc. E.S. is a paid consultant for Code Ocean Inc. The remaining authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Bo Yuan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Tables 1–8.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

So, E., Yu, F., Wang, B. et al. Reusability report: Evaluating reproducibility and reusability of a fine-tuned model to predict drug response in cancer patient samples. Nat Mach Intell 5, 792–798 (2023).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing