The clinical translation of promising products, technologies and interventions from the disciplines of nanomedicine and cell therapy has been slow and inefficient. In part, translation has been hampered by suboptimal research practices that propagate biases and hinder reproducibility. These include the publication of small and underpowered preclinical studies, suboptimal study design (in particular, biased allocation of experimental groups, experimenter bias and lack of necessary controls), the use of uncharacterized or poorly characterized materials, poor understanding of the relevant biology and mechanisms, poor use of statistics, large between-model heterogeneity, absence of replication, lack of interdisciplinarity, poor scientific training in study design and methods, a culture that does not incentivize transparency and sharing, poor or selective reporting, misaligned incentives and rewards, high costs of materials and protocols, and complexity of the developed products, technologies and interventions. In this Perspective, we discuss special manifestations of these problems in nanomedicine and in cell therapy, and describe mitigating strategies. Progress on reducing bias and enhancing reproducibility early on ought to enhance the translational potential of biomedical findings and technologies.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Bowen, A. & Casadevall, A. Increasing disparities between resource inputs and outcomes, as measured by certain health deliverables, in biomedical research. Proc. Natl Acad. Sci. USA 112, 11335–11340 (2015).
Contopoulos-Ioannidis, D. G., Ntzani, E. & Ioannidis, J. P. Translation of highly promising basic science research into clinical applications. Am. J. Med. 114, 477–484 (2003).
Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).
Allison, D. B., Brown, A. W., George, B. J. & Kaiser, K. A. Reproducibility: a tragedy of errors. Nature 530, 27–29 (2016).
Lithgow, G. J., Driscoll, M. & Phillips, P. A long journey to reproducible results. Nature 548, 387–388 (2017).
Bissell, M. Reproducibility: the risks of the replication drive. Nature 503, 333–334 (2013).
Ioannidis, J. P. The reproducibility wars: successful, unsuccessful, uninterpretable, exact, conceptual, triangulated, contested replication. Clin. Chem. 63, 943–945 (2017).
Ioannidis, J. P. Acknowledging and overcoming nonreproducibility in basic and preclinical research. JAMA 317, 1019–1020 (2017).
Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712 (2011).
Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).
Nosek, B. A. & Errington, T. M. Making sense of replications. eLife 6, e23383 (2017).
Begley, C. G. & Ioannidis, J. P. Reproducibility in science: improving the standard for basic and preclinical research. Circ. Res. 116, 116–126 (2015).
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).
Szucs, D. & Ioannidis, J. P. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol. 15, e2000797 (2017).
Tsilidis, K. K. et al. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol. 11, e1001609 (2013).
Hess, K. R. Statistical design considerations in animal studies published recently in Cancer Research. Cancer Res. 71, 625 (2011).
Kilkenny, C. et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS ONE 4, e7824 (2009).
Steward, O., Popovich, P. G., Dietrich, W. D. & Kleitman, N. Replication and reproducibility in spinal cord injury research. Exp. Neurol. 233, 597–605 (2012).
Szucs, D. & Ioannidis, J. P. A. When null hypothesis significance testing is unsuitable for research: a reassessment. Front. Hum. Neurosci. 11, 390 (2017).
Greenland, S. et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur. J. Epidemiol. 31, 337–350 (2016).
Benjamin, D. J. et al. Redefine statistical significance. Nat. Hum. Behav. 2, 6–10 (2018).
Bruns, S. B. & Ioannidis, J. P. p-curve and p-hacking in observational research. PLoS ONE 11, e0149144 (2016).
Veresoglou, S. D. P hacking in biology: an open secret. Proc. Natl Acad. Sci. USA 112, E5112–E5113 (2015).
Fanelli, D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4, e5738 (2009).
Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13, e1002165 (2015).
Ioannidis, J. P. A. et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383, 166–175 (2014).
Kimmelman, J., Mogil, J. S. & Dirnagl, U. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol. 12, e1001863 (2014).
Simeon-Dubach, D., Burt, A. D. & Hall, P. A. Quality really matters: the need to improve specimen quality in biomedical research. J. Pathol. 228, 431–433 (2012).
Dirnagl, U. et al. A concerted appeal for international cooperation in preclinical stroke research. Stroke 44, 1754–1760 (2013).
Goodman, S. N. Introduction to Bayesian methods I: measuring the strength of evidence. Clin. Trials 2, 282–290 (2005).
Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).
Wasserstein, R. L. & Lazar, N. A. The ASA’s statement on p-values: context, process, and purpose. Am. Statistician 70, 129–133 (2016).
Colquhoun, D. The reproducibility of research and the misinterpretation of p-values. R. Soc. Open Sci. 4, 171085 (2017).
Colquhoun, D. The false positive risk: a proposal concerning what to do about p values. Preprint at https://arXiv.org/abs/1802.04888 (2018).
Ioannidis, J. P. The proposal to lower P value thresholds to .005. JAMA 319, 1429–1430 (2018).
Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422–1425 (2015).
Stodden, V. et al. Enhancing reproducibility for computational methods. Science 354, 1240–1241 (2016).
Simera, I., Moher, D., Hoey, J., Schulz, K. F. & Altman, D. G. A catalogue of reporting guidelines for health research. Eur. J. Clin. Invest. 40, 35–53 (2010).
Henderson, V. C., Kimmelman, J., Fergusson, D., Grimshaw, J. M. & Hackam, D. G. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med. 10, e1001489 (2013).
Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P. & Willmes, K. Registered reports: realigning incentives in scientific publishing. Cortex 66, A1–A2 (2015).
Moher, D., Goodman, S. N. & Ioannidis, J. P. Academic criteria for appointment, promotion and rewards in medical research: where’s the evidence? Eur. J. Clin. Invest. 46, 383–385 (2016).
Ioannidis, J. P. & Khoury, M. J. Assessing value in biomedical research: the PQRST of appraisal and reward. JAMA 312, 483–484 (2014).
Jackman, J. A., Lee, J. & Cho, N. J. Nanomedicine for infectious disease applications: innovation towards broad-spectrum treatment of viral infections. Small 12, 1133–1139 (2016).
Barranco, C. Nanomedicine, meet autoimmune disease. Nat. Rev. Rheum. 12, 193 (2016).
Clemente-Casares, X. et al. Expanding antigen-specific regulatory networks to treat autoimmunity. Nature 530, 434–440 (2016).
Wilhelm, S. et al. Analysis of nanoparticle delivery to tumours. Nat. Rev. Mater. 1, 16014 (2016).
Shi, J., Kantoff, P. W., Wooster, R. & Farokhzad, O. C. Cancer nanomedicine: progress, challenges and opportunities. Nat. Rev. Cancer 17, 20–37 (2017).
Jiang, W. et al. Designing nanomedicine for immuno-oncology. Nat. Biomed. Eng. 1, 0029 (2017).
Blanco, E., Shen, H. & Ferrari, M. Principles of nanoparticle design for overcoming biological barriers to drug delivery. Nat. Biotechnol. 33, 941–951 (2015).
von Roemeling, C. A., Jiang, W., Chan, C. K., Weissman, I. L. & Kim, B. Y. S. Breaking down the barriers to precision cancer nanomedicine. Trends Biotechnol. 35, 159–171 (2017).
Rolland, J. P., Hagberg, E. C., Denison, G. M., Carter, K. R. & De Simone, J. M. High-resolution soft lithography: enabling materials for nanotechnologies. Angew. Chem. Int. Ed. 43, 5796–5799 (2004).
Xu, J. et al. Future of the particle replication in nonwetting templates (PRINT) technology. Angew. Chem. Int. Ed. 52, 6580–6589 (2013).
Peer, D. et al. Nanocarriers as an emerging platform for cancer therapy. Nat. Nanotech. 2, 751–760 (2007).
Rodriguez, P. L. et al. Minimal “self” peptides that inhibit phagocytic clearance and enhance delivery of nanoparticles. Science 339, 971–975 (2013).
Monopoli, M. P., Åberg, C., Salvati, A. & Dawson, K. A. Biomolecular coronas provide the biological identity of nanosized materials. Nat. Nanotech. 7, 779–786 (2012).
Albanese, A., Tang, P. S. & Chan, W. C. W. The effect of nanoparticle size, shape, and surface chemistry on biological systems. Annu. Rev. Biomed. Eng. 14, 1–16 (2012).
Walkey, C. D., Olsen, J. B., Guo, H., Emili, A. & Chan, W. C. W. Nanoparticle size and surface chemistry determine serum protein adsorption and macrophage uptake. J. Am. Chem. Soc. 134, 2139–2147 (2012).
Salvati, A. et al. Transferrin-functionalized nanoparticles lose their targeting capabilities when a biomolecule corona adsorbs on the surface. Nat. Nanotech. 8, 137–143 (2013).
Jiang, W., Kim, B. Y. S., Rutka, J. T. R. & Chan, W. C. W. Nanoparticle-mediated cellular response is size-dependent. Nat. Nanotech. 3, 145–150 (2008).
Rice, S. B. et al. Particle size distributions by transmission electron microscopy: an interlaboratory comparison case study. Metrologia 50, 663–678 (2013).
Krystek, P., Ulrich, A., Garcia, C. C., Manohar, S. & Ritsema, R. Application of plasma spectrometry for the analysis of engineered nanoparticles in suspensions and products. J. Anal. Atom. Spectrom. 26, 1701–1721 (2011).
Masters, J. R. Cell-line authentication: end the scandal of false cell lines. Nature 492, 186 (2012).
Allen, M., Bjerke, M., Edlund, H., Nelander, S. & Westermark, B. Origin of the U87MG glioma cell line: good news and bad news. Sci. Transl. Med. 8, 354re3 (2016).
Pollard, S. M. et al. Glioma stem cell lines expanded in adherent culture have tumor-specific phenotypes and are suitable for chemical and genetic screens. Cell Stem Cell 4, 568–580 (2009).
Tsuchiya, S. et al. Establishment and characterization of a human acute monocytic leukemia cell line (THP-1). Int. J. Cancer 26, 171–176 (1980).
Prideaux, B. & Stoeckli, M. Mass spectrometry imaging for drug distribution studies. J. Proteomics 75, 4999–5013 (2012).
Limberis, M. P., Bell, C. L. & Wilson, J. M. Identification of the murine firefly luciferase-specific CD8 T-cell epitopes. Gene Therapy 16, 441–447 (2009).
Baklaushev, V. P. et al. Luciferase expression allows bioluminescence imaging but imposes limitations on the orthotopic mouse (4T1) model of breast cancer. Sci. Rep. 7, 7715 (2017).
Dimmeler, S., Ding, S., Rando, T. A. & Trounson, A. Translational strategies and challenges in regenerative medicine. Nat. Med. 20, 814–821 (2014).
Rennert, R. C. et al. High-resolution microfluidic single-cell transcriptional profiling reveals clinically relevant subtypes among human stem cell populations commonly utilized in cell-based therapies. Front. Neurology 7, 41 (2016).
Ortmann, D. & Vallier, L. Variability of human pluripotent stem cell lines. Curr. Opin. Genet. Dev. 46, 179–185 (2017).
Paladino, F. V., Sardinha, L. R., Piccinato, C. A. & Goldberg, A. C. Intrinsic variability present in Wharton’s jelly mesenchymal stem cells and T cell responses may impact cell therapy. Stem Cells Int. 2017, 8492797 (2017).
Bianco, P. “Mesenchymal” stem cells. Annu. Rev. Cell Dev. Biol. 30, 677–704 (2014).
Merkle, F. T. et al. Human pluripotent stem cells recurrently acquire and expand dominant P53 mutations. Nature 545, 229–233 (2017).
Trounson, A. Potential pitfall of pluripotent stem cells. N. Engl. J. Med. 377, 490–491 (2017).
Wang, X. et al. Tumor suppressor gene alterations of spontaneously transformed cells from human embryonic muscle in vitro. Oncol. Rep. 24, 555–561 (2010).
Tang, C., Weissman, I. L. & Drukker, M. The safety of embryonic stem cell therapy relies on teratoma removal. Oncotarget 3, 7–8 (2012).
Barker, R. A. et al. Are stem cell-based therapies for Parkinson’s disease ready for the clinic in 2016? J. Parkinson’s Dis. 6, 57–63 (2016).
Kriks, S. et al. Floor plate-derived dopamine neurons from hESCs efficiently engraft in animal models of PD. Nature 480, 547–551 (2011).
Gonzalez, R. et al. Neural stem cells from human parthenogenetic stem cells engraft and promote recovery in a nonhuman primate model of Parkinson’s disease. Cell Transplant. 25, 1945–1966 (2016).
Wagers, A. J. & Weissman, I. L. Plasticity of adult stem cells. Cell 116, 639–648 (2004).
Kikuchi, T. et al. Human iPS cell-derived dopaminergic neurons function in a primate Parkinson’s disease model. Nature 548, 592–596 (2017).
Schulz, T. C. Concise review: manufacture of pancreatic endoderm cells for clinical trials in type 1 diabetes. Stem Cells Transl. Med. 4, 927–931 (2015).
Vegas, A. J. et al. Long term glycemic control using polymer encapsulated human stem-cell derived β-cells in immune competent mice. Nat. Med. 22, 306–311 (2016).
Brudno, J. N. & Kochenderfer, J. N. Chimeric antigen receptor T-cell therapies for lymphoma. Nat. Rev. Clin. Oncol. 15, 31–46 (2018).
Hombach, A. A. & Abken, H. Most do, but some do not: CD4+CD25– T cells, but not CD4+CD25+ Treg cells, are cytolytic when redirected by chimeric antigen receptor (CAR). Cancers (Basel) 9, 112 (2017).
Temple, S. & Studder, L. Lessons learned from pioneering neural stem cell studies. Stem Cell Rep. 8, 191–193 (2017).
Prestwich, G. D. et al. What is the greatest regulatory challenge in the translation of biomaterials to the clinic? Sci. Transl. Med. 4, 160cm14 (2012).
Trounson, A. & McDonald, C. Stem cell therapies in clinical trials: progress and challenges. Cell Stem Cell 17, 11–22 (2015).
Karnik, R. et al. Microfluidic platform for controlled synthesis of polymeric nanoparticles. Nano Lett. 8, 2906–2912 (2008).
Tropsha, A., Mills, K. C. & Hickey, A. J. Reproducibility, sharing and progress in nanomaterial databases. Nat. Nanotech. 12, 1111–1114 (2017).
Rolland, J. P. et al. Direct fabrication and harvesting of monodisperse, shape-specific nanobiomaterials. J. Am. Chem. Soc. 127, 10096–10100 (2005).
Huh, D. et al. Reconstituting organ-level lung functions on a chip. Science 328, 1662–1668 (2010).
Albanese, A., Lam, A. K., Sykes, E. A., Rocheleau, J. V. & Chan, W. C. Tumour-on-a-chip provides an optical window into nanoparticle tissue transport. Nat. Commun. 4, 2718 (2013).
Rongvaux, A. et al. Development and function of human innate immune cells in a humanized mouse model. Nat. Biotechnol. 32, 364–372 (2014).
Liu, Q., Shepherd, B. E., Li, C. & Harrell, F. E. Jr Modeling continuous response using ordinal regression. Stat. Med. https://doi.org/10.1002/sim.7433 (2017).
Barish, S., Ochs, M. F., Sontag, E. D. & Gevertz, J. L. Evaluating optimal therapy robustness by virtual expansion of a sample population, with a case study in cancer immunotherapy. Proc. Natl Acad. Sci. USA 114, E6277–E6286 (2017).
Lin-Gibson, S., Sarkar, S. & Ito, Y. Defining quality attributes to enable measurement assurance for cell therapy products. Cytotherapy 18, 1241–1244 (2016).
Maus, M. V. & Kikiforow, S. The why, what, and how of the new FACT standards for immune effector cells. J. Immunother. Cancer 5, 36 (2017).
Dropulic, B. Reference standards for gene and cell therapy products. Mol. Therapy 25, 1259–1260 (2017).
Stacey, G. N. et al. Preservation and stability of cell therapy products: recommendations from an expert workshop. Regen. Med. 12, 553–564 (2017).
Williams, D. J. et al. Comparability: manufacturing, characterization and controls, report of a UK regenerative medicine platform pluripotent stem cell platform workshop, Trinity Hall, Cambridge, 14-15 September 2015. Regen. Med. 11, 483–492 (2016).
Rama, P., Ferrari, G. & Pellegrini, G. Cultivated limbal epithelial transplantation. Curr. Opin. Ophthalmol. 28, 387–389 (2017).
Wei, G., Wang, J., Huang, H. & Zhao, Y. Novel immunotherapies for adult patients with B-lineage acute lymphoblastic leukemia. J. Hemat. Oncol. 10, 150 (2017).
Brudno, J. N. & Kochenderfer, J. N. Toxicities of chimeric antigen receptor T cells: recognition and management. Blood 127, 3321–3330 (2016).
Benjamin, D., Mandel, D. R. & Kimmelman, J. Can cancer researchers accurately judge whether preclinical reports will reproduce? PLoS Biol. 15, e2002212 (2017).
Fesnak, A., June, C. H. & Levine, B. L. Engineered T cells: the promise and challenges of cancer immunotherapy. Nat. Rev. Cancer 16, 566–681 (2016).
Trounson, A., DeWitt, N. D. & Feigal, E. G. The alpha stem cell clinic: a model for evaluating and delivering stem cell-based therapies. Stem Cells Transl. Med. 1, 9–14 (2012).
Lomax, G. P. et al. Accelerating stem cell treatments for patients: the value of networks and collaborations. Stem Cells Portal http://stemcellsportal.com/content/2015-0090 (2015).
Mayo-Wilson, E., Doshi, P. & Dickersin, K. Are manufacturers sharing data as promised?. Br. Med. J. 351, h4169 (2015).
Guinney, J. et al. Prediction of overall survival for patients with metastatic castration-resistant prostate cancer: development of a prognostic model through a crowdsourced challenge with open clinical trial data. Lancet Oncol. 18, 132–142 (2017).
Corrie, B. D. et al. iReceptor: a platform for querying and analysing antibody/B-cell and T-cell receptor repertoire data across federated repositories. Immunol. Rev. 284, 24–41 (2018).
Clevers, H. Modeling development and disease with organoids. Cell 165, 1586–1597 (2016).
Esch, E. W., Bahinski, A. & Huh, D. Organs-on-chips at the frontiers of drug discovery. Nat. Rev. Drug Discov. 14, 248–260 (2015).
Verma, S. et al. Trastuzumab emtansine for HER2-positive advanced breast cancer. N. Engl. J. Med. 367, 1783–1791 (2012).
Lewis Phillips, G. D. et al. Targeting HER2-positive breast cancer with trastuzumab-DM1, an antibody-cytotoxic drug conjugate. Cancer Res. 68, 9280–9290 (2008).
Erickson, H. K. et al. Antibody-maytansinoid conjugates are activated in targeted cancer cells by lysosomal degradation and linker-dependent intracellular processing. Cancer Res. 66, 4426–4433 (2006).
Holden, S. N. et al. A phase I study of weekly dosing of trastuzumab-DM1 (T-DM1) in patients (pts) with advanced HER2+ breast cancer. J. Clin. Oncol. 26, 1029 (2008).
Vieweg, J. et al. Immunotherapy of prostate cancer in the Dunning rat model: use of cytokine gene modified tumor vaccines. Cancer Res. 54, 1760–1765 (1994).
Hurwitz, A. A. et al. Combination immunotherapy of primary prostate cancer in a transgenic mouse model using CTLA-4 blockade. Cancer Res. 60, 2444–2448 (2000).
Kantoff, P. W. et al. Sipuleucel-T immunotherapy for castration-resistant prostate cancer. N. Engl. J. Med. 363, 411–422 (2010).
Higano, C. S. et al. Integrated data from 2 randomized, double-blind, placebo-controlled, phase 3 trials of active cellular immunotherapy with sipuleucel-T in advanced prostate cancer. Cancer 115, 3670–3679 (2009).
Gong, C. L. & Hay, J. W. Cost-effectiveness analysis of abiraterone and sipuleucel-T in asymptomatic metastatic castration-resistant prostate cancer. J. Natl Compr. Canc. Netw. 12, 1417–1425 (2014).
Geynisman, D. M., Chien, C. R., Smieliauskas, F., Shen, C. & Shih, Y. C. Economic evaluation of therapeutic cancer vaccines and immunotherapy: a systematic review. Hum. Vaccin. Immunother. 10, 3415–3424 (2014).
Jarosławski, S. & Toumi, M. Sipuleucel-T (Provenge)-autopsy of an innovative paradigm change in cancer treatment: why a single-product biotech company failed to capitalize on its breakthrough invention. BioDrugs 29, 301–307 (2015).
Simpson, E. L., Davis, S., Thokala, P., Breeze, P. R., Bryden, P. & Wong, R. Sipuleucel-T for the treatment of metastatic hormone-relapsed prostate cancer: a NICE single technology appraisal; an evidence review group perspective. Pharmacoeconomics 33, 1187–1194 (2015).
We thank N. Boyd for helping create Fig. 2. We also acknowledge funding from the Mayo Clinic Center for Regenerative Medicine (B.Y.S.K.), the National Institute of Neurological Disorders and Stroke Grant R01 NS104315 (B.Y.S.K.) and the Laura and John Arnold Foundation for providing funding for the Meta-Research Innovation Center at Stanford (METRICS) (J.P.A.I.).
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ioannidis, J.P.A., Kim, B.Y.S. & Trounson, A. How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation. Nat Biomed Eng 2, 797–809 (2018). https://doi.org/10.1038/s41551-018-0314-y
Π electron-stabilized polymeric micelles potentiate docetaxel therapy in advanced-stage gastrointestinal cancer
Size and charge correlations in spherical electric double layers: a case study with fully asymmetric mixed electrolytes within the solvent primitive model
RSC Advances (2020)
Actively targeted nanomedicines for precision cancer therapy: Concept, construction, challenges and clinical translation
Journal of Controlled Release (2020)
Advanced Functional Materials (2020)
Photosensitizer-stabilized self-assembling nanoparticles potentiate chemo/photodynamic efficacy of patient-derived melanoma
Journal of Controlled Release (2020)