Real-world data: towards achieving the achievable in cancer care


The use of data from the real world to address clinical and policy-relevant questions that cannot be answered using data from clinical trials is garnering increased interest. Indeed, data from cancer registries and linked treatment records can provide unique insights into patients, treatments and outcomes in routine oncology practice. In this Review, we explore the quality of real-world data (RWD), provide a framework for the use of RWD and draw attention to the methodological pitfalls inherent to using RWD in studies of comparative effectiveness. Randomized controlled trials and RWD remain complementary forms of medical evidence; studies using RWD should not be used as substitutes for clinical trials. The comparison of outcomes between nonrandomized groups of patients who have received different treatments in routine practice remains problematic. Accordingly, comparative effectiveness studies need to be designed and interpreted very carefully. With due diligence, RWD can be used to identify and close gaps in health care, offering the potential for short-term improvement in health-care systems by enabling them to achieve the achievable.

Key points

  • In the past decade, interest in linking electronic health records (EHRs) of treatment and outcome to cancer registry data has increased; these sources of real-world data (RWD) can offer unique insights into patients, treatments and outcomes in routine oncology practice.

  • The quality of RWD relates to the quality of the primary data (completeness, accuracy and comprehensiveness), data linkages and derived variables assessed.

  • Emerging sources of RWD from EHRs, mobile applications and wearable technologies, can offer improved granularity over traditional sources of RWD, but their internal validity and applicability remain largely unknown.

  • For health-care systems, the use of RWD enables measurement of performance and can be used to identify targets for future quality improvement interventions.

  • RWD can offer important insights into outcomes achieved with new anticancer therapies in routine practice; however, the comparison of outcomes between nonrandomized groups of patients who have received different treatments in routine practice remains problematic and is not a substitute for randomized controlled trials.

  • Investigators working with RWD need to move beyond simply describing gaps in care and towards designing intervention studies to improve patient care and outcomes.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Fig. 1: Increased reporting of studies using real-world data.
Fig. 2: Conceptual framework depicting a general scheme for research on health-care system performance.


  1. 1.

    Goldberg, R. M., Wei, L. & Fernandez, S. The evolution of clinical trials in oncology: defining who benefits from new drugs using innovative study designs. Oncologist 22, 1015–1019 (2017).

    PubMed  PubMed Central  Google Scholar 

  2. 2.

    Sherman, R. E. et al. Real-world evidence — what is it and what can it tell us? N. Engl. J. Med. 375, 2293–2297 (2016).

    PubMed  Google Scholar 

  3. 3.

    Potosky, A. L., Riley, G. F., Lubitz, J. D., Mentnech, R. M. & Kessler, L. G. Potential for cancer related health services research using a linked Medicare-tumor registry database. Med. Care 31, 732–748 (1993).

    CAS  PubMed  Google Scholar 

  4. 4.

    Mackillop, W. J. et al. Waiting for radiotherapy in Ontario. Int. J. Radiat. Oncol. Biol. Phys. 30, 221–228 (1994).

    CAS  PubMed  Google Scholar 

  5. 5.

    Parkin, D. M. The evolution of the population-based cancer registry. Nat. Rev. Cancer 6, 603–612 (2006).

    CAS  PubMed  Google Scholar 

  6. 6.

    Friends of Cancer Research. Establishing a framework to evaluate real-world endpoints. FOCR (2018).

  7. 7.

    Cook, J. A. & Collins, G. S. The rise of big clinical databases. Br. J. Surg. 102, e93–e101 (2015).

    CAS  PubMed  Google Scholar 

  8. 8.

    Khozin, S. et al. Characteristics of real-world metastatic non-small cell lung cancer patients treated with nivolumab and pembrolizumab during the year following approval. Oncologist 23, 328–336 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Harrison, L. D. et al. Comparing effectiveness with efficacy: outcomes of palliative chemotherapy for non-small-cell lung cancer in routine practice. Curr. Oncol. 22, 184–191 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Lorenzi, M. et al. Educational outcomes among survivors of childhood cancer in British Columbia, Canada: report of the Childhood/Adolescent/Young Adult Cancer Survivors (CAYACS) Program. Cancer 115, 2234–2245 (2009).

    PubMed  Google Scholar 

  11. 11.

    Krzyzanowska, M. K. et al. Can chemotherapy-related acute care visits be accurately identified in administrative data? J. Oncol. Pract. 14, e51–e58 (2018).

    PubMed  Google Scholar 

  12. 12.

    Satkunam, N., Wei, X., Biagi, J. J., Nanji, S. & Booth, C. M. Delivery of adjuvant oxaliplatin for colon cancer: insights from routine clinical practice. J. Natl Compr. Canc. Netw. 14, 1548–1554 (2016).

    CAS  PubMed  Google Scholar 

  13. 13.

    Booth, C. M. et al. Perioperative chemotherapy for muscle-invasive bladder cancer: a population-based outcomes study. Cancer 120, 1630–1638 (2014).

    PubMed  Google Scholar 

  14. 14.

    Fellegi, I. P. & Sunter, A. B. A theory of record linkage. J. Am. Stat. Assoc. 61, 1183–1210 (1969).

    Google Scholar 

  15. 15.

    Zhu, Y., Matsuyama, Y., Ohashi, Y. & Setoguchi, S. When to conduct probabilistic linkage versus deterministic linkage? A simulation study. J. Biomed. Inform. 56, 80–86 (2015).

    PubMed  Google Scholar 

  16. 16.

    Booth, C. M. et al. Adoption of adjuvant chemotherapy for non-small-cell lung cancer: a population-based outcomes study. J. Clin. Oncol. 28, 3472–3478 (2010).

    PubMed  PubMed Central  Google Scholar 

  17. 17.

    Seisen, T. et al. Efficacy of high-intensity local treatment for metastatic urothelial carcinoma of the bladder: a propensity score-weighted analysis from the National Cancer Data Base. J. Clin. Oncol. 34, 3529–3536 (2016).

    PubMed  Google Scholar 

  18. 18.

    Booth, C. M. et al. Radical treatment of the primary tumor in metastatic bladder cancer: potentially dangerous findings from observational data. J. Clin. Oncol. 36, 533–535 (2017).

    PubMed  Google Scholar 

  19. 19.

    Black, N. & Payne, M. Directory of clinical databases: improving and promoting their use. Qual. Saf. Health Care 12, 348–352 (2003).

    CAS  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Raftery, J., Roderick, P. & Stevens, A. Potential use of routine databases in health technology assessment. Health Technol. Assess. 9, 1–92 (2005).

    CAS  PubMed  Google Scholar 

  21. 21.

    Haider, A. H., Bilimoria, K. Y. & Kibbe, M. R. A checklist to elevate the science of surgical database research. JAMA Surg. 153, 505–507 (2018).

    PubMed  Google Scholar 

  22. 22.

    Doll, K. M., Rademaker, A. & Sosa, J. A. Practical guide to surgical data sets: surveillance, epidemiology, and end results (SEER) database. JAMA Surg. 153, 588–589 (2018).

    PubMed  Google Scholar 

  23. 23.

    Merkow, R. P., Rademaker, A. W. & Bilimoria, K. Y. Practical guide to surgical data sets: national cancer database (NCDB). JAMA Surg. 153, 850–851 (2018).

    PubMed  Google Scholar 

  24. 24.

    Kaji, A. H., Rademaker, A. W. & Hyslop, T. Tips for analyzing large data sets from the JAMA surgery statistical editors. JAMA Surg. 153, 508–509 (2018).

    PubMed  Google Scholar 

  25. 25.

    Bray, F., Jemal, A., Grey, N., Ferlay, J. & Forman, D. Global cancer transitions according to the Human Development Index (2008–2030): a population-based study. Lancet Oncol. 13, 790–801 (2012).

    PubMed  Google Scholar 

  26. 26.

    Rabkin, C. S., Biggar, R. J. & Horm, J. W. Increasing incidence of cancers associated with the human immunodeficiency virus epidemic. Int. J. Cancer 47, 692–696 (1991).

    CAS  PubMed  Google Scholar 

  27. 27.

    Chaturvedi, A. K., Engels, E. A., Anderson, W. F. & Gillison, M. L. Incidence trends for human papillomavirus-related and -unrelated oral squamous cell carcinomas in the United States. J. Clin. Oncol. 26, 612–619 (2008).

    PubMed  Google Scholar 

  28. 28.

    Mackillop, W. J., Zhang-Salomons, J., Boyd, C. J. & Groome, P. A. Associations between community income and cancer incidence in Canada and the United States. Cancer 89, 901–912 (2000).

    CAS  PubMed  Google Scholar 

  29. 29.

    Booth, C. M. & Tannock, I. F. Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence. Br. J. Cancer 110, 551–555 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  30. 30.

    Mitchell, A. P. et al. Clinical trial participants with metastatic renal cell carcinoma differ from patients treated in real-world practice. J. Oncol. Pract. 11, 491–497 (2015).

    PubMed  Google Scholar 

  31. 31.

    Seow, H. et al. Trajectory of performance status and symptom scores for patients with cancer during the last six months of life. J. Clin. Oncol. 29, 1151–1158 (2011).

    PubMed  Google Scholar 

  32. 32.

    Mackillop, W. J. et al. Does a centralized radiotherapy system provide adequate access to care? J. Clin. Oncol. 15, 1261–1271 (1997).

    CAS  PubMed  Google Scholar 

  33. 33.

    Schrag, D., Cramer, L. D., Bach, P. B. & Begg, C. B. Age and adjuvant chemotherapy use after surgery for stage III colon cancer. J. Natl Cancer Inst. 93, 850–857 (2001).

    CAS  PubMed  Google Scholar 

  34. 34.

    Goossens-Laan, C. A. et al. Effects of age and comorbidity on treatment and survival of patients with muscle-invasive bladder cancer. Int. J. Cancer 135, 905–912 (2014).

    CAS  PubMed  Google Scholar 

  35. 35.

    Booth, C. M., Siemens, D. R., Peng, Y. & Mackillop, W. J. Patterns of referral for perioperative chemotherapy among patients with muscle-invasive bladder cancer: a population-based study. Urol. Oncol. 32, 1200–1208 (2014).

    PubMed  Google Scholar 

  36. 36.

    Chandhoke, G. et al. Patterns of referral for adjuvant chemotherapy for stage II and III colon cancer: a population-based study. Ann. Surg. Oncol. 23, 2529–2538 (2016).

    PubMed  Google Scholar 

  37. 37.

    Booth, C. M. et al. Perioperative chemotherapy for bladder cancer in the general population: are practice patterns finally changing? Urol. Oncol. 36, 89.e13–89.e20 (2017).

    Google Scholar 

  38. 38.

    Cuffe, S. et al. Adjuvant chemotherapy for non-small-cell lung cancer in the elderly: a population-based study in Ontario, Canada. J. Clin. Oncol. 30, 1813–1821 (2012).

    PubMed  Google Scholar 

  39. 39.

    Kankesan, J. et al. Factors associated with referral to medical oncology and subsequent use of adjuvant chemotherapy for non-small-cell lung cancer: a population-based study. Curr. Oncol. 20, 30–37 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  40. 40.

    Quirt, J. S., Siemens, D. R., Zaza, K., Mackillop, W. J. & Booth, C. M. Patterns of referral to radiation oncology among patients with bladder cancer: a population-based study. Clin. Oncol. 29, 171–179 (2017).

    CAS  Google Scholar 

  41. 41.

    Kumachev, A., Trudeau, M. E. & Chan, K. K. Associations among socioeconomic status, patterns of care and outcomes in breast cancer patients in a universal health care system: Ontario’s experience. Cancer 122, 893–898 (2016).

    PubMed  Google Scholar 

  42. 42.

    Deb, S. et al. The effect of socioeconomic status on gross total resection, radiation therapy and overall survival in patients with gliomas. J. Neurooncol. 132, 447–453 (2017).

    PubMed  Google Scholar 

  43. 43.

    Huang, J. et al. Factors affecting the use of palliative radiotherapy in Ontario. J. Clin. Oncol. 19, 137–144 (2001).

    CAS  PubMed  Google Scholar 

  44. 44.

    Jacobs, L. K., Kelley, K. A., Rosson, G. D., Detrani, M. E. & Chang, D. C. Disparities in urban and rural mastectomy populations: the effects of patient- and county-level factors on likelihood of receipt of mastectomy. Ann. Surg. Oncol. 15, 2644–2652 (2008).

    PubMed  Google Scholar 

  45. 45.

    Atkins, G. T., Kim, T. & Munson, J. Residence in rural areas of the United States and lung cancer mortality. Disease incidence, treatment disparities, and stage-specific survival. Ann. Am. Thorac. Soc. 14, 403–411 (2017).

    PubMed  Google Scholar 

  46. 46.

    Xu, L., Kim, Y., Spolverato, G., Gani, F. & Pawlik, T. M. Racial disparities in treatment and survival of patients with hepatocellular carcinoma in the United States. Hepatobiliary Surg. Nutr. 5, 43–52 (2016).

    PubMed  PubMed Central  Google Scholar 

  47. 47.

    Lawrenson, R. et al. Treatment and survival disparities by ethnicity in New Zealand women with stage I-III breast cancer tumour subtypes. Cancer Causes Control 28, 1417–1427 (2017).

    PubMed  Google Scholar 

  48. 48.

    Tyldesley, S. et al. Association between age and the utilization of radiotherapy in Ontario. Int. J. Radiat. Oncol. Biol. Phys. 47, 469–480 (2000).

    CAS  PubMed  Google Scholar 

  49. 49.

    Magrath, I. et al. Paediatric cancer in low-income and middle-income countries. Lancet Oncol. 14, e104–e116 (2013).

    PubMed  Google Scholar 

  50. 50.

    Duthey, B. & Scholten, W. Adequacy of opioid analgesic consumption at country, global, and regional levels in 2010, its relationship with development level, and changes compared with 2006. J. Pain Symptom Manage. 47, 283–297 (2014).

    PubMed  Google Scholar 

  51. 51.

    McLaughlin, P. Y. et al. Do radiation oncology outreach clinics affect the use of radiotherapy? Radiother. Oncol. 127, 143–149 (2018).

    PubMed  Google Scholar 

  52. 52.

    Robinson, D. et al. Waiting times for radiotherapy: variation over time and between cancer networks in southeast England. Br. J. Cancer 92, 1201–1208 (2005).

    CAS  PubMed  PubMed Central  Google Scholar 

  53. 53.

    Hershman, D. et al. Timing of adjuvant chemotherapy initiation after surgery for stage III colon cancer. Cancer 107, 2581–2588 (2006).

    PubMed  Google Scholar 

  54. 54.

    Wasserman, D. W. et al. Reasons for delay in time to initiation of adjuvant chemotherapy for colon cancer. J. Oncol. Pract. 11, e28–e35 (2014).

    PubMed  Google Scholar 

  55. 55.

    Arndt, V. et al. Patient delay and stage of diagnosis among breast cancer patients in Germany — a population based study. Br. J. Cancer 86, 1034–1040 (2002).

    CAS  PubMed  PubMed Central  Google Scholar 

  56. 56.

    McKenzie, F. et al. Drivers of advanced stage at breast cancer diagnosis in the multicountry African breast cancer — disparities in outcomes (ABC-DO) study. Int. J. Cancer 142, 1568–1579 (2018).

    CAS  PubMed  Google Scholar 

  57. 57.

    Simos, D. et al. Imaging for distant metastases in women with early-stage breast cancer: a population-based cohort study. CMAJ 187, E387–E397 (2015).

    PubMed  PubMed Central  Google Scholar 

  58. 58.

    Parmar, A. D. et al. Quality of post-treatment surveillance of early stage breast cancer in Texas. Surgery 154, 214–225 (2013).

    PubMed  PubMed Central  Google Scholar 

  59. 59.

    Arnaout, A. et al. Use of preoperative magnetic resonance imaging for breast cancer: a Canadian population-based study. JAMA Oncol. 1, 1238–1250 (2015).

    PubMed  Google Scholar 

  60. 60.

    Ashworth, A., Kong, W., Chow, E. & Mackillop, W. J. Fractionation of palliative radiation therapy for bone metastases in Ontario: do practice guidelines guide practice? Int. J. Radiat. Oncol. Biol. Phys. 94, 31–39 (2016).

    PubMed  Google Scholar 

  61. 61.

    Ashworth, A., Kong, W., Whelan, T. & Mackillop, W. J. A population-based study of the fractionation of postlumpectomy breast radiation therapy. Int. J. Radiat. Oncol. Biol. Phys. 86, 51–57 (2013).

    PubMed  Google Scholar 

  62. 62.

    Schnipper, L. E. et al. American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: the top five list for oncology. J. Clin. Oncol. 30, 1715–1724 (2012).

    PubMed  Google Scholar 

  63. 63.

    Mitera, G. et al. Choosing Wisely Canada cancer list: ten low-value or harmful practices that should be avoided in cancer care. J. Oncol. Pract. 11, e296–e303 (2015).

    PubMed  Google Scholar 

  64. 64.

    Olson, R. A. et al. Impact of using audit data to improve the evidence-based use of single-fraction radiation therapy for bone metastases in British Columbia. Int. J. Radiat. Oncol. Biol. Phys. 94, 40–47 (2016).

    PubMed  Google Scholar 

  65. 65.

    Tyldesley, S., Boyd, C., Schulze, K., Walker, H. & Mackillop, W. J. Estimating the need for radiotherapy for lung cancer: an evidence-based, epidemiologic approach. Int. J. Radiat. Oncol. Biol. Phys. 49, 973–985 (2001).

    CAS  PubMed  Google Scholar 

  66. 66.

    Round, C. E. et al. Radiotherapy demand and activity in England 2006–2020. Clin. Oncol. 25, 522–530 (2013).

    CAS  Google Scholar 

  67. 67.

    Mackillop, W. J. et al. A comparison of evidence-based estimates and empirical benchmarks of the appropriate rate of use of radiation therapy in Ontario. Int. J. Radiat. Oncol. Biol. Phys. 91, 1099–1107 (2015).

    PubMed  Google Scholar 

  68. 68.

    Barbera, L., Zhang-Salomons, J., Huang, J., Tyldesley, S. & Mackillop, W. Defining the need for radiotherapy for lung cancer in the general population: a criterion-based, benchmarking approach. Med. Care 41, 1074–1085 (2003).

    PubMed  Google Scholar 

  69. 69.

    Cancer Quality Council of Ontario. Access to radiation treatment. Cancer System Quality Index (2017).

  70. 70.

    Rahal, R., Klein-Geltink, J., Forte, T., Lockwood, G. & Bryant, H. Measuring concordance with guidelines for the diagnosis and treatment of colon cancer. Curr. Oncol. 20, 227–229 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  71. 71.

    Salloum, R. G., Smith, T. J., Jensen, G. A. & Lafata, J. E. Factors associated with adherence to chemotherapy guidelines in patients with non-small cell lung cancer. Lung Cancer 75, 255–260 (2012).

    PubMed  Google Scholar 

  72. 72.

    Young, J. M. et al. Concordance with national guidelines for colorectal cancer care in New South Wales: a population-based patterns of care study. Med. J. Aust. 186, 292–295 (2007).

    PubMed  Google Scholar 

  73. 73.

    Wong, S. J. et al. Longitudinal Oncology Registry of Head and Neck Carcinoma (LORHAN): analysis of chemoradiation treatment approaches in the United States. Cancer 117, 1679–1686 (2011).

    PubMed  Google Scholar 

  74. 74.

    Nandi, M., Mandal, A. & Asthana, A. K. Audit of cancer patients from eastern Uttar Pradesh (UP), India: a university hospital based two year retrospective analysis. Asian Pac. J. Cancer Prev. 14, 4993–4998 (2013).

    Google Scholar 

  75. 75.

    Porter, G. A., Urquhart, R., Bu, J., Johnson, P. & Grunfeld, E. The impact of audit and feedback on nodal harvest in colorectal cancer. BMC Cancer 11, 2 (2011).

    PubMed  PubMed Central  Google Scholar 

  76. 76.

    Alvarez, E. et al. Improvement in treatment abandonment in pediatric patients with cancer in Guatemala. Pediatr. Blood Cancer 64, 10 (2017).

    Google Scholar 

  77. 77.

    Gadalla, S. M. et al. A population-based assessment of mortality and morbidity patterns among patients with thymoma. Int. J. Cancer 128, 2688–2694 (2011).

    CAS  PubMed  Google Scholar 

  78. 78.

    Lee, Y. Y. et al. Incidence and outcomes of pregnancy-associated cancer in Australia, 1994-2008: a population-based linkage study. BJOG 119, 1572–1582 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  79. 79.

    Chu, C. N. et al. Increase in stroke risk in patients with head and neck cancer: a retrospective cohort study. Br. J. Cancer 105, 1419–1423 (2011).

    PubMed  PubMed Central  Google Scholar 

  80. 80.

    Marmor, S., Burke, E. E., Virnig, B. A., Jensen, E. H. & Tuttle, T. M. A comparative analysis of survival outcomes between pancreatectomy and chemotherapy for elderly patients with adenocarcinoma of the pancreas. Cancer 122, 3378–3385 (2016).

    PubMed  Google Scholar 

  81. 81.

    Paszat, L. F., Mackillop, W. J., Groome, P. A., Schulze, K. & Holowaty, E. Mortality from myocardial infarction following postlumpectomy radiotherapy for breast cancer: a population-based study in Ontario, Canada. Int. J. Radiat. Oncol. Biol. Phys. 43, 755–762 (1999).

    CAS  PubMed  Google Scholar 

  82. 82.

    Seruga, B., Sterling, L., Wang, L. & Tannock, I. F. Reporting of serious adverse drug reactions of targeted anticancer agents in pivotal phase III clinical trials. J. Clin. Oncol. 29, 174–185 (2011).

    CAS  PubMed  Google Scholar 

  83. 83.

    Groome, P. A. et al. Management and outcome of glottic cancer: a population-based comparison between Ontario, Canada and the SEER areas of the United States. Surveillance, epidemiology and end results. J. Otolaryngol. 29, 67–77 (2000).

    CAS  PubMed  Google Scholar 

  84. 84.

    Capri, S. & Russo, A. Cost of breast cancer based on real-world data: a cancer registry study in Italy. BMC Health Serv. Res. 17, 84 (2017).

    PubMed  PubMed Central  Google Scholar 

  85. 85.

    Siemens, D. R. et al. Processes of care and the impact of surgical volumes on cancer-specific survival: a population-based study in bladder cancer. Urology 84, 1049–1057 (2014).

    PubMed  Google Scholar 

  86. 86.

    Booth, C. M., Nanji, S., Wei, X. & Mackillop, W. J. Management and outcome of colorectal cancer liver metastases in elderly patients: a population-based study. JAMA Oncol. 1, 1111–1119 (2015).

    PubMed  Google Scholar 

  87. 87.

    Patel, M. I., Bang, A., Gillett, D., Cheluvappa, R. & Smith, D. P. Poor survival of females with bladder cancer is limited to those aged 70 years or over: a population-wide linkage study, New South Wales, Australia. Cancer Med. 4, 1145–1152 (2015).

    PubMed  PubMed Central  Google Scholar 

  88. 88.

    Walker, M. et al. A call for theory-informed approaches to knowledge translation studies: an example of chemotherapy for bladder cancer. Curr. Oncol. 22, 178–181 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  89. 89.

    Templeton, A. J. et al. Translating clinical trials to clinical practice: outcomes of men with metastatic castration resistant prostate cancer treated with docetaxel and prednisone in and out of clinical trials. Ann. Oncol. 24, 2972–2977 (2013).

    CAS  PubMed  Google Scholar 

  90. 90.

    Booth, C. M. et al. Curative therapy for bladder cancer in routine clinical practice: a population-based outcomes study. Clin. Oncol. 26, 506–514 (2014).

    CAS  Google Scholar 

  91. 91.

    Stein, J. P. et al. Radical cystectomy in the treatment of invasive bladder cancer: long-term results in 1,054 patients. J. Clin. Oncol. 19, 666–675 (2001).

    CAS  PubMed  Google Scholar 

  92. 92.

    Lau, K. et al. The effect of a regional hepatopancreaticobiliary surgical program on clinical volume, quality of cancer care, and outcomes in the Veterans Affairs system. JAMA Surg. 149, 1153–1161 (2014).

    PubMed  Google Scholar 

  93. 93.

    Simunovic, M. et al. Assessing the volume-outcome hypothesis and region-level quality improvement interventions: pancreas cancer surgery in two Canadian Provinces. Ann. Surg. Oncol. 17, 2537–2544 (2010).

    PubMed  Google Scholar 

  94. 94.

    van Roest, M. H., van der Aa, M. A., van der Geest, L. G. & de Jong, K. P. The impact of socioeconomic status, surgical resection and type of hospital on survival in patients with pancreatic cancer. A population-based study in The Netherlands. PLOS ONE 11, e0166449 (2016).

    PubMed  PubMed Central  Google Scholar 

  95. 95.

    Bendzsak, A. M., Baxter, N. N., Darling, G. E., Austin, P. C. & Urbach, D. R. Regionalization and outcomes of lung cancer surgery in Ontario, Canada. J. Clin. Oncol. 35, 2772–2780 (2017).

    PubMed  Google Scholar 

  96. 96.

    Nienhuijs, S. W. et al. Reduction of in-hospital mortality following regionalisation of pancreatic surgery in the south-east of the Netherlands. Eur. J. Surg. Oncol. 36, 652–656 (2010).

    CAS  PubMed  Google Scholar 

  97. 97.

    De Angelis, R. et al. The EUROCARE-4 database on cancer survival in Europe: data standardisation, quality control and methods of statistical analysis. Eur. J. Cancer 45, 909–930 (2009).

    PubMed  Google Scholar 

  98. 98.

    Thomson, C. S. & Forman, D. Cancer survival in England and the influence of early diagnosis: what can we learn from recent EUROCARE results? Br. J. Cancer 101 (Suppl. 2), 102–109 (2009).

    Google Scholar 

  99. 99.

    Lozano, R. et al. Benchmarking of performance of Mexican states with effective coverage. Lancet 368, 1729–1741 (2006).

    PubMed  Google Scholar 

  100. 100.

    Zubizarreta, E. H., Fidarova, E., Healy, B. & Rosenblatt, E. Need for radiotherapy in low and middle income countries — the silent crisis continues. Clin. Oncol. 27, 107–114 (2015).

    CAS  Google Scholar 

  101. 101.

    Nandakumar, A. et al. Concurrent chemoradiation for cancer of the cervix: results of a multi-institutional study from the setting of a developing country (India). J. Glob. Oncol. 1, 11–22 (2015).

    PubMed  PubMed Central  Google Scholar 

  102. 102.

    Veenstra, C. M. et al. Long-term economic and employment outcomes among partners of women with early-stage breast cancer. J. Oncol. Pract. 13, e916–e926 (2017).

    PubMed  PubMed Central  Google Scholar 

  103. 103.

    Lin, P. J. et al. Linking costs and survival in the treatment of older adults with chronic myeloid leukemia: an analysis of SEER-Medicare data from 1995 to 2007. Med. Care 54, 380–385 (2016).

    PubMed  Google Scholar 

  104. 104.

    Roehrborn, C. G. & Black, L. K. The economic burden of prostate cancer. BJU Int. 108, 806–813 (2011).

    PubMed  Google Scholar 

  105. 105.

    Mullins, C. D., Hsiao, F. Y., Onukwugha, E., Pandya, N. B. & Hanna, N. Comparative and cost-effectiveness of oxaliplatin-based or irinotecan-based regimens compared with 5-fluorouracil/leucovorin alone among US elderly stage IV colon cancer patients. Cancer 118, 3173–3181 (2012).

    CAS  PubMed  Google Scholar 

  106. 106.

    Mackillop, W. J. in Clinical Radiation Oncology 3rd edn (eds Gunderson, L. & Tepper, J.) 203–222 (Churchill Livingstone, Philadelphia, 2012).

  107. 107.

    Ong, M. B. H. Flatiron, BMS form collaboration to curate regulatory-grade real-world data. The Cancer Letter (2018).

  108. 108.

    Goodwin, P. J., Ballman, K. V., Small, E. J. & Cannistra, S. A. Evaluation of treatment benefit in journal of clinical oncology. J. Clin. Oncol. 31, 1123–1124 (2013).

    PubMed  Google Scholar 

  109. 109.

    Chang, G. J. Is there validity in propensity score-matched estimates of adjuvant chemotherapy effects for patients with rectal cancer? JAMA Oncol. 4, 921–923 (2018).

    PubMed  Google Scholar 

  110. 110.

    Choudhury, A. & Hoskin, P. J. Bladder cancer and the National Cancer Data Base: new insight or misinformation? Cancer 124, 1105–1107 (2018).

    PubMed  Google Scholar 

  111. 111.

    Unger, J. M., Cook, E., Tai, E. & Bleyer, A. The role of clinical trial participation in cancer research: barriers, evidence, and strategies. Am. Soc. Clin. Oncol. Educ. Book 35, 185–198 (2016).

    PubMed  PubMed Central  Google Scholar 

  112. 112.

    The Children’s Oncology Group. What is a clincial trial? COG (2018).

  113. 113.

    Yusuf, S., Collins, R. & Peto, R. Why do we need some large, simple randomized trials? Stat. Med. 3, 409–422 (1984).

    CAS  PubMed  Google Scholar 

  114. 114.

    Parmar, M. K. et al. Paclitaxel plus platinum-based chemotherapy versus conventional platinum-based chemotherapy in women with relapsed ovarian cancer: the ICON4/AGO-OVAR-2.2 trial. Lancet 361, 2099–2106 (2003).

    CAS  PubMed  Google Scholar 

  115. 115.

    James, N. D. et al. Addition of docetaxel, zoledronic acid, or both to first-line long-term hormone therapy in prostate cancer (STAMPEDE): survival results from an adaptive, multiarm, multistage, platform randomised controlled trial. Lancet 387, 1163–1177 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  116. 116.

    Khozin, S., Blumenthal, G. M. & Pazdur, R. Real-world data for clinical evidence generation in oncology. J. Natl Cancer Inst. 109, djx187 (2017).

    Google Scholar 

  117. 117.

    Corrigan-Curay, J., Sacks, L. & Woodcock, J. Real-world evidence and real-world data for evaluating drug safety and effectiveness. JAMA 320, 867–868 (2018).

    PubMed  Google Scholar 

  118. 118.

    Sackett, D. L. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest 95, 2S–4S (1989).

    CAS  PubMed  Google Scholar 

  119. 119.

    Oxford Centre for Evidence-Based Medicine. OCEBM levels of evidence. CEBM (2018).

  120. 120.

    Stukel, T. A. et al. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA 297, 278–285 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  121. 121.

    Schneeweiss, S. & Maclure, M. Use of comorbidity scores for control of confounding in studies using administrative databases. Int. J. Epidemiol. 29, 891–898 (2000).

    CAS  PubMed  Google Scholar 

  122. 122.

    Groome, P. A. & Mackillop, W. J. Uses of ecologic studies in the assessment of intended treatment effects. J. Clin. Epidemiol. 52, 903–904 (1999).

    CAS  PubMed  Google Scholar 

  123. 123.

    Pearcey, R., Miao, Q., Kong, W., Zhang-Salomons, J. & Mackillop, W. J. Impact of adoption of chemoradiotherapy on the outcome of cervical cancer in Ontario: results of a population-based cohort study. J. Clin. Oncol. 25, 2383–2388 (2007).

    CAS  PubMed  Google Scholar 

  124. 124.

    Groome, P. A. et al. Management and outcome differences in supraglottic cancer between Ontario, Canada, and the surveillance, epidemiology, and end results areas of the United States. J. Clin. Oncol. 21, 496–505 (2003).

    PubMed  Google Scholar 

  125. 125.

    Gupta, S., Kong, W., Booth, C. M. & Mackillop, W. J. Impact of concomitant chemotherapy on outcomes of radiation therapy for head-and-neck cancer: a population-based study. Int. J. Radiat. Oncol. Biol. Phys. 88, 115–121 (2014).

    PubMed  Google Scholar 

  126. 126.

    Booth, C. M. et al. Adjuvant chemotherapy for non-small cell lung cancer: practice patterns and outcomes in the general population of Ontario, Canada. J. Thorac. Oncol. 7, 559–566 (2012).

    PubMed  Google Scholar 

  127. 127.

    Erlichman, C. Efficacy of adjuvant fluorouracil and folinic acid in B2 colon cancer. J. Clin. Oncol. 17, 1356–1363 (1999).

    Google Scholar 

  128. 128.

    van Gijn, W. et al. Preoperative radiotherapy combined with total mesorectal excision for resectable rectal cancer: 12-year follow-up of the multicentre, randomised controlled TME trial. Lancet Oncol. 12, 575–582 (2011).

    PubMed  Google Scholar 

  129. 129.

    Casadaban, L. et al. Adjuvant chemotherapy is associated with improved survival in patients with stage II colon cancer. Cancer 122, 3277–3287 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  130. 130.

    Freischlag, K. et al. Association between incomplete neoadjuvant radiotherapy and survival for patients with locally advanced rectal cancer. JAMA Surg. 152, 558–564 (2017).

    PubMed  PubMed Central  Google Scholar 

  131. 131.

    Glasziou, P., Vandenbroucke, J. P. & Chalmers, I. Assessing the quality of research. BMJ 328, 39–41 (2004).

    PubMed  PubMed Central  Google Scholar 

  132. 132.

    Sanderson, S., Tatt, I. D. & Higgins, J. P. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36, 666–676 (2007).

    PubMed  Google Scholar 

  133. 133.

    Vandenbroucke, J. P. et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Ann. Intern. Med. 147, W163–W194 (2007).

    PubMed  Google Scholar 

  134. 134.

    Wells, G. S. et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute (2009).

  135. 135.

    Stang, A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur. J. Epidemiol. 25, 603–605 (2010).

    PubMed  Google Scholar 

  136. 136.

    Suissa, S. Immortal time bias in pharmaco-epidemiology. Am. J. Epidemiol. 167, 492–499 (2008).

    PubMed  Google Scholar 

  137. 137.

    Hill, A. B. The environment and disease: association of causation. Proc. R. Soc. Med. 58, 295–300 (1965).

    CAS  PubMed  PubMed Central  Google Scholar 

  138. 138.

    Rothman, K. J., Greenland, S. & Lash, T. L. Modern epidemiology (Lippincott Williams & Wilkins, Philadelphia, 2008).

  139. 139.

    Makady, A. et al. Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies. Value Health 20, 520–532 (2017).

    PubMed  Google Scholar 

  140. 140.

    Risk Sciences International. Evidence synthesis on post-approval surveillance of approved novel cancer drugs. Canadian Partnership Against Cancer (2015).

  141. 141.

    Jarow, J. P., LaVange, L. & Woodcock, J. Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data. JAMA 318, 703–704 (2017).

    PubMed  Google Scholar 

  142. 142.

    Food and Drug Administration. Statement from FDA Commissioner Scott Gottlieb, M. D., on FDA’s new strategic framework to advance use of real-world evidence to support development of drugs and biologics. FDA (2018).

  143. 143.

    Hagen, T. ASCO partners with FDA to incorporate real-world evidence into drug decisions. OncLive (2017).

  144. 144.

    Hall, P. S. Real-world data for efficient health technology assessment. Eur. J. Cancer 79, 235–237 (2017).

    PubMed  Google Scholar 

  145. 145.

    Lewis, J. R., Kerridge, I. & Lipworth, W. Coverage with evidence development and managed entry in the funding of personalized medicine: practical and ethical challenges for oncology. J. Clin. Oncol. 33, 4112–4117 (2015).

    CAS  PubMed  Google Scholar 

  146. 146.

    Espin, J., Rovira, J. & Garcia, L. Experiences and impact of European risk-sharing schemes focusing on oncology medicines. European Commission (2011).

  147. 147.

    Fujiwara, Y. & Kobayashi, K. Oncology drug clinical development and approval in Japan: the role of the pharmaceuticals and medical devices evaluation center (PMDEC). Crit. Rev. Oncol. Hematol. 42, 145–155 (2002).

    PubMed  Google Scholar 

  148. 148.

    Mackillop, W. J. & Kong, W. Estimating the need for palliative radiation therapy: a benchmarking approach. Int. J. Radiat. Oncol. Biol. Phys. 94, 51–59 (2016).

    PubMed  Google Scholar 

Download references


The work of C.M.B. is supported by his position as the Canada Research Chair in Population Cancer Care. All the authors thank I. Tannock and E. Eisenhauer for their comments on an earlier draft of this manuscript.

Reviewer information

Nature Reviews Clinical Oncology thanks A. Abernethy, C. Uyl-de-Groot and the other anonymous reviewer(s) for their contribution to the peer review of this work.

Author information




All authors made a substantial contribution to all aspects of the preparation of this manuscript.

Corresponding author

Correspondence to Christopher M. Booth.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Booth, C.M., Karim, S. & Mackillop, W.J. Real-world data: towards achieving the achievable in cancer care. Nat Rev Clin Oncol 16, 312–325 (2019).

Download citation

Further reading