The use of data from the real world to address clinical and policy-relevant questions that cannot be answered using data from clinical trials is garnering increased interest. Indeed, data from cancer registries and linked treatment records can provide unique insights into patients, treatments and outcomes in routine oncology practice. In this Review, we explore the quality of real-world data (RWD), provide a framework for the use of RWD and draw attention to the methodological pitfalls inherent to using RWD in studies of comparative effectiveness. Randomized controlled trials and RWD remain complementary forms of medical evidence; studies using RWD should not be used as substitutes for clinical trials. The comparison of outcomes between nonrandomized groups of patients who have received different treatments in routine practice remains problematic. Accordingly, comparative effectiveness studies need to be designed and interpreted very carefully. With due diligence, RWD can be used to identify and close gaps in health care, offering the potential for short-term improvement in health-care systems by enabling them to achieve the achievable.
In the past decade, interest in linking electronic health records (EHRs) of treatment and outcome to cancer registry data has increased; these sources of real-world data (RWD) can offer unique insights into patients, treatments and outcomes in routine oncology practice.
The quality of RWD relates to the quality of the primary data (completeness, accuracy and comprehensiveness), data linkages and derived variables assessed.
Emerging sources of RWD from EHRs, mobile applications and wearable technologies, can offer improved granularity over traditional sources of RWD, but their internal validity and applicability remain largely unknown.
For health-care systems, the use of RWD enables measurement of performance and can be used to identify targets for future quality improvement interventions.
RWD can offer important insights into outcomes achieved with new anticancer therapies in routine practice; however, the comparison of outcomes between nonrandomized groups of patients who have received different treatments in routine practice remains problematic and is not a substitute for randomized controlled trials.
Investigators working with RWD need to move beyond simply describing gaps in care and towards designing intervention studies to improve patient care and outcomes.
Subscribe to Journal
Get full journal access for 1 year
only $17.75 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Goldberg, R. M., Wei, L. & Fernandez, S. The evolution of clinical trials in oncology: defining who benefits from new drugs using innovative study designs. Oncologist 22, 1015–1019 (2017).
Sherman, R. E. et al. Real-world evidence — what is it and what can it tell us? N. Engl. J. Med. 375, 2293–2297 (2016).
Potosky, A. L., Riley, G. F., Lubitz, J. D., Mentnech, R. M. & Kessler, L. G. Potential for cancer related health services research using a linked Medicare-tumor registry database. Med. Care 31, 732–748 (1993).
Mackillop, W. J. et al. Waiting for radiotherapy in Ontario. Int. J. Radiat. Oncol. Biol. Phys. 30, 221–228 (1994).
Parkin, D. M. The evolution of the population-based cancer registry. Nat. Rev. Cancer 6, 603–612 (2006).
Friends of Cancer Research. Establishing a framework to evaluate real-world endpoints. FOCR https://www.focr.org/sites/default/files/pdf/RWE_FINAL%207.6.18.pdf (2018).
Cook, J. A. & Collins, G. S. The rise of big clinical databases. Br. J. Surg. 102, e93–e101 (2015).
Khozin, S. et al. Characteristics of real-world metastatic non-small cell lung cancer patients treated with nivolumab and pembrolizumab during the year following approval. Oncologist 23, 328–336 (2018).
Harrison, L. D. et al. Comparing effectiveness with efficacy: outcomes of palliative chemotherapy for non-small-cell lung cancer in routine practice. Curr. Oncol. 22, 184–191 (2015).
Lorenzi, M. et al. Educational outcomes among survivors of childhood cancer in British Columbia, Canada: report of the Childhood/Adolescent/Young Adult Cancer Survivors (CAYACS) Program. Cancer 115, 2234–2245 (2009).
Krzyzanowska, M. K. et al. Can chemotherapy-related acute care visits be accurately identified in administrative data? J. Oncol. Pract. 14, e51–e58 (2018).
Satkunam, N., Wei, X., Biagi, J. J., Nanji, S. & Booth, C. M. Delivery of adjuvant oxaliplatin for colon cancer: insights from routine clinical practice. J. Natl Compr. Canc. Netw. 14, 1548–1554 (2016).
Booth, C. M. et al. Perioperative chemotherapy for muscle-invasive bladder cancer: a population-based outcomes study. Cancer 120, 1630–1638 (2014).
Fellegi, I. P. & Sunter, A. B. A theory of record linkage. J. Am. Stat. Assoc. 61, 1183–1210 (1969).
Zhu, Y., Matsuyama, Y., Ohashi, Y. & Setoguchi, S. When to conduct probabilistic linkage versus deterministic linkage? A simulation study. J. Biomed. Inform. 56, 80–86 (2015).
Booth, C. M. et al. Adoption of adjuvant chemotherapy for non-small-cell lung cancer: a population-based outcomes study. J. Clin. Oncol. 28, 3472–3478 (2010).
Seisen, T. et al. Efficacy of high-intensity local treatment for metastatic urothelial carcinoma of the bladder: a propensity score-weighted analysis from the National Cancer Data Base. J. Clin. Oncol. 34, 3529–3536 (2016).
Booth, C. M. et al. Radical treatment of the primary tumor in metastatic bladder cancer: potentially dangerous findings from observational data. J. Clin. Oncol. 36, 533–535 (2017).
Black, N. & Payne, M. Directory of clinical databases: improving and promoting their use. Qual. Saf. Health Care 12, 348–352 (2003).
Raftery, J., Roderick, P. & Stevens, A. Potential use of routine databases in health technology assessment. Health Technol. Assess. 9, 1–92 (2005).
Haider, A. H., Bilimoria, K. Y. & Kibbe, M. R. A checklist to elevate the science of surgical database research. JAMA Surg. 153, 505–507 (2018).
Doll, K. M., Rademaker, A. & Sosa, J. A. Practical guide to surgical data sets: surveillance, epidemiology, and end results (SEER) database. JAMA Surg. 153, 588–589 (2018).
Merkow, R. P., Rademaker, A. W. & Bilimoria, K. Y. Practical guide to surgical data sets: national cancer database (NCDB). JAMA Surg. 153, 850–851 (2018).
Kaji, A. H., Rademaker, A. W. & Hyslop, T. Tips for analyzing large data sets from the JAMA surgery statistical editors. JAMA Surg. 153, 508–509 (2018).
Bray, F., Jemal, A., Grey, N., Ferlay, J. & Forman, D. Global cancer transitions according to the Human Development Index (2008–2030): a population-based study. Lancet Oncol. 13, 790–801 (2012).
Rabkin, C. S., Biggar, R. J. & Horm, J. W. Increasing incidence of cancers associated with the human immunodeficiency virus epidemic. Int. J. Cancer 47, 692–696 (1991).
Chaturvedi, A. K., Engels, E. A., Anderson, W. F. & Gillison, M. L. Incidence trends for human papillomavirus-related and -unrelated oral squamous cell carcinomas in the United States. J. Clin. Oncol. 26, 612–619 (2008).
Mackillop, W. J., Zhang-Salomons, J., Boyd, C. J. & Groome, P. A. Associations between community income and cancer incidence in Canada and the United States. Cancer 89, 901–912 (2000).
Booth, C. M. & Tannock, I. F. Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence. Br. J. Cancer 110, 551–555 (2014).
Mitchell, A. P. et al. Clinical trial participants with metastatic renal cell carcinoma differ from patients treated in real-world practice. J. Oncol. Pract. 11, 491–497 (2015).
Seow, H. et al. Trajectory of performance status and symptom scores for patients with cancer during the last six months of life. J. Clin. Oncol. 29, 1151–1158 (2011).
Mackillop, W. J. et al. Does a centralized radiotherapy system provide adequate access to care? J. Clin. Oncol. 15, 1261–1271 (1997).
Schrag, D., Cramer, L. D., Bach, P. B. & Begg, C. B. Age and adjuvant chemotherapy use after surgery for stage III colon cancer. J. Natl Cancer Inst. 93, 850–857 (2001).
Goossens-Laan, C. A. et al. Effects of age and comorbidity on treatment and survival of patients with muscle-invasive bladder cancer. Int. J. Cancer 135, 905–912 (2014).
Booth, C. M., Siemens, D. R., Peng, Y. & Mackillop, W. J. Patterns of referral for perioperative chemotherapy among patients with muscle-invasive bladder cancer: a population-based study. Urol. Oncol. 32, 1200–1208 (2014).
Chandhoke, G. et al. Patterns of referral for adjuvant chemotherapy for stage II and III colon cancer: a population-based study. Ann. Surg. Oncol. 23, 2529–2538 (2016).
Booth, C. M. et al. Perioperative chemotherapy for bladder cancer in the general population: are practice patterns finally changing? Urol. Oncol. 36, 89.e13–89.e20 (2017).
Cuffe, S. et al. Adjuvant chemotherapy for non-small-cell lung cancer in the elderly: a population-based study in Ontario, Canada. J. Clin. Oncol. 30, 1813–1821 (2012).
Kankesan, J. et al. Factors associated with referral to medical oncology and subsequent use of adjuvant chemotherapy for non-small-cell lung cancer: a population-based study. Curr. Oncol. 20, 30–37 (2013).
Quirt, J. S., Siemens, D. R., Zaza, K., Mackillop, W. J. & Booth, C. M. Patterns of referral to radiation oncology among patients with bladder cancer: a population-based study. Clin. Oncol. 29, 171–179 (2017).
Kumachev, A., Trudeau, M. E. & Chan, K. K. Associations among socioeconomic status, patterns of care and outcomes in breast cancer patients in a universal health care system: Ontario’s experience. Cancer 122, 893–898 (2016).
Deb, S. et al. The effect of socioeconomic status on gross total resection, radiation therapy and overall survival in patients with gliomas. J. Neurooncol. 132, 447–453 (2017).
Huang, J. et al. Factors affecting the use of palliative radiotherapy in Ontario. J. Clin. Oncol. 19, 137–144 (2001).
Jacobs, L. K., Kelley, K. A., Rosson, G. D., Detrani, M. E. & Chang, D. C. Disparities in urban and rural mastectomy populations: the effects of patient- and county-level factors on likelihood of receipt of mastectomy. Ann. Surg. Oncol. 15, 2644–2652 (2008).
Atkins, G. T., Kim, T. & Munson, J. Residence in rural areas of the United States and lung cancer mortality. Disease incidence, treatment disparities, and stage-specific survival. Ann. Am. Thorac. Soc. 14, 403–411 (2017).
Xu, L., Kim, Y., Spolverato, G., Gani, F. & Pawlik, T. M. Racial disparities in treatment and survival of patients with hepatocellular carcinoma in the United States. Hepatobiliary Surg. Nutr. 5, 43–52 (2016).
Lawrenson, R. et al. Treatment and survival disparities by ethnicity in New Zealand women with stage I-III breast cancer tumour subtypes. Cancer Causes Control 28, 1417–1427 (2017).
Tyldesley, S. et al. Association between age and the utilization of radiotherapy in Ontario. Int. J. Radiat. Oncol. Biol. Phys. 47, 469–480 (2000).
Magrath, I. et al. Paediatric cancer in low-income and middle-income countries. Lancet Oncol. 14, e104–e116 (2013).
Duthey, B. & Scholten, W. Adequacy of opioid analgesic consumption at country, global, and regional levels in 2010, its relationship with development level, and changes compared with 2006. J. Pain Symptom Manage. 47, 283–297 (2014).
McLaughlin, P. Y. et al. Do radiation oncology outreach clinics affect the use of radiotherapy? Radiother. Oncol. 127, 143–149 (2018).
Robinson, D. et al. Waiting times for radiotherapy: variation over time and between cancer networks in southeast England. Br. J. Cancer 92, 1201–1208 (2005).
Hershman, D. et al. Timing of adjuvant chemotherapy initiation after surgery for stage III colon cancer. Cancer 107, 2581–2588 (2006).
Wasserman, D. W. et al. Reasons for delay in time to initiation of adjuvant chemotherapy for colon cancer. J. Oncol. Pract. 11, e28–e35 (2014).
Arndt, V. et al. Patient delay and stage of diagnosis among breast cancer patients in Germany — a population based study. Br. J. Cancer 86, 1034–1040 (2002).
McKenzie, F. et al. Drivers of advanced stage at breast cancer diagnosis in the multicountry African breast cancer — disparities in outcomes (ABC-DO) study. Int. J. Cancer 142, 1568–1579 (2018).
Simos, D. et al. Imaging for distant metastases in women with early-stage breast cancer: a population-based cohort study. CMAJ 187, E387–E397 (2015).
Parmar, A. D. et al. Quality of post-treatment surveillance of early stage breast cancer in Texas. Surgery 154, 214–225 (2013).
Arnaout, A. et al. Use of preoperative magnetic resonance imaging for breast cancer: a Canadian population-based study. JAMA Oncol. 1, 1238–1250 (2015).
Ashworth, A., Kong, W., Chow, E. & Mackillop, W. J. Fractionation of palliative radiation therapy for bone metastases in Ontario: do practice guidelines guide practice? Int. J. Radiat. Oncol. Biol. Phys. 94, 31–39 (2016).
Ashworth, A., Kong, W., Whelan, T. & Mackillop, W. J. A population-based study of the fractionation of postlumpectomy breast radiation therapy. Int. J. Radiat. Oncol. Biol. Phys. 86, 51–57 (2013).
Schnipper, L. E. et al. American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: the top five list for oncology. J. Clin. Oncol. 30, 1715–1724 (2012).
Mitera, G. et al. Choosing Wisely Canada cancer list: ten low-value or harmful practices that should be avoided in cancer care. J. Oncol. Pract. 11, e296–e303 (2015).
Olson, R. A. et al. Impact of using audit data to improve the evidence-based use of single-fraction radiation therapy for bone metastases in British Columbia. Int. J. Radiat. Oncol. Biol. Phys. 94, 40–47 (2016).
Tyldesley, S., Boyd, C., Schulze, K., Walker, H. & Mackillop, W. J. Estimating the need for radiotherapy for lung cancer: an evidence-based, epidemiologic approach. Int. J. Radiat. Oncol. Biol. Phys. 49, 973–985 (2001).
Round, C. E. et al. Radiotherapy demand and activity in England 2006–2020. Clin. Oncol. 25, 522–530 (2013).
Mackillop, W. J. et al. A comparison of evidence-based estimates and empirical benchmarks of the appropriate rate of use of radiation therapy in Ontario. Int. J. Radiat. Oncol. Biol. Phys. 91, 1099–1107 (2015).
Barbera, L., Zhang-Salomons, J., Huang, J., Tyldesley, S. & Mackillop, W. Defining the need for radiotherapy for lung cancer in the general population: a criterion-based, benchmarking approach. Med. Care 41, 1074–1085 (2003).
Cancer Quality Council of Ontario. Access to radiation treatment. Cancer System Quality Index https://www.csqi.on.ca/indicators/access-radiation-treatment (2017).
Rahal, R., Klein-Geltink, J., Forte, T., Lockwood, G. & Bryant, H. Measuring concordance with guidelines for the diagnosis and treatment of colon cancer. Curr. Oncol. 20, 227–229 (2013).
Salloum, R. G., Smith, T. J., Jensen, G. A. & Lafata, J. E. Factors associated with adherence to chemotherapy guidelines in patients with non-small cell lung cancer. Lung Cancer 75, 255–260 (2012).
Young, J. M. et al. Concordance with national guidelines for colorectal cancer care in New South Wales: a population-based patterns of care study. Med. J. Aust. 186, 292–295 (2007).
Wong, S. J. et al. Longitudinal Oncology Registry of Head and Neck Carcinoma (LORHAN): analysis of chemoradiation treatment approaches in the United States. Cancer 117, 1679–1686 (2011).
Nandi, M., Mandal, A. & Asthana, A. K. Audit of cancer patients from eastern Uttar Pradesh (UP), India: a university hospital based two year retrospective analysis. Asian Pac. J. Cancer Prev. 14, 4993–4998 (2013).
Porter, G. A., Urquhart, R., Bu, J., Johnson, P. & Grunfeld, E. The impact of audit and feedback on nodal harvest in colorectal cancer. BMC Cancer 11, 2 (2011).
Alvarez, E. et al. Improvement in treatment abandonment in pediatric patients with cancer in Guatemala. Pediatr. Blood Cancer 64, 10 (2017).
Gadalla, S. M. et al. A population-based assessment of mortality and morbidity patterns among patients with thymoma. Int. J. Cancer 128, 2688–2694 (2011).
Lee, Y. Y. et al. Incidence and outcomes of pregnancy-associated cancer in Australia, 1994-2008: a population-based linkage study. BJOG 119, 1572–1582 (2012).
Chu, C. N. et al. Increase in stroke risk in patients with head and neck cancer: a retrospective cohort study. Br. J. Cancer 105, 1419–1423 (2011).
Marmor, S., Burke, E. E., Virnig, B. A., Jensen, E. H. & Tuttle, T. M. A comparative analysis of survival outcomes between pancreatectomy and chemotherapy for elderly patients with adenocarcinoma of the pancreas. Cancer 122, 3378–3385 (2016).
Paszat, L. F., Mackillop, W. J., Groome, P. A., Schulze, K. & Holowaty, E. Mortality from myocardial infarction following postlumpectomy radiotherapy for breast cancer: a population-based study in Ontario, Canada. Int. J. Radiat. Oncol. Biol. Phys. 43, 755–762 (1999).
Seruga, B., Sterling, L., Wang, L. & Tannock, I. F. Reporting of serious adverse drug reactions of targeted anticancer agents in pivotal phase III clinical trials. J. Clin. Oncol. 29, 174–185 (2011).
Groome, P. A. et al. Management and outcome of glottic cancer: a population-based comparison between Ontario, Canada and the SEER areas of the United States. Surveillance, epidemiology and end results. J. Otolaryngol. 29, 67–77 (2000).
Capri, S. & Russo, A. Cost of breast cancer based on real-world data: a cancer registry study in Italy. BMC Health Serv. Res. 17, 84 (2017).
Siemens, D. R. et al. Processes of care and the impact of surgical volumes on cancer-specific survival: a population-based study in bladder cancer. Urology 84, 1049–1057 (2014).
Booth, C. M., Nanji, S., Wei, X. & Mackillop, W. J. Management and outcome of colorectal cancer liver metastases in elderly patients: a population-based study. JAMA Oncol. 1, 1111–1119 (2015).
Patel, M. I., Bang, A., Gillett, D., Cheluvappa, R. & Smith, D. P. Poor survival of females with bladder cancer is limited to those aged 70 years or over: a population-wide linkage study, New South Wales, Australia. Cancer Med. 4, 1145–1152 (2015).
Walker, M. et al. A call for theory-informed approaches to knowledge translation studies: an example of chemotherapy for bladder cancer. Curr. Oncol. 22, 178–181 (2015).
Templeton, A. J. et al. Translating clinical trials to clinical practice: outcomes of men with metastatic castration resistant prostate cancer treated with docetaxel and prednisone in and out of clinical trials. Ann. Oncol. 24, 2972–2977 (2013).
Booth, C. M. et al. Curative therapy for bladder cancer in routine clinical practice: a population-based outcomes study. Clin. Oncol. 26, 506–514 (2014).
Stein, J. P. et al. Radical cystectomy in the treatment of invasive bladder cancer: long-term results in 1,054 patients. J. Clin. Oncol. 19, 666–675 (2001).
Lau, K. et al. The effect of a regional hepatopancreaticobiliary surgical program on clinical volume, quality of cancer care, and outcomes in the Veterans Affairs system. JAMA Surg. 149, 1153–1161 (2014).
Simunovic, M. et al. Assessing the volume-outcome hypothesis and region-level quality improvement interventions: pancreas cancer surgery in two Canadian Provinces. Ann. Surg. Oncol. 17, 2537–2544 (2010).
van Roest, M. H., van der Aa, M. A., van der Geest, L. G. & de Jong, K. P. The impact of socioeconomic status, surgical resection and type of hospital on survival in patients with pancreatic cancer. A population-based study in The Netherlands. PLOS ONE 11, e0166449 (2016).
Bendzsak, A. M., Baxter, N. N., Darling, G. E., Austin, P. C. & Urbach, D. R. Regionalization and outcomes of lung cancer surgery in Ontario, Canada. J. Clin. Oncol. 35, 2772–2780 (2017).
Nienhuijs, S. W. et al. Reduction of in-hospital mortality following regionalisation of pancreatic surgery in the south-east of the Netherlands. Eur. J. Surg. Oncol. 36, 652–656 (2010).
De Angelis, R. et al. The EUROCARE-4 database on cancer survival in Europe: data standardisation, quality control and methods of statistical analysis. Eur. J. Cancer 45, 909–930 (2009).
Thomson, C. S. & Forman, D. Cancer survival in England and the influence of early diagnosis: what can we learn from recent EUROCARE results? Br. J. Cancer 101 (Suppl. 2), 102–109 (2009).
Lozano, R. et al. Benchmarking of performance of Mexican states with effective coverage. Lancet 368, 1729–1741 (2006).
Zubizarreta, E. H., Fidarova, E., Healy, B. & Rosenblatt, E. Need for radiotherapy in low and middle income countries — the silent crisis continues. Clin. Oncol. 27, 107–114 (2015).
Nandakumar, A. et al. Concurrent chemoradiation for cancer of the cervix: results of a multi-institutional study from the setting of a developing country (India). J. Glob. Oncol. 1, 11–22 (2015).
Veenstra, C. M. et al. Long-term economic and employment outcomes among partners of women with early-stage breast cancer. J. Oncol. Pract. 13, e916–e926 (2017).
Lin, P. J. et al. Linking costs and survival in the treatment of older adults with chronic myeloid leukemia: an analysis of SEER-Medicare data from 1995 to 2007. Med. Care 54, 380–385 (2016).
Roehrborn, C. G. & Black, L. K. The economic burden of prostate cancer. BJU Int. 108, 806–813 (2011).
Mullins, C. D., Hsiao, F. Y., Onukwugha, E., Pandya, N. B. & Hanna, N. Comparative and cost-effectiveness of oxaliplatin-based or irinotecan-based regimens compared with 5-fluorouracil/leucovorin alone among US elderly stage IV colon cancer patients. Cancer 118, 3173–3181 (2012).
Mackillop, W. J. in Clinical Radiation Oncology 3rd edn (eds Gunderson, L. & Tepper, J.) 203–222 (Churchill Livingstone, Philadelphia, 2012).
Ong, M. B. H. Flatiron, BMS form collaboration to curate regulatory-grade real-world data. The Cancer Letter https://cancerletter.com/articles/20180504_1/ (2018).
Goodwin, P. J., Ballman, K. V., Small, E. J. & Cannistra, S. A. Evaluation of treatment benefit in journal of clinical oncology. J. Clin. Oncol. 31, 1123–1124 (2013).
Chang, G. J. Is there validity in propensity score-matched estimates of adjuvant chemotherapy effects for patients with rectal cancer? JAMA Oncol. 4, 921–923 (2018).
Choudhury, A. & Hoskin, P. J. Bladder cancer and the National Cancer Data Base: new insight or misinformation? Cancer 124, 1105–1107 (2018).
Unger, J. M., Cook, E., Tai, E. & Bleyer, A. The role of clinical trial participation in cancer research: barriers, evidence, and strategies. Am. Soc. Clin. Oncol. Educ. Book 35, 185–198 (2016).
The Children’s Oncology Group. What is a clincial trial? COG https://childrensoncologygroup.org/index.php/what-is-a-clinical-trial (2018).
Yusuf, S., Collins, R. & Peto, R. Why do we need some large, simple randomized trials? Stat. Med. 3, 409–422 (1984).
Parmar, M. K. et al. Paclitaxel plus platinum-based chemotherapy versus conventional platinum-based chemotherapy in women with relapsed ovarian cancer: the ICON4/AGO-OVAR-2.2 trial. Lancet 361, 2099–2106 (2003).
James, N. D. et al. Addition of docetaxel, zoledronic acid, or both to first-line long-term hormone therapy in prostate cancer (STAMPEDE): survival results from an adaptive, multiarm, multistage, platform randomised controlled trial. Lancet 387, 1163–1177 (2016).
Khozin, S., Blumenthal, G. M. & Pazdur, R. Real-world data for clinical evidence generation in oncology. J. Natl Cancer Inst. 109, djx187 (2017).
Corrigan-Curay, J., Sacks, L. & Woodcock, J. Real-world evidence and real-world data for evaluating drug safety and effectiveness. JAMA 320, 867–868 (2018).
Sackett, D. L. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest 95, 2S–4S (1989).
Oxford Centre for Evidence-Based Medicine. OCEBM levels of evidence. CEBM https://www.cebm.net/2016/05/ocebm-levels-of-evidence/ (2018).
Stukel, T. A. et al. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA 297, 278–285 (2007).
Schneeweiss, S. & Maclure, M. Use of comorbidity scores for control of confounding in studies using administrative databases. Int. J. Epidemiol. 29, 891–898 (2000).
Groome, P. A. & Mackillop, W. J. Uses of ecologic studies in the assessment of intended treatment effects. J. Clin. Epidemiol. 52, 903–904 (1999).
Pearcey, R., Miao, Q., Kong, W., Zhang-Salomons, J. & Mackillop, W. J. Impact of adoption of chemoradiotherapy on the outcome of cervical cancer in Ontario: results of a population-based cohort study. J. Clin. Oncol. 25, 2383–2388 (2007).
Groome, P. A. et al. Management and outcome differences in supraglottic cancer between Ontario, Canada, and the surveillance, epidemiology, and end results areas of the United States. J. Clin. Oncol. 21, 496–505 (2003).
Gupta, S., Kong, W., Booth, C. M. & Mackillop, W. J. Impact of concomitant chemotherapy on outcomes of radiation therapy for head-and-neck cancer: a population-based study. Int. J. Radiat. Oncol. Biol. Phys. 88, 115–121 (2014).
Booth, C. M. et al. Adjuvant chemotherapy for non-small cell lung cancer: practice patterns and outcomes in the general population of Ontario, Canada. J. Thorac. Oncol. 7, 559–566 (2012).
Erlichman, C. Efficacy of adjuvant fluorouracil and folinic acid in B2 colon cancer. J. Clin. Oncol. 17, 1356–1363 (1999).
van Gijn, W. et al. Preoperative radiotherapy combined with total mesorectal excision for resectable rectal cancer: 12-year follow-up of the multicentre, randomised controlled TME trial. Lancet Oncol. 12, 575–582 (2011).
Casadaban, L. et al. Adjuvant chemotherapy is associated with improved survival in patients with stage II colon cancer. Cancer 122, 3277–3287 (2016).
Freischlag, K. et al. Association between incomplete neoadjuvant radiotherapy and survival for patients with locally advanced rectal cancer. JAMA Surg. 152, 558–564 (2017).
Glasziou, P., Vandenbroucke, J. P. & Chalmers, I. Assessing the quality of research. BMJ 328, 39–41 (2004).
Sanderson, S., Tatt, I. D. & Higgins, J. P. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36, 666–676 (2007).
Vandenbroucke, J. P. et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Ann. Intern. Med. 147, W163–W194 (2007).
Wells, G. S. et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm (2009).
Stang, A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur. J. Epidemiol. 25, 603–605 (2010).
Suissa, S. Immortal time bias in pharmaco-epidemiology. Am. J. Epidemiol. 167, 492–499 (2008).
Hill, A. B. The environment and disease: association of causation. Proc. R. Soc. Med. 58, 295–300 (1965).
Rothman, K. J., Greenland, S. & Lash, T. L. Modern epidemiology (Lippincott Williams & Wilkins, Philadelphia, 2008).
Makady, A. et al. Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies. Value Health 20, 520–532 (2017).
Risk Sciences International. Evidence synthesis on post-approval surveillance of approved novel cancer drugs. Canadian Partnership Against Cancer https://content.cancerview.ca/download/cv/quality_and_planning/system_planning/quality_initiatives_microsite/documents/evidence_synthesis_post_approval_surveillance_cancer_drugs_enpdf?attachment=0 (2015).
Jarow, J. P., LaVange, L. & Woodcock, J. Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data. JAMA 318, 703–704 (2017).
Food and Drug Administration. Statement from FDA Commissioner Scott Gottlieb, M. D., on FDA’s new strategic framework to advance use of real-world evidence to support development of drugs and biologics. FDA https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm627760.htm (2018).
Hagen, T. ASCO partners with FDA to incorporate real-world evidence into drug decisions. OncLive http://www.onclive.com/conference-coverage/asco-2017/asco-partners-with-fda-to-incorporate-realworld-evidence-into-drug-decisions (2017).
Hall, P. S. Real-world data for efficient health technology assessment. Eur. J. Cancer 79, 235–237 (2017).
Lewis, J. R., Kerridge, I. & Lipworth, W. Coverage with evidence development and managed entry in the funding of personalized medicine: practical and ethical challenges for oncology. J. Clin. Oncol. 33, 4112–4117 (2015).
Espin, J., Rovira, J. & Garcia, L. Experiences and impact of European risk-sharing schemes focusing on oncology medicines. European Commission http://ec.europa.eu/DocsRoom/documents/7608 (2011).
Fujiwara, Y. & Kobayashi, K. Oncology drug clinical development and approval in Japan: the role of the pharmaceuticals and medical devices evaluation center (PMDEC). Crit. Rev. Oncol. Hematol. 42, 145–155 (2002).
Mackillop, W. J. & Kong, W. Estimating the need for palliative radiation therapy: a benchmarking approach. Int. J. Radiat. Oncol. Biol. Phys. 94, 51–59 (2016).
The work of C.M.B. is supported by his position as the Canada Research Chair in Population Cancer Care. All the authors thank I. Tannock and E. Eisenhauer for their comments on an earlier draft of this manuscript.
Nature Reviews Clinical Oncology thanks A. Abernethy, C. Uyl-de-Groot and the other anonymous reviewer(s) for their contribution to the peer review of this work.
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Booth, C.M., Karim, S. & Mackillop, W.J. Real-world data: towards achieving the achievable in cancer care. Nat Rev Clin Oncol 16, 312–325 (2019). https://doi.org/10.1038/s41571-019-0167-7
Signal Transduction and Targeted Therapy (2020)
Real-World Experience with Ceftolozane-Tazobactam for Multidrug-Resistant Gram-Negative Bacterial Infections
Antimicrobial Agents and Chemotherapy (2020)
Systematic Review of Efficacy and Health Economic Implications of Real-world Treatment Sequencing in Prostate Cancer: Where Do the Newer Agents Enzalutamide and Abiraterone Fit in?
European Urology Focus (2020)
Developing a framework to incorporate real-world evidence in cancer drug funding decisions: the Canadian Real-world Evidence for Value of Cancer Drugs (CanREValue) collaboration
BMJ Open (2020)
Assessing the efficacy‐effectiveness gap for cancer therapies: A comparison of overall survival and toxicity between clinical trial and population‐based, real‐world data for contemporary parenteral cancer therapeutics