Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Avoidable flaws in observational analyses: an application to statins and cancer

Nature Medicine volume 25, pages 1601–1606 (2019)Cite this article

Subjects

Abstract

The increasing availability of large healthcare databases is fueling an intense debate on whether real-world data should play a role in the assessment of the benefit–risk of medical treatments. In many observational studies, for example, statin users were found to have a substantially lower risk of cancer than in meta-analyses of randomized trials. Although such discrepancies are often attributed to a lack of randomization in the observational studies, they might be explained by flaws that can be avoided by explicitly emulating a target trial (the randomized trial that would answer the question of interest). Using the electronic health records of 733,804 UK adults, we emulated a target trial of statins and cancer and compared our estimates with those obtained using previously applied analytic approaches. Over the 10-yr follow-up, 28,408 individuals developed cancer. Under the target trial approach, estimated observational analogs of intention-to-treat and per-protocol 10-yr cancer-free survival differences were −0.5% (95% confidence interval (CI) −1.0%, 0.0%) and −0.3% (95% CI −1.5%, 0.5%), respectively. By contrast, previous analytic approaches yielded estimates that appeared to be strongly protective. Our findings highlight the importance of explicitly emulating a target trial to reduce bias in the effect estimates derived from observational analyses.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Flowchart for selection of eligible individuals from CALIBER for emulating a target trial of statin therapy and cancer risk (1999–2016).
Fig. 2: Standardized cancer-free survival curves comparing statin therapy with no statin therapy, CALIBER, 1999–2016.

Similar content being viewed by others

Data availability

This study is based in part on data from the Clinical Practice Research Datalink obtained under license from the UK Medicines and Healthcare Products Regulatory Agency. The data are provided by patients and collected by the UK National Health Service (NHS) as part of their care and support. The interpretation and conclusions contained in this study are those of the authors alone. Because electronic health records are classified as sensitive data by the UK Data Protection Act, information governance restrictions (to protect patient confidentiality) prevent data sharing via public deposition. Data are available with approval through the individual constituent entities controlling access to the data. Specifically, the primary care data can be requested via application to the Clinical Practice Research Datalink (https://www.cprd.com).

Code availability

Access to the computer code used in this research is available by request to the corresponding author.

References

  1. Hernán, M. A. & Robins, J. M. Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183, 758–764 (2016).

    Article  Google Scholar 

  2. Soni, P. D. et al. Comparison of population-based observational studies with randomized trials in oncology. J. Clin. Oncol. 37, 1209–1216 (2019).

    Article  CAS  Google Scholar 

  3. Visvanathan, K. et al. Untapped potential of observational research to inform clinical decision making: American Society of Clinical Oncology Research Statement. J. Clin. Oncol. 35, 1845–1854 (2017).

    Article  Google Scholar 

  4. Hemingway, H. et al. Big data from electronic health records for early and late translational cardiovascular research: challenges and potential. Eur. Heart J. 39, 1481–1495 (2018).

    Article  Google Scholar 

  5. Gerstein, H. C., McMurray, J. & Holman, R. R. Real-world studies no substitute for RCTs in establishing efficacy. Lancet 393, 210–211 (2019).

    Article  Google Scholar 

  6. Framework for FDA’s Real-World Evidence Program (U.S. Food and Drug Administration, 2018).

  7. Hernán, M. A., Sauer, B. C., Hernandez-Diaz, S., Platt, R. & Shrier, I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J. Clin. Epidemiol. 79, 70–75 (2016).

    Article  Google Scholar 

  8. Graaf, M. R., Beiderbeck, A. B., Egberts, A. C., Richel, D. J. & Guchelaar, H. J. The risk of cancer in users of statins. J. Clin. Oncol. 22, 2388–2394 (2004).

    Article  CAS  Google Scholar 

  9. Poynter, J. N. et al. Statins and the risk of colorectal cancer. N. Engl. J. Med. 352, 2184–2192 (2005).

    Article  CAS  Google Scholar 

  10. Friis, S. et al. Cancer risk among statin users: a population-based cohort study. Int. J. Cancer 114, 643–647 (2005).

    Article  CAS  Google Scholar 

  11. Chen, M. J. et al. Statins and the risk of pancreatic cancer in type 2 diabetic patients—a population-based cohort study. Int. J. Cancer 138, 594–603 (2016).

    Article  CAS  Google Scholar 

  12. Khurana, V., Bejjanki, H. R., Caldito, G. & Owens, M. W. Statins reduce the risk of lung cancer in humans: a large case-control study of US veterans. Chest 131, 1282–1288 (2007).

    Article  Google Scholar 

  13. Clancy, Z. et al. Statins and colorectal cancer risk: a longitudinal study. Cancer Causes Control 24, 777–782 (2013).

    Article  Google Scholar 

  14. Pradelli, D. et al. Statins use and the risk of all and subtype hematological malignancies: a meta-analysis of observational studies. Cancer Med. 4, 770–780 (2015).

    Article  CAS  Google Scholar 

  15. Shannon, J. et al. Statins and prostate cancer risk: a case-control study. Am. J. Epidemiol. 162, 318–325 (2005).

    Article  Google Scholar 

  16. Flick, E. D. et al. Statin use and risk of colorectal cancer in a cohort of middle-aged men in the US: a prospective cohort study. Drugs 69, 1445–1457 (2009).

    Article  CAS  Google Scholar 

  17. Flick, E. D. et al. Statin use and risk of prostate cancer in the California Men’s Health Study cohort. Cancer Epidemiol. Biomark. Prev. 16, 2218–2225 (2007).

    Article  CAS  Google Scholar 

  18. Hoffmeister, M., Chang-Claude, J. & Brenner, H. Individual and joint use of statins and low-dose aspirin and risk of colorectal cancer: a population-based case-control study. Int. J. Cancer 121, 1325–1330 (2007).

    Article  CAS  Google Scholar 

  19. Boudreau, D. M. et al. The association between 3-hydroxy-3-methylglutaryl conenzyme A inhibitor use and breast carcinoma risk among postmenopausal women: a case-control study. Cancer 100, 2308–2316 (2004).

    Article  CAS  Google Scholar 

  20. Cholesterol Treatment Trialists (CTT) Collaboration et al. Lack of effect of lowering LDL cholesterol on cancer: meta-analysis of individual data from 175,000 people in 27 randomised trials of statin therapy. PLoS One 7, e29849 (2012).

  21. Dale, K. M., Coleman, C. I., Henyan, N. N., Kluger, J. & White, C. M. Statins and cancer risk: a meta-analysis. JAMA 295, 74–80 (2006).

    Article  CAS  Google Scholar 

  22. Maisonneuve, P. & Lowenfels, A. B. Statins and the risk of colorectal cancer. N. Engl. J. Med. 353, 952–954 (2005).

    Article  CAS  Google Scholar 

  23. Setoguchi, S., Avorn, J. & Schneeweiss, S. Statins and the risk of colorectal cancer. N. Engl. J. Med. 353, 952–954 (2005).

    Article  CAS  Google Scholar 

  24. Miettinen, O. S. The need for randomization in the study of intended effects. Stat. Med 2, 267–271 (1983).

    Article  CAS  Google Scholar 

  25. Sattar, N. et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet 375, 735–742 (2010).

    Article  CAS  Google Scholar 

  26. Denaxas, S. C. et al. Data resource profile: cardiovascular disease research using linked bespoke studies and electronic health records (CALIBER). Int. J. Epidemiol. 41, 1625–1638 (2012).

    Article  Google Scholar 

  27. Denaxas, S. et al. UK phenomics platform for developing and validating electronic health record phenotypes: CALIBER. J. Am. Med. Inform. Assoc. https://doi.org/10.1093/jamia/ocz105 (2019).

  28. Robins, J. M., Hernán, M. A. & Rotnitzky, A. Effect modification by time-varying covariates. Am. J. Epidemiol. 166, 994–1002 (2007).

    Article  Google Scholar 

  29. Grodstein, F. et al. Postmenopausal estrogen and progestin use and the risk of cardiovascular disease. N. Engl. J. Med. 335, 453–461 (1996).

    Article  CAS  Google Scholar 

  30. Manson, J. E. et al. Estrogen plus progestin and the risk of coronary heart disease. N. Engl. J. Med. 349, 523–534 (2003).

    Article  CAS  Google Scholar 

  31. Hernán, M. A. & Robins, J. M. Authors’ response, part I: observational studies analyzed like randomized experiments: best of both worlds. Epidemiology 19, 789–792 (2008).

    Article  Google Scholar 

  32. Hernán, M. A. et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology 19, 766–779 (2008).

    Article  Google Scholar 

  33. Margulis, A. V. et al. Validation of cancer cases using primary care, cancer registry, and hospitalization data in the United Kingdom. Epidemiology 29, 308–313 (2018).

    Article  Google Scholar 

  34. Herrett, E., Thomas, S. L., Schoonen, W. M., Smeeth, L. & Hall, A. J. Validation and validity of diagnoses in the General Practice Research Database: a systematic review. Br. J. Clin. Pharmacol. 69, 4–14 (2010).

    Article  CAS  Google Scholar 

  35. Bonovas, S., Filioussi, K. & Sitaras, N. M. Statin use and the risk of prostate cancer: a metaanalysis of 6 randomized clinical trials and 13 observational studies. Int. J. Cancer 123, 899–904 (2008).

    Article  CAS  Google Scholar 

  36. Collin, S. M. et al. Prostate-cancer mortality in the USA and UK in 1975–2004: an ecological study. Lancet Oncol. 9, 445–452 (2008).

    Article  Google Scholar 

  37. Mainous, A. G. 3rd, Baker, R., Everett, C. J. & King, D. E. Impact of a policy allowing for over-the-counter statins. Qual. Prim. Care 18, 301–306 (2010).

    PubMed  Google Scholar 

  38. Thompson, W. A. Jr. On the treatment of grouped observations in life studies. Biometrics 33, 463–470 (1977).

    Article  Google Scholar 

  39. Hernán, M. A., Lanoy, E., Costagliola, D. & Robins, J. M. Comparison of dynamic treatment regimes via inverse probability weighting. Basic Clin. Pharmacol. Toxicol. 98, 237–242 (2006).

    Article  Google Scholar 

  40. Herrett, E. et al. Data resource profile: Clinical Practice Research Datalink (CPRD). Int. J. Epidemiol. 44, 827–836 (2015).

    Article  Google Scholar 

  41. O’Neil, M., Payne, C. & Read, J. Read codes version 3: a user led terminology. Methods Inf. Med. 34, 187–192 (1995).

    Article  Google Scholar 

  42. Morley, K. I. et al. Defining disease phenotypes using national linked electronic health records: a case study of atrial fibrillation. PLoS One 9, e110900 (2014).

    Article  Google Scholar 

  43. Kuan, V. et al. A chronological map of 308 physical and mental health conditions from 4 million individuals in the English National Health Service. Lancet Digital Health 1, e63–e77 (2019).

    Article  Google Scholar 

  44. García-Albéniz, X., Hsu, J. & Hernán, M. A. The value of explicitly emulating a target trial when using real world evidence: an application to colorectal cancer screening. Eur. J. Epidemiol. 32, 495–500 (2017).

    Article  Google Scholar 

Download references

Acknowledgements

This research was partly supported by NIH grant P01 CA134294. B.A.D. is supported by an ASISA Fellowship.

Author information

Authors and Affiliations

  1. Department of Epidemiology, Harvard T. H. Chan School of Public Health, Boston, MA, USA

    Barbra A. Dickerman, Xabier García-Albéniz, Roger W. Logan & Miguel A. Hernán

  2. RTI Health Solutions, Barcelona, Spain

    Xabier García-Albéniz

  3. Institute of Health Informatics Research, University College London, London, UK

    Spiros Denaxas

  4. Health Data Research UK (HDR UK) London, University College London, London, UK

    Spiros Denaxas

  5. The Alan Turing Institute, London, UK

    Spiros Denaxas

  6. Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, MA, USA

    Miguel A. Hernán

  7. Harvard-MIT Division of Health Sciences and Technology, Boston, MA, USA

    Miguel A. Hernán

Authors
  1. Barbra A. Dickerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xabier García-Albéniz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roger W. Logan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Spiros Denaxas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Miguel A. Hernán
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

B.A.D., X.G.-A., S.D. and M.A.H. conceived the overall study. B.A.D. analyzed the data. All authors contributed to the design and analyses. R.W.L. provided key input in processing data from the database. All authors contributed to the interpretation of the results. B.A.D. and M.A.H. drafted the manuscript, which was reviewed, revised and approved by all authors.

Corresponding author

Correspondence to Barbra A. Dickerman.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Jennifer Sargent was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1

Estimated hazard ratios for cancer diagnosis comparing statin therapy with no statin therapy, stratified by age, sex and coronary heart disease status, CALIBER, 1999–2016.

Extended Data Fig. 2

Sensitivity analysis with a 60-d, rather than 30-d, maximum gap between successive prescriptions. Estimated hazard ratios for cancer diagnosis comparing statin therapy with no statin therapy, CALIBER, 1999–2016.

Extended Data Fig. 3

Sensitivity analysis additionally adjusting for physical activity, alcohol consumption, family history of cancer, practice region, influenza vaccination in the past year. and cancer screening in the past year. Estimated hazard ratios for cancer diagnosis comparing statin therapy with no statin therapy, CALIBER, 1999–2016.

Extended Data Fig. 4

Sensitivity analysis adjusting for ever-diagnosis (i.e., having ever received a diagnosis) with cardiovascular disease and diabetes by carrying forward indicators. Estimated hazard ratios for cancer diagnosis comparing statin therapy with no statin therapy, CALIBER, 1999–2016.

Extended Data Fig. 5

Sensitivity analysis truncating weights at their 99.5th percentile. Estimated hazard ratios for cancer diagnosis comparing statin therapy with no statin therapy, CALIBER, 1999–2016.

Extended Data Fig. 6

Sensitivity analysis additionally applying weights for censoring due to loss to follow-up. Estimated hazard ratios for cancer diagnosis comparing statin therapy with no statin therapy, CALIBER, 1999–2016.

Extended Data Fig. 7

Estimated hazard ratios and 95% confidence intervals for total cancer diagnosis and type 2 diabetes diagnosis comparing statin therapy with no statin therapy, when emulating a target trial and when replicating the approach of previous observational analyses, CALIBER, 1999–2016.

Extended Data Fig. 8

Covariates used in the primary and sensitivity analyses when emulating a target trial of statin therapy and cancer risk, CALIBER, 1999–2016.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dickerman, B.A., García-Albéniz, X., Logan, R.W. et al. Avoidable flaws in observational analyses: an application to statins and cancer. Nat Med 25, 1601–1606 (2019). https://doi.org/10.1038/s41591-019-0597-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-019-0597-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer