Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Assessing risk in chronic kidney disease: a methodological review

Nature Reviews Nephrology volume 9, pages 18–25 (2013)Cite this article

Subjects

Abstract

Chronic kidney disease (CKD) is an increasingly common public health issue associated with substantial morbidity and mortality. Risk prediction models provide a useful clinical and research framework for forecasting the probability of adverse events and stratifying patients with CKD according to risk; however, accurate absolute risk prediction requires careful model specification. Competing events that preclude the event of interest (for example, death in studies of end-stage renal disease) must be taken into account. Functional forms of predictor variables and underlying effect modification must be accurately specified; nonlinearity and possible interactions should be evaluated. The potential effect of measurement error should also be considered. Misspecification of any of these components can dramatically affect absolute risk prediction. Evaluation of prognostic models should encompass not only traditional tests of calibration and discrimination, such as the Hosmer–Lemeshow test of 'goodness of fit' and the area under the receiver operating curve, but also newer metrics, such as risk reclassification tables and net reclassification indices. The latter two tests are particularly useful when considering the addition of novel predictors to established models. Finally, models of absolute risk prediction should be internally and externally validated as they typically generalize only to populations with similar baseline characteristics and rates of competing events.

Key Points

  • Chronic kidney disease (CKD) is associated with an increased risk of adverse outcomes, including end-stage renal disease, cardiovascular disease, and death

  • Models of absolute risk prediction can help inform counselling strategies, referral strategies, and the evaluation of interventions for patients with CKD

  • Risk prediction models require thoughtful specification of timescale, functional form, effect modification, and competing risk events in order to provide accurate risk forecasts

  • New risk prediction models may be assessed by standard measures of calibration and discrimination; however, tools such as reclassification tables and net reclassification indices may be more informative

  • Risk prediction models should be internally and externally validated where possible

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Improper accounting for competing events overestimates absolute risk.
Figure 2: Nonlinear associations between variables and events can be evaluated by expanding the variable into multiple terms, such as piecewise linear splines.
Figure 3: The c-statistic is relatively insensitive to the addition of a new marker Y to an existing risk score X, particularly when the existing model displays strong association with the outcome.

Similar content being viewed by others

References

  1. Coresh, J. et al. Prevalence of chronic kidney disease in the United States. JAMA 298, 2038–2047 (2007).

    Article  CAS  Google Scholar 

  2. U S Renal Data System. USRDS 2011 Annual Data Report: atlas of chronic kidney disease and end-stage renal disease in the United States [online], (2011).

  3. Go, A. S., Chertow, G. M., Fan, D., McCulloch, C. E. & Hsu, C. Y. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N. Engl. J. Med. 351, 1296–1305 (2004).

    Article  CAS  Google Scholar 

  4. Astor, B. C. et al. Lower estimated glomerular filtration rate and higher albuminuria are associated with mortality and end-stage renal disease. A collaborative meta-analysis of kidney disease population cohorts. Kidney Int. 79, 1331–1340 (2011).

    Article  CAS  Google Scholar 

  5. Gansevoort, R. T. et al. Lower estimated GFR and higher albuminuria are associated with adverse kidney outcomes. A collaborative meta-analysis of general and high-risk population cohorts. Kidney Int. 80, 93–104 (2011).

    Article  CAS  Google Scholar 

  6. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. The Diabetes Control and Complications Trial Research Group. N. Engl. J. Med. 329, 977–986 (1993).

  7. Jafar, T. H. et al. Proteinuria as a modifiable risk factor for the progression of non-diabetic renal disease. Kidney Int. 60, 1131–1140 (2001).

    Article  CAS  Google Scholar 

  8. Giatras, I., Lau, J. & Levey, A. S. Effect of angiotensin-converting enzyme inhibitors on the progression of nondiabetic renal disease: a meta-analysis of randomized trials. Angiotensin-Converting-Enzyme Inhibition and Progressive Renal Disease Study Group. Ann. Intern. Med. 127, 337–345 (1997).

    Article  CAS  Google Scholar 

  9. Kasiske, B. L., Lakatua, J. D., Ma, J. Z. & Louis, T. A. A meta-analysis of the effects of dietary protein restriction on the rate of decline in renal function. Am. J. Kidney Dis. 31, 954–961 (1998).

    Article  CAS  Google Scholar 

  10. Writing Team for the Diabetes Control and Complications Trial/Epidemiology of Diabetes Interventions and Complications Research Group. Sustained effect of intensive treatment of type 1 diabetes mellitus on development and progression of diabetic nephropathy: the Epidemiology of Diabetes Interventions and Complications (EDIC) study. JAMA 290, 2159–2167 (2003).

  11. Brenner, B. M. et al. Effects of losartan on renal and cardiovascular outcomes in patients with type 2 diabetes and nephropathy. N. Engl. J. Med. 345, 861–869 (2001).

    Article  CAS  Google Scholar 

  12. Stern, L. Fibroblast growth factor 23, cardiovascular disease, and inflammation. Clin. J. Am. Soc. Nephrol. 7, 1061–1062 (2012).

    Article  CAS  Google Scholar 

  13. Matsushita, K. et al. Association of estimated glomerular filtration rate and albuminuria with all-cause and cardiovascular mortality in general population cohorts: a collaborative meta-analysis. Lancet 375, 2073–2081 (2010).

    Article  Google Scholar 

  14. French, B., Saha-Chaudhuri, P., Ky, B., Cappola, T. P. & Heagerty, P. J. Development and evaluation of multi-marker risk scores for clinical prognosis. Stat. Methods Med. Res. http://dx.doi.org/10.1177/0962280212451881.

  15. Grams, M. E. et al. vascular disease, ESRD, and death: interpreting competing risk analyses. Clin. J. Am. Soc. Nephrol. 7, 1606–1614 (2012).

    Article  Google Scholar 

  16. Rothman, K. J., Greenland, S., Lash, T. L. in Modern Epidemiology (Lippincott Williams & Wilkins, Philadelphia, PA, 2008).

    Google Scholar 

  17. Berry, J. D. et al. Lifetime risks of cardiovascular disease. N. Engl. J. Med. 366, 321–329 (2012).

    Article  CAS  Google Scholar 

  18. Szklo, M. & Nieto, F. J. in Epidemiology: beyond the basics (Jones and Bartlett Publishers, Sudbury, Massachusetts, 2007).

    Google Scholar 

  19. Narayan, K. M., Boyle, J. P., Thompson, T. J., Sorensen, S. W. & Williamson, D. F. Lifetime risk for diabetes mellitus in the United States. JAMA 290, 1884–1890 (2003).

    Article  CAS  Google Scholar 

  20. Royston, P. & Lambert, P. C. in Flexible parametric survival analysis using Stata: Beyond the Cox model (Stata Press, College Station, TX, 2011).

    Google Scholar 

  21. Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults. Executive summary of the third report of the national cholesterol education program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (adult treatment panel iii). JAMA 285, 2486–2497 (2001).

  22. Wilson, P. W. et al. Prediction of coronary heart disease using risk factor categories. Circulation 97, 1837–1847 (1998).

    Article  CAS  Google Scholar 

  23. Tzoulaki, I., Liberopoulos, G. & Ioannidis, J. P. Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 302, 2345–2352 (2009).

    Article  CAS  Google Scholar 

  24. Strikas, R. A. et al. US civilian smallpox preparedness and response program, 2003. Clin. Infect. Dis. 46 (Suppl. 3), S157–S167 (2008).

    Article  Google Scholar 

  25. Gail, M. H. et al. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J. Natl Cancer Inst. 81, 1879–1886 (1989).

    Article  CAS  Google Scholar 

  26. Thakar, C. V., Arrigain, S., Worley, S., Yared, J. P. & Paganini, E. P. A clinical score to predict acute renal failure after cardiac surgery. J. Am. Soc. Nephrol. 16, 162–168 (2005).

    Article  Google Scholar 

  27. Wolbers, M., Koller, M. T., Witteman, J. C. & Steyerberg, E. W. Prognostic models with competing risks: methods and application to coronary risk prediction. Epidemiology 20, 555–561 (2009).

    Article  Google Scholar 

  28. Turin, T. C. et al. Lifetime risk of ESRD. J. Am. Soc. Nephrol. 23, 1569–1578 (2012).

    Article  Google Scholar 

  29. Keith, D. S., Nichols, G. A., Gullion, C. M., Brown, J. B. & Smith, D. H. Longitudinal follow-up and outcomes among a population with chronic kidney disease in a large managed care organization. Arch. Intern. Med. 164, 659–663 (2004).

    Article  Google Scholar 

  30. Prentice, R. L. et al. The analysis of failure times in the presence of competing risks. Biometrics 34, 541–554 (1978).

    Article  CAS  Google Scholar 

  31. Fine, J. P. & Gray, R. J. A proportional hazards model for the subdistribution of a competing risk. J. Am. Stat. Assoc. 94, 496–509 (1999).

    Article  Google Scholar 

  32. Steyerberg, E. W. in Clinical Prediction Models: a practical approach to development, validation, and updating (Springer, New York, NY, 2008).

  33. Pintilie, M. Analysing and interpreting competing risk data. Stat. Med. 26, 1360–1367 (2007).

    Article  Google Scholar 

  34. Hemmelgarn, B. R. et al. Rates of treated and untreated kidney failure in older vs younger adults. JAMA 307, 2507–2515 (2012).

    Article  CAS  Google Scholar 

  35. Harrell, F. E. Jr, Lee, K. L. & Mark, D. B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat. Med. 15, 361–387 (1996).

    Article  Google Scholar 

  36. Kottgen, A. et al. Reduced kidney function as a risk factor for incident heart failure: the atherosclerosis risk in communities (ARIC) study. J. Am. Soc. Nephrol. 18, 1307–1315 (2007).

    Article  CAS  Google Scholar 

  37. Harrell, F. E. Jr, Lee, K. L. & Pollock, B. G. Regression models in clinical studies: determining relationships between predictors and response. J. Natl Cancer Inst. 80, 1198–1202 (1988).

    Article  Google Scholar 

  38. Grambsch, P. M., Therneau, T. M. & Fleming, T. R. Diagnostic plots to reveal functional form for covariates in multiplicative intensity models. Biometrics 51, 1469–1482 (1995).

    Article  CAS  Google Scholar 

  39. Fox, C. S. et al. Associations of kidney disease measures with mortality and end-stage renal disease in individuals with and without diabetes: a meta-analysis. Lancet http://dx.doi.org/10.1016/S0140-6736(12)61350-6.

  40. Mahmoodi, B. K. et al. Associations of kidney disease measures with mortality and end-stage renal disease in individuals with and without hypertension: a meta-analysis. Lancet http://dx.doi.org/10.1016/S0140-6736(12)61272-0.

  41. Hallan, S. I. et al. Age and association of kidney measures with mortality and end-stage renal disease. JAMA http://dx.doi:10.1001/jama.2012.16817.

  42. Cox, C., Chu, H., Schneider, M. F. & Munoz, A. Parametric survival analysis and taxonomy of hazard functions for the generalized gamma distribution. Stat. Med. 26, 4352–4374 (2007).

    Article  Google Scholar 

  43. Coresh, J. et al. Calibration and random variation of the serum creatinine assay as critical elements of using equations to estimate glomerular filtration rate. Am. J. Kidney Dis. 39, 920–929 (2002).

    Article  CAS  Google Scholar 

  44. Larsson, A., Hansson, L. O., Flodin, M., Katz, R. & Shlipak, M. G. Calibration of the Siemens cystatin C immunoassay has changed over time. Clin. Chem. 57, 777–778 (2011).

    Article  CAS  Google Scholar 

  45. Ferguson, M. A. & Waikar, S. S. Established and emerging markers of kidney function. Clin. Chem. 58, 680–689 (2012).

    Article  CAS  Google Scholar 

  46. Selvin, E. et al. Calibration of cystatin C in the National Health and Nutrition Examination Surveys. Am. J. Kidney Dis. (in press).

  47. White, C. A. et al. The impact of interlaboratory differences in cystatin C assay measurement on glomerular filtration rate estimation. Clin. J. Am. Soc. Nephrol. 6, 2150–2156 (2011).

    Article  CAS  Google Scholar 

  48. Selvin, E. et al. Calibration of serum creatinine in the National Health and Nutrition Examination Surveys (NHANES) 1988–1994, 1999–2004. Am. J. Kidney Dis. 50, 918–926 (2007).

    Article  CAS  Google Scholar 

  49. Stevens, L. A., Coresh, J., Greene, T. & Levey, A. S. Assessing kidney function--measured and estimated glomerular filtration rate. N. Engl. J. Med. 354, 2473–2483 (2006).

    Article  CAS  Google Scholar 

  50. Stevens, L. A. et al. Evaluation of the modification of diet in renal disease study equation in a large diverse population. J. Am. Soc. Nephrol. 18, 2749–2757 (2007).

    Article  Google Scholar 

  51. Rodondi, N. et al. Framingham risk score and alternatives for prediction of coronary heart disease in older adults. PLoS ONE 7, e34287 (2012).

    Article  CAS  Google Scholar 

  52. D'Agostino, R. B. S., Grundy, S., Sullivan, L. M., Wilson, P. & CHD Risk Prediction Group. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA 286, 180–187 (2001).

    Article  Google Scholar 

  53. Gail, M. H. & Pfeiffer, R. M. On criteria for evaluating models of absolute risk. Biostatistics 6, 227–239 (2005).

    Article  Google Scholar 

  54. Steyerberg, E. W. et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21, 128–138 (2010).

    Article  Google Scholar 

  55. Justice, A. C., Covinsky, K. E. & Berlin, J. A. Assessing the generalizability of prognostic information. Ann. Intern. Med. 130, 515–524 (1999).

    Article  CAS  Google Scholar 

  56. Inker, L. A. et al. Estimating glomerular filtration rate from serum creatinine and cystatin C. N. Engl. J. Med. 367, 20–29 (2012).

    Article  CAS  Google Scholar 

  57. Hosmer, D. W., Hosmer, T., Le Cessie, S. & Lemeshow, S. A comparison of goodness-of-fit tests for the logistic regression model. Stat. Med. 16, 965–980 (1997).

    Article  CAS  Google Scholar 

  58. Vittinghoff, E., Glidden, D. V., Shiboski, S. C. & McCulloch, C. E. in Regression Methods in Biostatistics: linear, logistic, survival, and repeated measures models (Springer, Breinigsville, PA, 2004).

    Google Scholar 

  59. Chambless, L. E. & Diao, G. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat. Med. 25, 3474–3486 (2006).

    Article  Google Scholar 

  60. Matsushita, K. et al. Comparison of risk prediction using the CKD–EPI equation and the MDRD study equation for estimated glomerular filtration rate. JAMA 307, 1941–1951 (2012).

    Article  CAS  Google Scholar 

  61. Folsom, A. R. et al. An assessment of incremental coronary risk prediction using C-reactive protein and other novel risk markers: the atherosclerosis risk in communities study. Arch. Intern. Med. 166, 1368–1373 (2006).

    Article  CAS  Google Scholar 

  62. Cook, N. R. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin. Chem. 54, 17–23 (2008).

    Article  CAS  Google Scholar 

  63. Pepe, M. S., Janes, H., Longton, G., Leisenring, W. & Newcomb, P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am. J. Epidemiol. 159, 882–890 (2004).

    Article  Google Scholar 

  64. Cook, N. R. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115, 928–935 (2007).

    Article  Google Scholar 

  65. Cook, N. R. & Ridker, P. M. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann. Intern. Med. 150, 795–802 (2009).

    Article  Google Scholar 

  66. Pencina, M. J., D'Agostino, R. B. Sr, D'Agostino, R. B. Jr & Vasan, R. S. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat. Med. 27, 157–172; discussion 207–212 (2008).

    Article  Google Scholar 

  67. Chambless, L. E., Cummiskey, C. P. & Cui, G. Several methods to assess improvement in risk prediction models: extension to survival analysis. Stat. Med. 30, 22–38 (2011).

    Article  Google Scholar 

  68. Dalton, J. E. & Kattan, M. W. Recent advances in evaluating the prognostic value of a marker. Scand. J. Clin. Lab. Invest. Suppl. 242, 59–62 (2010).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Medicine, Department of Epidemiology, Johns Hopkins University, 1830 East Monument Street, Suite 416, Baltimore, 21205, MD, USA

    Morgan E. Grams

  2. Department of Medicine, Department of Epidemiology and Department of Biostatistics, Johns Hopkins University, 2024 East Monument Street, Suite 2–630, Baltimore, 21287, MD, USA

    Josef Coresh

Authors
  1. Morgan E. Grams
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Josef Coresh
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors contributed equally to researching data for the article, discussion of content, and review and editing the manuscript before submission. M. E. Grams wrote the first draft.

Corresponding author

Correspondence to Morgan E. Grams.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Grams, M., Coresh, J. Assessing risk in chronic kidney disease: a methodological review. Nat Rev Nephrol 9, 18–25 (2013). https://doi.org/10.1038/nrneph.2012.248

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrneph.2012.248

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing