Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The importance of prediction model validation and assessment in obesity and nutrition research

Abstract

Deriving statistical models to predict one variable from one or more other variables, or predictive modeling, is an important activity in obesity and nutrition research. To determine the quality of the model, it is necessary to quantify and report the predictive validity of the derived models. Conducting validation of the predictive measures provides essential information to the research community about the model. Unfortunately, many articles fail to account for the nearly inevitable reduction in predictive ability that occurs when a model derived on one data set is applied to a new data set. Under some circumstances, the predictive validity can be reduced to nearly zero. In this overview, we explain why reductions in predictive validity occur, define the metrics commonly used to estimate the predictive validity of a model (for example, coefficient of determination (R2), mean squared error, sensitivity, specificity, receiver operating characteristic and concordance index) and describe methods to estimate the predictive validity (for example, cross-validation, bootstrap, and adjusted and shrunken R2). We emphasize that methods for estimating the expected reduction in predictive ability of a model in new samples are available and this expected reduction should always be reported when new predictive models are introduced.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

References

  1. Heshka S, Feld K, Yang MU, Allison DB, Heymsfield SB . Resting energy expenditure in the obese: a cross-validation and comparison of prediction equations. J Am Diet Assoc 1993; 93: 1031–1036.

    Article  CAS  Google Scholar 

  2. Tzoulaki I, Liberopoulos G, Ioannidis JP . Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 2009; 302: 2345–2352.

    Article  CAS  Google Scholar 

  3. Baan CA, Ruige JB, Stolk RP, Witteman JC, Dekker JM, Heine RJ et al. Performance of a predictive model to identify undiagnosed diabetes in a health care setting. Diabetes Care 1999; 22: 213–219.

    Article  CAS  Google Scholar 

  4. Dixon JB, Chuang LM, Chong K, Chen SC, Lambert GW, Straznicky NE et al. Predicting the glycemic response to gastric bypass surgery in patients with type 2 diabetes. Diabetes Care 2013; 36: 20–26.

    Article  CAS  Google Scholar 

  5. Forns X, Ampurdanes S, Llovet JM, Aponte J, Quinto L, Martinez-Bauer E et al. Identification of chronic hepatitis C patients without hepatic fibrosis by a simple predictive model. Hepatology 2002; 36: 986–992.

    Article  Google Scholar 

  6. Hayes MT, Hunt LA, Foo J, Tychinskaya Y, Stubbs RS . A model for predicting the resolution of type 2 diabetes in severely obese subjects following Roux-en Y gastric bypass surgery. Obes Surg 2011; 21: 910–916.

    Article  Google Scholar 

  7. Li S, Zhao JH, Luan J, Luben RN, Rodwell SA, Khaw KT et al. Cumulative effects and predictive value of common obesity-susceptibility variants identified by genome-wide association studies. Am J Clin Nutr 2010; 91: 184–190.

    Article  CAS  Google Scholar 

  8. Thomas DM, Ivanescu AE, Martin CK, Heymsfield SB, Marshall K, Bodrato VE et al. Predicting successful long-term weight loss from short-term weight-loss outcomes: new insights from a dynamic energy balance model (the POUNDS Lost study). Am J Clin Nutr 2015; 101: 449–454.

    Article  CAS  Google Scholar 

  9. Cancello R, Tordjman J, Poitou C, Guilhem G, Bouillot JL, Hugol D et al. Increased infiltration of macrophages in omental adipose tissue is associated with marked hepatic lesions in morbid human obesity. Diabetes 2006; 55: 1554–1561.

    Article  CAS  Google Scholar 

  10. Chen H, Sullivan G, Quon MJ . Assessing the predictive accuracy of QUICKI as a surrogate index for insulin sensitivity using a calibration model. Diabetes 2005; 54: 1914–1925.

    Article  CAS  Google Scholar 

  11. Clasey JL, Bradley KD, Bradley JW, Long DE, Griffith JR . A new BIA equation estimating the body composition of young children. Obesity 2011; 19: 1813–1817.

    Article  Google Scholar 

  12. Garcia AL, Wagner K, Hothorn T, Koebnick C, Zunft HJ, Trippo U . Improved prediction of body fat by measuring skinfold thickness, circumferences, and bone breadths. Obes Res 2005; 13: 626–634.

    Article  Google Scholar 

  13. Huang TT, Watkins MP, Goran MI . Predicting total body fat from anthropometry in Latino children. Obes Res 2003; 11: 1192–1199.

    Article  Google Scholar 

  14. Rush EC, Chandu V, Plank LD . Prediction of fat-free mass by bioimpedance analysis in migrant Asian Indian men and women: a cross validation study. Int J Obes (Lond) 2006; 30: 1125–1131.

    Article  CAS  Google Scholar 

  15. Russell M, Mendes N, Miller KK, Rosen CJ, Lee H, Klibanski A et al. Visceral fat is a negative predictor of bone density measures in obese adolescent girls. J Clin Endorinol Metab 2010; 95: 1247–1255.

    Article  CAS  Google Scholar 

  16. Kuhn M, Johnson K . Applied Predictive Modeling. Springer, 2013.

    Book  Google Scholar 

  17. Copas JB . Regression, prediction and shrinkage. J R Stat Soc Ser B Methodol 1983; 45: 311–354.

    Google Scholar 

  18. Schmid M, Riganti-Fulginei F, Bernabucci I, Laudani A, Bibbo D, Muscillo R et al. SVM versus MAP on accelerometer data to distinguish among locomotor activities executed at different speeds. Comput Math Methods Med 2013; 343084: 1–7.

    Article  Google Scholar 

  19. Hitchcock C, Sober E . Prediction versus accommodation and the risk of overfitting. Brit J Philos Sci 2004; 55: 1–34.

    Article  Google Scholar 

  20. Harrell FE Jr, Lee KL, Mark DB . Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361–387.

    Article  Google Scholar 

  21. Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; 98: 691–698.

    Article  Google Scholar 

  22. Collins GS, Reitsma JB, Altman DG, Moons KG . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med 2015; 13: 1–10.

    Article  Google Scholar 

  23. Heymsfield SB, Thomas D, Bosy-Westphal A, Shen W, Peterson CM, Muller MJ . Evolving concepts on adjusting human resting energy expenditure measurements for body size. Obes Rev 2012; 13: 1001–1014.

    Article  CAS  Google Scholar 

  24. Ley SH, Hamdy O, Mohan V, Hu FB . Prevention and management of type 2 diabetes: dietary components and nutritional strategies. Lancet 2014; 383: 1999–2007.

    Article  CAS  Google Scholar 

  25. Mittlbock M, Schemper M . Explained variation for logistic regression. Stat Med 1996; 15: 1987–1997.

    Article  CAS  Google Scholar 

  26. Zou KH, O'Malley AJ, Mauri L . Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 2007; 115: 654–657.

    Article  Google Scholar 

  27. Ozenne B, Subtil F, Maucort-Boulch D . The precision-recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J Clin Epidemiol 2015; 68: 855–859.

    Article  Google Scholar 

  28. Harrell FFJ . Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer: New York, 2001.

    Book  Google Scholar 

  29. Hastie T, Tibshirani R, Friedman J . The Elements of Statistical Learning. Springer, 2009.

    Book  Google Scholar 

  30. James G, Witten D, Hastie T, Tibshirani R . An Introduction to Statistical Learning. Springer, 2013.

    Book  Google Scholar 

  31. Efron B, Tibshirani RJ . An Introduction to the Bootstrap vol. 57. Chapman & Hall/CRC Monographs on Statistics & Applied Probability: New York, 1993.

    Book  Google Scholar 

  32. Ambroise C, McLachlan GJ . Selection bias in gene extraction on the basis of microarray gene-expression data. Proc Natl Acad Sci USA 2002; 99: 6562–6566.

    Article  CAS  Google Scholar 

  33. Molinaro AM, Simon R, Pfeiffer RM . Prediction error estimation: a comparison of resampling methods. Bioinformatics 2005; 21: 3301–3307.

    Article  CAS  Google Scholar 

  34. Bobko P . Correlation and Regression: Applications for Industrial Organizational Psychology and Management. Sage Publications, 2001.

    Book  Google Scholar 

  35. Cattin P . Estimation of the predictive power of a regression model. J Appl Psychol 1980; 65: 407–414.

    Article  Google Scholar 

  36. Yamanaka N, Okamoto E, Kuwata K, Tanaka N . A multiple regression equation for prediction of posthepatectomy liver failure. Ann Surg 1984; 200: 658–663.

    Article  CAS  Google Scholar 

  37. Scalfi L, Marra M, De Filippo E, Caso G, Pasanisi F, Contaldo F . The prediction of basal metabolic rate in female patients with anorexia nervosa. Int J Obes Relat Metab Disord 2001; 25: 359–364.

    Article  CAS  Google Scholar 

  38. Mahon AD, Marjerrison AD, Lee JD, Woodruff ME, Hanna LE . Evaluating the prediction of maximal heart rate in children and adolescents. Res Q Exerc Sport 2010; 81: 466–471.

    Article  Google Scholar 

  39. Roediger HL 3rd, Watson JM, McDermott KB, Gallo DA . Factors that determine false recall: a multiple regression analysis. Psychon Bull Rev 2001; 8: 385–407.

    Article  Google Scholar 

  40. Nomani MZ, Khan AH, Shahda MM, Nomani AK, Sattar SA . Predicting serum gastrin levels among men during Ramadan fasting. East Mediterr Health J 2005; 11: 119–125.

    CAS  PubMed  Google Scholar 

  41. McKeon JL, Murree-Allen K, Saunders NA . Prediction of oxygenation during sleep in patients with chronic obstructive lung disease. Thorax 1988; 43: 312–317.

    Article  CAS  Google Scholar 

  42. Puyau MR, Adolph AL, Vohra FA, Zakeri I, Butte NF . Prediction of activity energy expenditure using accelerometers in children. Med Sci Sports Exerc 2004; 36: 1625–1631.

    PubMed  Google Scholar 

  43. Blalock SJ, Currey SS, DeVellis RF, Anderson JJ, Gold DT, Dooley MA . Using a short food frequency questionnaire to estimate dietary calcium consumption: a tool for patient education. Arthritis Care Res 1998; 11: 479–484.

    Article  CAS  Google Scholar 

  44. Siervo M, Prado C, Hooper L, Munro A, Collerton J, Davies K et al. Serum osmolarity and haematocrit do not modify the association between the impedance index (Ht(2)/Z) and total body water in the very old: the Newcastle 85+ study. Arch Gerontol Geriatr 2015; 60: 227–232.

    Article  Google Scholar 

  45. Hoffman DJ, Toro-Ramos T, Sawaya AL, Roberts SB, Rondo P . Estimating total body fat using a skinfold prediction equation in Brazilian children. Ann Hum Biol 2012; 39: 156–160.

    Article  Google Scholar 

  46. Lee JJ, Freeland-Graves JH, Pepper MR, Yao M, Xu B . Predictive equations for central obesity via anthropometrics, stereovision imaging and MRI in adults. Obesity 2014; 22: 852–862.

    Article  Google Scholar 

  47. Jackson AS, Stanforth PR, Gagnon J, Rankinen T, Leon AS, Rao DC et al. The effect of sex, age and race on estimating percentage body fat from body mass index: The Heritage Family Study. Int J Obes Relat Metab Disord 2002; 26: 789–796.

    Article  CAS  Google Scholar 

  48. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138.

    Article  Google Scholar 

  49. Apfelbacher CJ, Loerbroks A, Cairns J, Behrendt H, Ring J, Kramer U . Predictors of overweight and obesity in five to seven-year-old children in Germany: results from cross-sectional studies. BMC Public Health 2008; 8: 171.

    Article  Google Scholar 

  50. Gerhard GS, Benotti P, Wood GC, Chu X, Argyropoulos G, Petrick A et al. Identification of novel clinical factors associated with hepatic fat accumulation in extreme obesity. J Obes 2014; 368210: 1–8.

    Article  Google Scholar 

  51. Xi B, Zhao X, Shen Y, Wu L, Hou D, Cheng H et al. An obesity genetic risk score predicts risk of insulin resistance among Chinese children. Endocrine 2014; 47: 825–832.

    Article  CAS  Google Scholar 

  52. Little RJA, Rubin DB . Statistical Analysis with Missing Data. Wiley: New York, 1987.

    Google Scholar 

  53. Graham JW . Missing data analysis: making it work in the real world. Annu Rev Psychol 2009; 60: 549–576.

    Article  Google Scholar 

  54. Tibshirani R . Regression shrinkage and selection via the Lasso. J R Stat Soc Ser B Methodol 1996; 58: 267–288.

    Google Scholar 

  55. de los Campos G, Gianola D, Allison DB . Predicting genetic predisposition in humans: the promise of whole-genome markers. Nat Rev Genet 2010; 11: 880–886.

    Article  CAS  Google Scholar 

  56. Cotillard A, Poitou C, Duchateau-Nguyen G, Aron-Wisnewsky J, Bouillot J-L, Schindler T et al. Type 2 diabetes remission after gastric bypass: what is the best prediction tool for clinicians? Obes Surg 2015; 25: 1128–1132.

    Article  Google Scholar 

Download references

Acknowledgements

This study was supported in part by NIH grants R25DK099080, R25HL124208 and P30DK056336. We gratefully acknowledge the anonymous reviewers for their helpful suggestions, which substantially improved this article.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to D B Allison.

Ethics declarations

Competing interests

The authors declare no conflict of interest.

Additional information

Disclaimer

The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ivanescu, A., Li, P., George, B. et al. The importance of prediction model validation and assessment in obesity and nutrition research. Int J Obes 40, 887–894 (2016). https://doi.org/10.1038/ijo.2015.214

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/ijo.2015.214

This article is cited by

Search

Quick links