Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Expert Review
  • Published:

Suicide prediction models: a critical review of recent research with recommendations for the way forward

Molecular Psychiatry volume 25, pages 168–179 (2020)Cite this article

Subjects

Abstract

Suicide is a leading cause of death. A substantial proportion of the people who die by suicide come into contact with the health care system in the year before their death. This observation has resulted in the development of numerous suicide prediction tools to help target patients for preventive interventions. However, low sensitivity and low positive predictive value have led critics to argue that these tools have no clinical value. We review these tools and critiques here. We conclude that existing tools are suboptimal and that improvements, if they can be made, will require developers to work with more comprehensive predictor sets, staged screening designs, and advanced statistical analysis methods. We also conclude that although existing suicide prediction tools currently have little clinical value, and in some cases might do more harm than good, an even-handed assessment of the potential value of refined tools of this sort cannot currently be made because such an assessment would depend on evidence that currently does not exist about the effectiveness of preventive interventions. We argue that the only way to resolve this uncertainty is to link future efforts to develop or evaluate suicide prediction tools with concrete questions about specific clinical decisions aimed at reducing suicides and to evaluate the clinical value of these tools in terms of net benefit rather than sensitivity or positive predictive value. We also argue for a focus on the development of individualized treatment rules to help select the right suicide-focused treatments for the right patients at the right times. Challenges will exist in doing this because of the rarity of suicide even among patients considered high-risk, but we offer practical suggestions for how these challenges can be addressed.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1

Similar content being viewed by others

References

  1. World Health Organization (WHO). Mental health: suicide data. 2018. http://www.who.int/mental_health/prevention/suicide/suicideprevent/en/. Accessed 30 April 2019.

  2. Ahmedani B, Simon G, Stewart C, Beck A, Waitzfelder B, Rossom R, et al. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29:870–7.

    PubMed  PubMed Central  Google Scholar 

  3. Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry. 2002;159:909–16.

    PubMed  PubMed Central  Google Scholar 

  4. Pearson A, Saini P, Da Cruz D, Miles C, While D, Swinson N, et al. Primary care contact prior to suicide in individuals with mental illness. Br J Gen Pr. 2009;59:825.

    Google Scholar 

  5. Schaffer A, Sinyor M, Kurdyak P, Vigod S, Sareen J, Reis C, et al. Population‐based analysis of health care contacts among suicide decedents: identifying opportunities for more targeted suicide prevention strategies. World Psychiatry. 2016;15:135–45.

    PubMed  PubMed Central  Google Scholar 

  6. Rosen A. Detection of suicidal patients: an example of some limitations in the prediction of infrequent events. J Consult Psychol. 1954;18:397–403.

    CAS  PubMed  Google Scholar 

  7. Murphy GE. Clinical identification of suicidal risk. Arch Gen Psychiatry. 1972;27:356–9.

    CAS  PubMed  Google Scholar 

  8. Belsher BE, Smolenski DJ, Pruitt LD, Bush NE, Beech EH, Workman DE, et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76:642–51.

    PubMed  Google Scholar 

  9. Kessler RC. Clinical epidemiological research on suicide-related behaviors: where we are and where we need to go. JAMA Psychiatry 2019;76:777–8.

    PubMed  Google Scholar 

  10. Garb HN, Wood JM. Methodological advances in statistical prediction. Psychol Assess. 2019. https://doi.org/10.1037/pas0000673.

    PubMed  Google Scholar 

  11. Naghavi M. Global, regional, and national burden of suicide mortality 1990–2016: systematic analysis for the Global Burden of Disease Study 2016. BMJ. 2019;364:l94.

  12. Katz C, Bolton J, Sareen J. The prevalence rates of suicide are likely underestimated worldwide: why it matters. Soc Psychiatry Psychiatr Epidemiol. 2016;51:125–7.

    PubMed  Google Scholar 

  13. Stone D, Simon T, Fowler K, Kegler S, Yuan K, Holland K et al. Vital signs: trends in state suicide rates - United States, 1999–2016 and circumstances contributing to suicide - 27 States, 2015. MMWR Morb Mortal Wkly Rep. 2018; 67: 617–24.

    PubMed  PubMed Central  Google Scholar 

  14. Heron M. Deaths: leading causes for 2016. National Vital Statistics Reports. 67. Hyattsville, MD: National Center for Health Statistics. 2018. https://www.cdc.gov/nchs/data/nvsr/nvsr67/nvsr67_06.pdf.

  15. Carroll R, Metcalfe C, Gunnell D. Hospital presenting self-harm and risk of fatal and non-fatal repetition: systematic review and meta-analysis. PLoS ONE. 2014;9:e89944.

    PubMed  PubMed Central  Google Scholar 

  16. Walsh G, Sara G, Ryan CJ, Large M. Meta‐analysis of suicide rates among psychiatric in‐patients. Acta Psychiatr Scand. 2015;131:174–84.

    CAS  PubMed  Google Scholar 

  17. Chung DT, Ryan CJ, Hadzi-Pavlovic D, Singh SP, Stanton C, Large MM. Suicide rates after discharge from psychiatric facilities: a systematic review and meta-analysis. JAMA Psychiatry. 2017;74:694–702.

    PubMed  PubMed Central  Google Scholar 

  18. Chung D, Hadzi-Pavlovic D, Wang M, Swaraj S, Olfson M, Large M. Meta-analysis of suicide rates in the first week and the first month after psychiatric hospitalisation. BMJ Open. 2019;9:e023883.

    PubMed  PubMed Central  Google Scholar 

  19. Carter G, Milner A, McGill K, Pirkis J, Kapur N, Spittal MJ. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry. 2017;210:387–95.

    PubMed  Google Scholar 

  20. Chan KCG, Yam SCP, Zhang Z. Globally efficient non‐parametric inference of average treatment effects by empirical balancing calibration weighting. J R Stat Soc Ser B Stat Methodol. 2016;78:673–700.

    Google Scholar 

  21. Katz C, Randall JR, Sareen J, Chateau D, Walld R, Leslie WD, et al. Predicting suicide with the SAD PERSONS scale. Depress Anxiety. 2017;34:809–16.

    PubMed  Google Scholar 

  22. Large M, Myles N, Myles H, Corderoy A, Weiser M, Davidson M, et al. Suicide risk assessment among psychiatric inpatients: a systematic review and meta-analysis of high-risk categories. Psychol Med. 2018;48:1119–27.

    CAS  PubMed  Google Scholar 

  23. Large M, Kaneson M, Myles N, Myles H, Gunaratne P, Ryan C. Meta-analysis of longitudinal cohort studies of suicide risk assessment among psychiatric patients: heterogeneity in results and lack of improvement over time. PLoS ONE. 2016;11:e0156322.

    PubMed  PubMed Central  Google Scholar 

  24. Larkin C, Di Blasi Z, Arensman E. Risk factors for repetition of self-harm: a systematic review of prospective hospital-based studies. PLoS ONE. 2014;9:e84282.

    PubMed  PubMed Central  Google Scholar 

  25. Quinlivan L, Cooper J, Davies L, Hawton K, Gunnell D, Kapur N. Which are the most useful scales for predicting repeat self-harm? A systematic review evaluating risk scales using measures of diagnostic accuracy. BMJ Open. 2016;6:e009297.

    CAS  PubMed  PubMed Central  Google Scholar 

  26. Runeson B, Odeberg J, Pettersson A, Edbom T, Jildevik Adamsson I, et al. Instruments for the assessment of suicide risk: a systematic review evaluating the certainty of the evidence. PLoS ONE. 2017;12:e0180292.

    PubMed  PubMed Central  Google Scholar 

  27. Bolton JM, Gunnell D, Turecki G. Suicide risk assessment and intervention in people with mental illness. Brit Med J. 2015;351:h4978.

    PubMed  Google Scholar 

  28. Woodford R, Spittal MJ, Milner A, McGill K, Kapur N, Pirkis J, et al. Accuracy of clinician predictions of future self-harm: a systematic review and meta-analysis of predictive studies. Suicide Life Threat Behav. 2019;49:23–40.

    PubMed  Google Scholar 

  29. Barak-Corren Y, Castro VM, Javitt S, Hoffnagle AG, Dai Y, Perlis RH, et al. Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry. 2017;174:154–62.

    PubMed  Google Scholar 

  30. Ben-Ari A, Hammond K. Text mining the EMR for modeling and predicting suicidal behavior among US veterans of the 1991 Persian gulf war. 2015 48th Hawaii International Conference on System Sciences. Kauai, HI; 2015;3168–75. https://doi.org/10.1109/HICSS.2015.

  31. Choi SB, Lee W, Yoon J-H, Won J-U, Kim DW. Ten-year prediction of suicide death using Cox regression and machine learning in a nationwide retrospective cohort study in South Korea. J Affect Disord. 2018;231:8–14.

    PubMed  Google Scholar 

  32. Kessler RC, Hwang I, Hoffmire CA, McCarthy JF, Petukhova MV, Rosellini AJ, et al. Developing a practical suicide risk prediction model for targeting high‐risk patients in the Veterans Health Administration. Int J Methods Psychiatr Res. 2017;26. https://doi.org/10.1002/mpr.1575

    PubMed Central  Google Scholar 

  33. Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clin Psychol Sci. 2017;5:457–69.

    Google Scholar 

  34. Simon R. Improving suicide risk assessment with evidence-based psychiatry. In: Pompili M, Taterelli R editors. Evidence-based practice in suicidology: a sourcebook. Cambridge MA: Hogrefe Publishing; 2011, p. 45–54.

  35. Beck A, Steer R. BHS, Beck Hopelessness Scale: manual. San Antonio TX: Psychological Corporation; 1988.

    Google Scholar 

  36. Beck A, Steer R, Brown G. Manual for the Beck Depression Inventory-II. San Antonio TX: Psychological corporation; 1996.

  37. Beck A, Schuyler D, Herman I. Development of suicidal intent scales. In: Beck A, Lettieri D, Resnik H editors. The prediction of suicide. Bowie, MD: Charles Press; 1974, p. 45–56.

  38. Koldsland BO, Mehlum L, Mellesdal LS, Walby FA, Diep LM. The suicide assessment scale: psychometric properties of a Norwegian language version. BMC Res Notes. 2012;5:417 https://doi.org/10.1186/1756-0500-5-417

    Article  PubMed  PubMed Central  Google Scholar 

  39. Kreitman N, Foster J. The construction and selection of predictive scales, with special reference to parasuicide. Br J Psychiatry. 1991;159:185–92.

    CAS  PubMed  Google Scholar 

  40. Randall JR, Rowe BH, Dong KA, Nock MK, Colman I. Assessment of self-harm risk using implicit thoughts. Psychol Assess. 2013;25:714–21.

    PubMed  Google Scholar 

  41. Robins JM, Rotnitzky A. Semiparametric efficiency in multivariate regression models with missing data. J Am Stat Assoc. 2015;90:122–9.

    Google Scholar 

  42. Bolton JM. Suicide risk assessment in the emergency department: out of the darkness. Depress Anxiety. 2015;32:73–75.

    PubMed  Google Scholar 

  43. Hoge CW. Suicide reduction and research efforts in service members and veterans-sobering realities. JAMA Psychiatry. 2019. https://doi.org/10.1001/jamapsychiatry.2018.4564.

    PubMed  Google Scholar 

  44. Mulder R, Newton-Howes G, Coid JW. The futility of risk prediction in psychiatry. Br J Psychiatry. 2016;209:271–2.

    PubMed  Google Scholar 

  45. Owens D, Kelley R. Predictive properties of risk assessment instruments following self-harm. Br J Psychiatry. 2017;210:384–6.

    PubMed  Google Scholar 

  46. Wortzel HS, Nazem S, Bahraini NH, Matarazzo BB. Why suicide risk assessment still matters. J Psychiatr Pr. 2017;23:436–40.

    Google Scholar 

  47. Hunter C, Chantler K, Kapur N, Cooper J. Service user perspectives on psychosocial assessment following self-harm and its impact on further help-seeking: a qualitative study. J Affect Disord. 2013;145:315–23.

    PubMed  Google Scholar 

  48. Owens C, Hansford L, Sharkey S, Ford T. Needs and fears of young people presenting at accident and emergency department following an act of self-harm: secondary analysis of qualitative data. Br J Psychiatry. 2016;208:286–91.

    PubMed  PubMed Central  Google Scholar 

  49. Taylor TL, Hawton K, Fortune S, Kapur N. Attitudes towards clinical services among people who self-harm: systematic review. Br J Psychiatry. 2009;194:104–10.

    PubMed  Google Scholar 

  50. Palmer L, Blackwell H, Strevens P. Service users’ experience of emergency services following self harm: a national survey of 509 patients. College Centre for Quality Improvement, Royal College of Psychiatrists. 2007. https://www.rcpsych.ac.uk/. Accessed 20 Feb 2018

  51. Rosen DC, Nakash O, Alegria M. The impact of computer use on therapeutic alliance and continuance in care during the mental health intake. Psychother (Chic). 2016;53:117–23.

    Google Scholar 

  52. Self-harm in over 8s: long-term management. National Institute for Health and Care Excellence (NICE). 2011. Accessed 30 April 2019

  53. O’Connor E, Gaynes BN, Burda BU, Soh C, Whitlock EP. Screening for and treatment of suicide risk relevant to primary care: a systematic review for the US Preventive Services Task Force. Ann Intern Med. 2013;158:741–54.

    PubMed  Google Scholar 

  54. National Institute for Health and Care Excellence. Preventing suicide in community and custodial settings. 2018. https://www.nspa.org.uk/wp-content/uploads/2018/09/preventing-suicide-in-community-and-custodial-settings-pdf-66141539632069.pdf.

  55. Bernert R, Hom M, Roberts L. A review of multidisciplinary clinical practice guidelines in suicide prevention: toward an emerging standard in suicide risk assessment and management, training and practice. Acad Psychiatry. 2014;38:585–92.

    PubMed  PubMed Central  Google Scholar 

  56. Silverman JJ, Galanter M, Jackson-Triche M, Jacobs DG, Lomax JW, Riba MB, et al. The American Psychiatric Association practice guidelines for the psychiatric evaluation of adults. Am J Psychiatry. 2015;172:798–802.

    PubMed  Google Scholar 

  57. Quinlivan L, Cooper J, Steeg S, Davies L, Hawton K, Gunnell D, et al. Scales for predicting risk following self-harm: an observational study in 32 hospitals in England. BMJ Open. 2014;4:e004732.

    CAS  PubMed  PubMed Central  Google Scholar 

  58. Rudd M. Core competencies, warning signs, and a framework for suicide risk assessment in clinical practice. In: Nock M editor. The Oxford handbook of suicide and self-injury. 1st ed. New York: Oxford University Press; 2014. p. 323–36.

  59. Cooper J, Steeg S, Bennewith O, Lowe M, Gunnell D, House A, et al. Are hospital services for self-harm getting better? An observational study examining management, service provision and temporal trends in England. BMJ Open. 2013;3:e003444.

    CAS  PubMed  PubMed Central  Google Scholar 

  60. Dawes RM, Faust D, Meehl PE. Clinical versus actuarial judgment. Science. 1989;243:1668–74.

    CAS  PubMed  Google Scholar 

  61. Ægisdóttir S, White MJ, Spengler PM, Maugherman AS, Anderson LA, Cook RS, et al. The meta-analysis of clinical judgment project: fifty-six years of accumulated research on clinical versus statistical prediction. Couns Psychol. 2006;34:341–82.

    Google Scholar 

  62. Large M, Sharma S, Cannon E, Ryan C, Nielssen O. Risk factors for suicide within a year of discharge from psychiatric hospital: a systematic meta-analysis. Aust N Z J Psychiatry. 2011;45:619–28.

    PubMed  Google Scholar 

  63. Jobes D, Au J, Siegelman A. Psychological approaches to suicide treatment and prevention. Curr Treat Options Psychiatry. 2015;2:363–70.

    Google Scholar 

  64. Smith KA, Cipriani A. Lithium and suicide in mood disorders: updated meta-review of the scientific literature. Bipolar Disord. 2017;19:575–86.

    CAS  PubMed  Google Scholar 

  65. Vermeulen JM, van Rooijen G, van de Kerkhof MPJ, Sutterland AL, Correll CU, de Haan L. Clozapine and long-term mortality risk in patients with schizophrenia: a systematic review and meta-analysis of Studies Lasting 1.1-12.5 Years. Schizophr Bull. 2019;45:315–29.

    PubMed  Google Scholar 

  66. United States. Public Health Service. Office of the Surgeon General. 2012 National strategy for suicide prevention: goals and objectives for action. National Action Alliance for Suicide Prevention. Washington, DC: National Action Alliance for Suicide Prevention; 2012. https://www.surgeongeneral.gov/library/reports/national-strategy-suicide-prevention/full-report.pdf.

  67. Brodsky BS, Spruch-Feiner A, Stanley B. The zero suicide model: applying evidence-based suicide prevention practices to clinical care. Front Psychiatry. 2018;9:33.

    PubMed  PubMed Central  Google Scholar 

  68. Jacobs DG. Suicide Assessment Five-step Evaluation and Triage for mental health professionals (SAFE-T). 2009. https://www.integration.samhsa.gov/images/res/SAFE_T.pdf.

  69. Fernhoff PM. Newborn screening for genetic disorders. Pediatr Clin North Am. 2009;56:505–13.

    PubMed  Google Scholar 

  70. Kessler R, Bernecker S, Bossarte R, Luedtke A, McCarthy JF, Nock MK, et al. The role of big data analytics in predicting suicide In: Passos I, Mwangi B, Kapczinski F, editors. Personalized Psychiatry—big data analytics in mental health. Springer Nature, 2019. p. 77–98.

  71. Hammond KW, Laundry RJ, O’Leary TM, Jones WP. Use of text search to effectively identify lifetime prevalence of suicide attempts among Veterans. 2013 46th Hawaii International Conference on System Sciences; Wailea, Maui, HI; 2013. p. 2676–83.

  72. Hammond KW, Laundry RJ. Application of a hybrid text mining approach to the study of suicidal behavior in a large population. 2014 47th Hawaii International Conference on System Science; Waikoloa, HI; 2014. p. 2555–61.

  73. Fernandes AC, Dutta R, Velupillai S, Sanyal J, Stewart R, Chandran D. Identifying suicide ideation and suicidal attempts in a psychiatric clinical research database using natural language processing. Sci Rep. 2018;8:7426.

    PubMed  PubMed Central  Google Scholar 

  74. Carson NJ, Mullin B, Sanchez MJ, Lu F, Yang K, Menezes M, et al. Identification of suicidal behavior among psychiatrically hospitalized adolescents using natural language processing and machine learning of electronic health records. PLoS ONE. 2019;14:e0211116.

    CAS  PubMed  PubMed Central  Google Scholar 

  75. Hammond KW, Ben‐Ari AY, Laundry RJ, Boyko EJ, Samore MH. The feasibility of using large‐scale text mining to detect adverse childhood experiences in a VA‐treated population. J Trauma Stress. 2015;28:505–14.

    PubMed  Google Scholar 

  76. McCoy TH, Pellegrini AM, Perlis RH. Research domain criteria scores estimated through natural language processing are associated with risk for suicide and accidental death. Depress Anxiety. 2019;36:392–9.

    PubMed  PubMed Central  Google Scholar 

  77. Simon GE, Johnson E, Lawrence JM, Rossom RC, Ahmedani B, Lynch FL, et al. Predicting Suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175:951–60.

    PubMed  PubMed Central  Google Scholar 

  78. American Community Survery (ACS). United States Census Bureau. 2018. https://www.census.gov/programs-surveys/acs/about.html. Accessed 8 Aug 2019.

  79. Lopez-Castroman J, Moulahi B, Aze J, Bringay S, Deninotti J, Guillaume S, et al. Mining social networks to improve suicide prevention: a scoping review. J Neurosci Res. 2019. https://doi.org/10.1002/jnr24404.

  80. Pestian JP, Sorter M, Connolly B, Bretonnel Cohen K, McCullumsmith C, Gee JT, et al. A machine learning approach to identifying the thought markers of suicidal subjects: a prospective multicenter trial. Suicide Life Threat Behav. 2017;47:112–21.

    PubMed  Google Scholar 

  81. Brown JL, Swartzendruber A, Diclemente RJ. Application of audio computer-assisted self-interviews to collect self-reported health data: an overview. Caries Res. 2013;47:40–5.

    PubMed  Google Scholar 

  82. Gnambs T, Kaspar K. Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behav Res Methods. 2015;47:1237–59.

    PubMed  Google Scholar 

  83. Greist JH, Laughren TP, Gustafson DH, Stauss FF, Rowse GL, Chiles JA. A computer interview for suicide-risk prediction. Am J Psychiatry. 1973;130:1327–32.

    CAS  PubMed  Google Scholar 

  84. Levine S, Ancill RJ, Roberts AP. Assessment of suicide risk by computer‐delivered self‐rating questionnaire: preliminary findings. Acta Psychiatr Scand. 1989;80:216–20.

    CAS  PubMed  Google Scholar 

  85. Nock MK, Park JM, Finn CT, Deliberto TL, Dour HJ, Banaji MR. Measuring the suicidal mind: implicit cognition predicts suicidal behavior. Psychol Sci. 2010;21:511–7.

    PubMed  Google Scholar 

  86. Bryan CJ, Rudd MD, Wertenberger E, Etienne N, Ray-Sannerud BN, Morrow CE, et al. Improving the detection and prediction of suicidal behavior among military personnel by measuring suicidal beliefs: an evaluation of the Suicide Cognitions Scale. J Affect Disord. 2014;159:15–22.

    PubMed  Google Scholar 

  87. Dhingra K, Boduszek D, O’Connor RC. Differentiating suicide attempters from suicide ideators using the Integrated Motivational-Volitional model of suicidal behaviour. J Affect Disord. 2015;186:211–8.

    PubMed  Google Scholar 

  88. Stefansson J, Nordstrom P, Runeson B, Asberg M, Jokinen J. Combining the Suicide Intent Scale and the Karolinska Interpersonal Violence Scale in suicide risk assessments. BMC Psychiatry. 2015;15:226.

    CAS  PubMed  PubMed Central  Google Scholar 

  89. Ursano RJ, Colpe LJ, Heeringa SG, Kessler RC, Schoenbaum M, Stein MB. The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Psychiatry. 2014;77:107–19.

    PubMed  Google Scholar 

  90. Bernecker SL, Zuromski KL, Gutierrez PM, Joiner TE, King AJ, Liu H, et al. Predicting suicide attempts among soldiers who deny suicidal ideation in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Behav Res Ther. 2018. https://doi.org/10.1016/j.brat.2018.11.018

    PubMed  Google Scholar 

  91. Boulesteix AL, Schmid M. Machine learning versus statistical modeling. Biom J. 2014;56:588–93.

    PubMed  Google Scholar 

  92. Harrell JFE. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. 2015. Cham, Switzerland: Springer International Publishing; 2015.

  93. Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019. https://doi.org/10.1016/j.clinepi.2019.02.004.

  94. Archer K, Kimes R. Empirical characterization of random forest variable importance measures. Comput Stat Data Anal. 2008;52:2249–60.

    Google Scholar 

  95. Polley E, LeDell E, van der Laan MJ. Super learner: super learner prediction. R package version 2.0-21: The Comprehensive R Archive Network; 2016. [Computer software]. Available at: https://cran.rstudio.org/. Accessed 30 April 2019.

  96. Feurer M, Klein A, Eggensperger K, Springenberg JT, Blum M, Hutter F. Efficient and robust automated machine learning. Proceedings of the 28th International Conference on Neural Information Processing Systems. Vol 2; Montreal, Canada; 2015.

  97. Olson RS, Sipper M, La Cava W, Tartarone S, Vitale S, Fu W et al. A system for accessible artificial intelligence. arXiv:1705.00594v2. 2017. Available from: https://arxiv.org/abs/1705.00594.

  98. Urbanowicz RJ, Meeker M, La Cava W, Olson RS, Moore JH. Relief based feature selection: introduction and review. arXiv:1711.08421. 2018. Available from: https://arxiv.org/abs/1711.08421.

  99. Chawla N. Data mining for imbalanced datasets: an overview. In: Maimon O, Rokach L editors. Data mining and knowledge discovery handbook. 2nd ed. Berlin/Heidelberg, Germany: Springer; 2010. p. 875–86.

    Google Scholar 

  100. Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, Bromet EJ, et al. Predicting suicides after psychiatric hospitalization in US Army Soldiers: the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2015;72:49–57.

    PubMed  PubMed Central  Google Scholar 

  101. McCarthy JF, Bossarte RM, Katz IR, Thompson C, Kemp J, Hannemann CM, et al. Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs. Am J Public Health. 2015;105:1935–42.

    PubMed  PubMed Central  Google Scholar 

  102. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. Brit Med J. 2016;352:i6.

    PubMed  PubMed Central  Google Scholar 

  103. Van Calster B, Wynants L, Verbeek JFM, Verbakel JY, Christodoulou E, Vickers AJ, et al. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol. 2018;74:796–804.

    PubMed  PubMed Central  Google Scholar 

  104. McKernan LC, Clayton EW, Walsh CG. Protecting life while preserving liberty: ethical recommendations for suicide prevention with artificial intelligence. Front Psychiatry. 2018;9:650.

    PubMed  PubMed Central  Google Scholar 

  105. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Mak. 2006;26:565–74.

    Google Scholar 

  106. Hoffmire C, Stephens B, Morley S, Thompson C, Kemp J, Bossarte RM. VA Suicide Prevention Applications Network: a national health care system–based suicide event tracking system. Public Health Rep. 2016;131:816–21.

    PubMed  PubMed Central  Google Scholar 

  107. Miller IW, Gaudiano BA, Weinstock LM. The coping long term with active suicide program: description and pilot. Suicide Life Threat Behav. 2018;46:752–61.

    Google Scholar 

  108. Stone JN, Robinson GJ, Lichtenstein HA, Bairey Merz NC, Blum BC, Eckel HR, et al. 2013 ACC/AHA Guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on practice guidelines. Circulation. 2014;129:S1–45.

    PubMed  Google Scholar 

  109. Comtois KA, Kerbrat AH, DeCou CR, Atkins DC, Majeres JJ, Baker JC, et al. Effect of augmenting standard care for military personnel with brief caring text messages for suicide prevention: a randomized clinical trial. JAMA Psychiatry. 2019;76:474–83.

    PubMed  PubMed Central  Google Scholar 

  110. Mortality risk valuation. United States Environmental Protection Agency; [updated February 8, 2018]; Available from: https://www.epa.gov/environmental-economics/mortality-risk-valuation. Accessed Mar 2019.

  111. Zalsman G, Hawton K, Wasserman D, van Heeringen K, Arensman E, Sarchiapone M, et al. Suicide prevention strategies revisited: 10-year systematic review. Lancet Psychiatry. 2016;3:646–59.

    PubMed  Google Scholar 

  112. Jobes DA. The Collaborative Assessment and Management of Suicidality (CAMS): an evolving evidence-based clinical approach to suicidal risk. Suicide Life Threat Behav. 2012;42:640–53.

    PubMed  Google Scholar 

  113. Frakt AB, Prentice JC, Pizer SD, Elwy AR, Garrido MM, Kilbourne AM, et al. Overcoming challenges to evidence-based policy development in a large, integrated delivery system. Health Serv Res. 2018;53:4789–807.

    PubMed  PubMed Central  Google Scholar 

  114. Cohen ZD, DeRubeis RJ. Treatment selection in depression. Annu Rev Clin Psychol. 2018;14:209–36.

    PubMed  Google Scholar 

  115. VanderWeele T, Luedtke A, van der Laan MJ, Kessler RC. Selecting optimal subgroups for treatment using many covariates. arXiv:1802.09642. 2018. Available from: https://arxiv.org/abs/1802.09642.

  116. Zubizarreta JR. Stable weights that balance covariates for estimation with incomplete outcome data. J Am Stat Assoc. 2015;110:910–22.

    CAS  Google Scholar 

  117. Luedtke AR, van der Laan MJ. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Ann Stat. 2016;44:713–42.

    PubMed  PubMed Central  Google Scholar 

  118. Steeg S, Carr M, Emsley R, Hawton K, Waters K, Bickley H, et al. Suicide and all-cause mortality following routine hospital management of self-harm: propensity score analysis using multicentre cohort data. PLoS ONE. 2018;13:e0204670.

    PubMed  PubMed Central  Google Scholar 

  119. Ichimura A, Kato K, Taira T, Otsuka H, Seki T, Nakagawa Y et al. Psychiatric hospitalization after emergency treatment for deliberate self-harm is associated with repeated deliberate self-harm. Arch Suicide Res. 2018. https://doi.org/10.1080/13811118.2018.1438323.

    PubMed  Google Scholar 

  120. Large MM, Kapur N. Psychiatric hospitalisation and the risk of suicide. Br J Psychiatry. 2018;212:269–73.

    PubMed  Google Scholar 

  121. Luedtke AR, van Der, Laan MJ. Optimal individualized treatments in resource-limited settings. Int J Biostat. 2016;12:283–303.

    PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported, in part, by the Department of Veterans Affairs Center of Excellence for Suicide Prevention and the Precision Treatment of Mental Disorders Initiative. The contents are solely the responsibility of the authors and do not necessarily represent the views of the Veteran’s Health administration. The authors appreciate the helpful comments of Matthew K. Nock, Vicki Shahly, Murray B. Stein, and Robert J. Ursano on an earlier version of the paper.

Author information

Authors and Affiliations

  1. Department of Health Care Policy, Harvard Medical School, Boston, MA, USA

    Ronald C. Kessler, Alan M. Zaslavsky & Jose R. Zubizarreta

  2. West Virginia University Injury Control Research Center and Department of Behavioral Medicine and Psychiatry, West Virginia University School of Medicine, Morgantown, WV, USA

    Robert M. Bossarte

  3. West Virginia and VISN 2 Center of Excellence for Suicide Prevention, Canandaigua, NY, USA

    Robert M. Bossarte

  4. Department of Statistics, University of Washington, Seattle, WA, USA

    Alex Luedtke

  5. Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, WA, USA

    Alex Luedtke

  6. Department of Statistics, Harvard University, Cambridge, MA, USA

    Jose R. Zubizarreta

Authors
  1. Ronald C. Kessler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert M. Bossarte
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alex Luedtke
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alan M. Zaslavsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jose R. Zubizarreta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ronald C. Kessler.

Ethics declarations

Conflict of interest

In the past 3 years, RCK was a consultant for Johnson & Johnson Wellness and Prevention, Sage, Shire, and Takeda and served on an advisory board for the Johnson & Johnson Services Inc. Lake Nona Life Project. Kessler is a co-owner of DataStat, Inc., a market research firm that carries out health care research. In the past 3 years, JRZ was a consultant for Johnson & Johnson. The remaining authors declare that they have no conflict of interest.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kessler, R.C., Bossarte, R.M., Luedtke, A. et al. Suicide prediction models: a critical review of recent research with recommendations for the way forward. Mol Psychiatry 25, 168–179 (2020). https://doi.org/10.1038/s41380-019-0531-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41380-019-0531-0

This article is cited by

Search

Advanced search

Quick links