Abstract
Suicide risk is elevated among military service members who recently transitioned to civilian life. Identifying high-risk service members before this transition could facilitate provision of targeted preventive interventions. We investigated the feasibility of doing this by attempting to develop a prediction model for self-reported suicide attempts (SAs) after leaving or being released from active duty in the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS). This study included two self-report panel surveys (LS1: 2016–2018, LS2: 2018–2019) administered to respondents who previously participated while on active duty in one of three Army STARRS 2011–2014 baseline self-report surveys. We focus on respondents who left active duty >12 months before their LS survey (n = 8899). An ensemble machine learning model using predictors available prior to leaving active duty was developed in a 70% training sample and validated in a 30% test sample. The 12-month self-reported SA prevalence (SE) was 1.0% (0.1). Test sample AUC (SE) was 0.74 (0.06). The 15% of respondents with highest predicted risk included nearly two-thirds of 12-month SAs and over 80% of medically serious 12-month SAs. These results show that it is possible to identify soldiers at high post-transition self-report SA risk before the transition. Future model development is needed to examine prediction of SAs assessed by administrative data and using surveys administered closer to the time of leaving active duty.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Department of Defense. Department of Defense Strategy for Suicide Prevention. 2015. https://www.dspo.mil/Portals/113/Documents/TAB%20B%20-%20dssp_final%20USD%20PR%20signed.pdf. Accessed 14 Jul 2021.
U.S. Department of Veterans Affairs. National Strategy for Preventing Veteran Suicide: 2018-2028. 2018. https://www.mentalhealth.va.gov/suicide_prevention/docs/Office-of-Mental-Health-and-Suicide-Prevention-National-Strategy-for-Preventing-Veterans-Suicide.pdf. Accessed 16 Jul 2021.
U.S. Department of Veterans Affairs. The Military to Civilian Transition 2018: A Review of Historical, Current, and Future Trends. 2018. https://benefits.va.gov/TRANSITION/docs/mct-report-2018.pdf. Accessed 16 Jul 2021.
Pease JL, Billera M, Gerard G. Military culture and the transition to civilian life: suicide risk and other considerations. Soc Work. 2016;61:83–86.
Sokol Y, Gromatsky M, Edwards ER, Greene AL, Geraci JC, Harris RE, et al. The deadly gap: Understanding suicide among veterans transitioning out of the military. Psychiatry Res. 2021;300:113875.
Vogt D, King MW, Borowski S, Finley EP, Perkins DF, Copeland LA. Identifying factors that contribute to military veterans’ post-military well-being. Appl Psychol Health Well Being. 2021;13:341–56.
Geraci JC, Mobbs M, Edwards ER, Doerries B, Armstrong N, Porcarelli R, et al. Expanded roles and recommendations for stakeholders to successfully reintegrate modern warriors and mitigate suicide risk. Front Psychol. 2020;11:1907.
Chu C, Zuromski KL, Bernecker SL, Gutierrez PM, Joiner TE, Liu H, et al. A test of the interpersonal theory of suicide in a large, representative, retrospective and prospective study: Results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Behav Res Ther. 2020;132:103688.
Blosnich JR, Montgomery AE, Dichter ME, Gordon AJ, Kavalieratos D, Taylor L, et al. Social Determinants and Military Veterans’ Suicide Ideation and Attempt: a Cross-sectional Analysis of Electronic Health Record Data. J Gen Intern Med. 2020;35:1759–67.
Shen YC, Cunha JM, Williams TV. Time-varying associations of suicide with deployments, mental health conditions, and stressful life events among current and former US military personnel: a retrospective multivariate analysis. Lancet Psychiatry. 2016;3:1039–48.
Reger MA, Smolenski DJ, Skopp NA, Metzger-Abamukang MJ, Kang HK, Bullman TA, et al. Risk of Suicide Among US Military Service Members Following Operation Enduring Freedom or Operation Iraqi Freedom Deployment and Separation From the US Military. JAMA Psychiatry. 2015;72:561–9.
Ravindran C, Morley SW, Stephens BM, Stanley IH, Reger MA. Association of Suicide Risk With Transition to Civilian Life Among US Military Service Members. JAMA Netw Open. 2020;3:e2016261–e2016261.
Executive Office of the President. Supporting Our Veterans During Their Transition From Uniformed Service to Civilian Life. 2018. https://www.federalregister.gov/documents/2018/01/12/2018-00630/supporting-our-veterans-during-their-transition-from-uniformed-service-to-civilian-life. Accessed 13 Jul 2021.
U.S. Secretary of Defense, U.S. Secretary of Veterans Affairs, U.S. Secretary of Homeland Security. Joint Action Plan for Supporting Veterans During Their Transition from Uniformed Service to Civilian Life. 2018. https://www.va.gov/opa/docs/joint-action-plan-05-03-18.pdf. Accessed 14 Jul 2021.
116th Congress (2019-2020). Commander John Scott Hannon Veterans Mental Health Care Improvement Act of 2019. 2020. https://www.congress.gov/bill/116th-congress/senate-bill/785. Accessed 24 Sep 2021.
Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US. JAMA Psychiatry. 2021;78:642–50.
Zuromski KL, Bernecker SL, Gutierrez PM, Joiner TE, King AJ, Liu H, et al. Assessment of a Risk Index for Suicide Attempts Among US Army Soldiers With Suicide Ideation: Analysis of Data From the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Netw Open. 2019;2:e190766–e190766.
Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, Bromet EJ, et al. Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2015;72:49–57.
Kessler RC, Stein MB, Petukhova MV, Bliese P, Bossarte RM, Bromet EJ, et al. Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Mol Psychiatry. 2017;22:544–51.
Kessler RC, Bauer MS, Bishop TM, Demler OV, Dobscha SK, Gildea SM, et al. Using administrative data to predict suicide after psychiatric hospitalization in the Veterans health administration system. Front Psychiatry. 2020;11:390.
Gradus JL, King MW, Galatzer-Levy I, Street AE. Gender Differences in Machine Learning Models of Trauma and Suicidal Ideation in Veterans of the Iraq and Afghanistan Wars. J Trauma Stress. 2017;30:362–71.
Kessler RC, Hwang I, Hoffmire CA, McCarthy JF, Petukhova MV, Rosellini AJ, et al. Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration. Int J Methods Psychiatr Res 2017;26:e1575.
Nock MK, Borges G, Bromet EJ, Cha CB, Kessler RC, Lee S. Suicide and suicidal behavior. Epidemiol Rev. 2008;30:133–54.
Stanley IH, Hom MA, Boffa JW, Stage DL, Joiner TE. PTSD from a suicide attempt: an empirical investigation among suicide attempt survivors. J Clin Psychol. 2019;75:1879–95.
Ribeiro JD, Franklin JC, Fox KR, Bentley KH, Kleiman EM, Chang BP, et al. Self-injurious thoughts and behaviors as risk factors for future suicide ideation, attempts, and death: a meta-analysis of longitudinal studies. Psychol Med. 2016;46:225–36.
Bostwick JM, Pabbati C, Geske JR, McKean AJ. Suicide attempt as a risk factor for completed suicide: even more lethal than we knew. Am J Psychiatry. 2016;173:1094–1100.
Klonsky ED, May AM, Saffer BY. Suicide, suicide attempts, and suicidal ideation. Annu Rev Clin Psychol. 2016;12:307–30.
Beautrais AL. Suicides and serious suicide attempts: two populations or one? Psychol Med. 2001;31:837–45.
Suokas J, Lönnqvist J. Outcome of attempted suicide and psychiatric consultation: risk factors and suicide mortality during a five-year follow-up. Acta Psychiatr Scand. 1991;84:545–9.
Ursano RJ, Colpe LJ, Heeringa SG, Kessler RC, Schoenbaum M, Stein MB. The Army study to assess risk and resilience in servicemembers (Army STARRS). Psychiatry. 2014;77:107–19.
Heeringa SG, Gebler N, Colpe LJ, Fullerton CS, Hwang I, Kessler RC, et al. Field procedures in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22:276–87.
Kessler RC, Heeringa SG, Colpe LJ, Fullerton CS, Gebler N, Hwang I, et al. Response bias, weighting adjustments, and design effects in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22:288–302.
Kessler RC, Colpe LJ, Fullerton CS, Gebler N, Naifeh JA, Nock MK, et al. Design of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22:267–75.
Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168:1266–77.
Lee DJ, Kearns JC, Wisco BE, Green JD, Gradus JL, Sloan DM, et al. A longitudinal study of risk factors for suicide attempts among Operation Enduring Freedom and Operation Iraqi Freedom veterans. Depress Anxiety. 2018;35:609–18.
Nock MK, Deming CA, Fullerton CS, Gilman SE, Goldenberg M, Kessler RC, et al. Suicide among soldiers: a review of psychosocial risk and protective factors. Psychiatry. 2013;76:97–125.
Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. 2017;143:187–232.
Holliday R, Borges LM, Stearns-Yoder KA, Hoffberg AS, Brenner LA, Monteith LL. Posttraumatic Stress Disorder, Suicidal Ideation, and Suicidal Self-Directed Violence Among U.S. Military Personnel and Veterans: A Systematic Review of the Literature From 2010 to 2018. Front Psychol. 2020;11:1998.
Steelesmith DL, Fontanella CA, Campo JV, Bridge JA, Warren KL, Root ED. Contextual Factors Associated With County-Level Suicide Rates in the United States, 1999 to 2016. JAMA Netw Open. 2019;2:e1910936–e1910936.
Rehkopf DH, Buka SL. The association between suicide and the socio-economic characteristics of geographical areas: a systematic review. Psychol Med. 2006;36:145–57.
Fontanella CA, Saman DM, Campo JV, Hiance-Steelesmith DL, Bridge JA, Sweeney HA, et al. Mapping suicide mortality in Ohio: a spatial epidemiological analysis of suicide clusters and area level correlates. Prev Med. 2018;106:177–84.
Kessler RC, Bernecker SL, Bossarte RM, Luedtke AR, McCarthy JF, Nock MK, et al. The Role of Big Data Analytics in Predicting Suicide. In: Passos I, MB Kapczinski F. editos. Personalized Psychiatry. Cham: Springer International Publishing; 2019. pp 77–98.
Polley EC, Rose S, van der Laan MJ Super Learning. In: Rose S, van der Laan MJ, editors. Targeted Learning: Causal Inference for Observational and Experimental Data. New York, NY: Springer New York; 2011. pp 43–66.
Polley EC, LeDell E, Kennedy C, Lendle S, van der Laan MJ. SuperLearner: Super Learner Prediction [R package]. 2.0-24. 2018.
LeDell E, van der Laan MJ, Petersen M. AUC-Maximizing Ensembles through Metalearning. Int J Biostat. 2016;12:203–18.
Kabir MF, Ludwig SA. Enhancing the Performance of Classification Using Super Learning. Data-enabled Disco Appl. 2019;3:1–13.
Kennedy CJ Guide to SuperLearner. 2017. https://cran.r-project.org/web/packages/SuperLearner/vignettes/Guide-to-SuperLearner.html. Accessed 21 Jul 2021.
Lundberg S, Lee S-I A Unified Approach to Interpreting Model Predictions. 2017. https://arxiv.org/abs/1705.07874. Accessed 14 Jul 2021.
Greenland S. Model-based estimation of relative risks and other epidemiologic measures in studies of common outcomes and in case-control studies. Am J Epidemiol. 2004;160:301–5.
Böken B. On the appropriateness of Platt scaling in classifier calibration. Info Syst. 2021;95:101641.
Naeini MP, Cooper GF, Hauskrecht M. Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proc Conf AAAI Artif Intell. 2015;2015:2901–7.
Austin PC, Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med. 2019;38:4051–65.
Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med. 2014;33:517–35.
Yuan M, Kumar V, Ahmad MA, Teredesai A. Assessing fairness in classification parity of machine learning models in healthcare. 2021. http://arxiv.org/abs/2102.03717. Accessed 20 Jul 2021.
SAS Institute Inc. SAS ®Software 9.4 edn. Cary, NC: SAS Institute Inc; 2013.
Lundberg S. Welcome to the SHAP documentation. 2018. https://shap.readthedocs.io/en/latest/index.html. Accessed 21 Jul 2021.
R Core Team. R: A language and environment for statistical computing. 2021. https://www.r-project.org/. Accessed 20 Jul 2021.
Bullman T, Hoffmire C, Schneiderman A, Bossarte R. Time dependent gender differences in suicide risk among Operation Enduring Freedom and Operation Iraqi Freedom veterans. Ann Epidemiol. 2015;25:964–5.
Louzon SA, Bossarte R, McCarthy JF, Katz IR. Does Suicidal Ideation as Measured by the PHQ-9 Predict Suicide Among VA Patients? Psychiatr Serv. 2016;67:517–22.
Simon GE, Coleman KJ, Rossom RC, Beck A, Oliver M, Johnson E, et al. Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry. 2016;77:221–7.
Rudd MD Recognizing flawed assumptions in suicide risk assessment research and clinical practice. Psychol Med. 2021. https://doi.org/10.1017/S0033291721002750.
Richards JE, Whiteside U, Ludman EJ, Pabiniak C, Kirlin B, Hidalgo R, et al. Understanding Why Patients May Not Report Suicidal Ideation at a Health Care Visit Prior to a Suicide Attempt: A Qualitative Study. Psychiatr Serv. 2019;70:40–45.
Tourangeau R, Yan T. Sensitive Questions in Surveys. Psychol Bull. 2007;133:859–83.
Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Coming to terms with the terms of risk. Arch Gen Psychiatry. 1997;54:337–43.
Hubbard AE, Kennedy CJ, van der Laan MJ Data-Adaptive Target Parameters. Targeted Learning in Data Science: Causal Inference for Complex Longitudinal Studies. Cham: Springer International Publishing; 2018, pp 125–42.
U.S. Department of Veterans Affairs. VA Solid Start. 2020. https://www.benefits.va.gov/transition/solid-start.asp. Accessed 14 Jul 2021.
Geraci J, Goodman M, Seim R. ETS Sponsorship Program Helps Communities Fulfill Their Responsibility to Help Service Members Successfully Transition to Civilian Life. VAntage. Washington, DC: VA Office of Mental Health and Suicide Prevention; 2019.
Miller IW, Gaudiano BA, Weinstock LM. The Coping Long Term with Active Suicide Program: Description and Pilot Data. Suicide Life Threat Behav. 2016;46:752–61.
Rudd MD, Bryan CJ, Wertenberger EG, Peterson AL, Young-McCaughan S, Mintz J, et al. Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up. Am J Psychiatry. 2015;172:441–9.
Mann JJ, Michel CA, Auerbach RP. Improving suicide prevention through evidence-based strategies: a systematic review. Am J Psychiatry. 2021;178:611–24.
Bernecker SL, Zuromski KL, Curry JC, Kim JJ, Gutierrez PM, Joiner TE, et al. Economic Evaluation of Brief Cognitive Behavioral Therapy vs Treatment as Usual for Suicidal US Army Soldiers. JAMA Psychiatry. 2020;77:256–64.
Klonsky ED, May AM. The Three-Step Theory (3ST): A New Theory of Suicide Rooted in the “Ideation-to-Action” Framework. Int J Cogn Ther. 2015;8:114–29.
Carroll D, Kearney LK, Miller MA. Addressing Suicide in the Veteran Population: Engaging a Public Health Approach. Front Psychiatry. 2020;11:569069.
Millner AJ, Lee MD, Nock MK. Single-item measurement of suicidal behaviors: validity and consequences of misclassification. PLoS One. 2015;10:e0141606.
Department of Defense Office of Inspector General. Evaluation of the Department of Defense’s Implementation of Suicide Prevention Resources for Transitioning Uniformed Service Members (DODIG-2022-030). 2021. https://www.dodig.mil/reports.html/article/2841764/evaluation-of-the-department-of-defenses-implementation-of-suicide-prevention-r/. Accessed 15 Nov 2021.
Acknowledgements
IHS was supported in part by a grant from the National Institute of Mental Health (T32MH019836). Army STARRS was sponsored by the Department of the Army and funded under cooperative agreement number U01MH087981 with the U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (NIH/NIMH). Subsequently, STARRS-LS was sponsored and funded by the Department of Defense (USUHS grant numbers HU00011520004 and HU0001202003). The grants were administered by the Henry M. Jackson Foundation for the Advancement of Military Medicine Inc. (HJF). The contents are solely the responsibility of the authors and do not necessarily represent the views of the Department of Health and Human Services, NIMH, the Department of the Army, Department of Defense or HJF. The Army STARRS Team consists of Co-Principal Investigators: RJU, MD (Uniformed Services University) and MBS, MD, MPH (University of California San Diego and VA San Diego Healthcare System). ite Principal Investigators: James Wagner, PhD (University of Michigan) and RCK, PhD (Harvard Medical School). Army scientific consultant/liaison: Kenneth Cox, MD, MPH (Office of the Assistant Secretary of the Army (Manpower and Reserve Affairs)). Other team members: Pablo A. Aliaga, MA (Uniformed Services University); David M. Benedek, MD (Uniformed Services University); Laura Campbell-Sills, PhD (University of California San Diego); Carol S. Fullerton, PhD (Uniformed Services University); Nancy Gebler, MA (University of Michigan); Meredith House, BA (University of Michigan); Paul E. Hurwitz, MPH (Uniformed Services University); Sonia Jain, PhD (University of California San Diego); Tzu-Cheg Kao, PhD (Uniformed Services University); Lisa Lewandowski-Romps, PhD (University of Michigan); AL, PhD (University of Washington and Fred Hutchinson Cancer Research Center); Holly Herberman Mash, PhD (Uniformed Services University); James A. Naifeh, PhD (Uniformed Services University); Matthew K. Nock, PhD (Harvard University); Victor Puac-Polanco, MD, DrPH (Harvard Medical School); NAS, BA (Harvard Medical School); and Alan M. Zaslavsky, PhD (Harvard Medical School). As a cooperative agreement, scientists employed by the National Institute of Mental Health and U.S. Army liaisons and consultants collaborated to develop the study protocol and data collection instruments, supervise data collection, interpret results, and prepare reports. Although a draft of the manuscript was submitted to the U.S. Army and National Institute of Mental Health for review and comment before submission for publication, this was done with the understanding that comments would be no more than advisory.
Author information
Authors and Affiliations
Contributions
RCK, MBS, and RJU developed the concept and design of the study. CJK, AJK, AL, MVP, NAS, MBS, RJU, and RCK analyzed the data. IHH, AJK, and MVP conducted statistical analysis. IHS and RCK wrote the paper, and IHS, CC, SMG, IHH, AJK, CJK, AL, BPM, RO, MVP, NAS, DV, MBS, RJU, and RCK provided critical input towards the manuscript draft. MBS, RJU, and RCK acquired the funding.
Corresponding author
Ethics declarations
Competing interests
In the past 3 years, RCK was a consultant for Datastat, Inc., Holmusk, RallyPoint Networks, Inc., and Sage Therapeutics. He has stock options in Mirah, PYM, and Roga Sciences. In the past 3 years MBS received consulting income from Actelion, Acadia Pharmaceuticals, Aptinyx, atai Life Sciences, Boehringer Ingelheim, Bionomics, BioXcel Therapeutics, Clexio, EmpowerPharm, Engrail Therapeutics, GW Pharmaceuticals, Janssen, Jazz Pharmaceuticals, and Roche/Genentech. MBS has stock options in Oxeia Biopharmaceuticals and EpiVario. He is paid for his editorial work on Depression and Anxiety (Editor-in-Chief), Biological Psychiatry (Deputy Editor), and UpToDate (Co-Editor-in-Chief for Psychiatry). The other authors declare that they have no conflict of interest.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Stanley, I.H., Chu, C., Gildea, S.M. et al. Predicting suicide attempts among U.S. Army soldiers after leaving active duty using information available before leaving active duty: results from the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS). Mol Psychiatry 27, 1631–1639 (2022). https://doi.org/10.1038/s41380-021-01423-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41380-021-01423-4
This article is cited by
-
Genetic, environmental, and behavioral correlates of lifetime suicide attempt: Analysis of additive and interactive effects in two cohorts of US Army soldiers
Neuropsychopharmacology (2023)
-
Building an Integrated Data Infrastructure to Examine the Spectrum of Suicide Risk Factors in Philadelphia Medicaid
Administration and Policy in Mental Health and Mental Health Services Research (2023)