Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Original Article
  • Published:

Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)

Molecular Psychiatry volume 22, pages 544–551 (2017)Cite this article

Subjects

Abstract

The 2013 US Veterans Administration/Department of Defense Clinical Practice Guidelines (VA/DoD CPG) require comprehensive suicide risk assessments for VA/DoD patients with mental disorders but provide minimal guidance on how to carry out these assessments. Given that clinician-based assessments are not known to be strong predictors of suicide, we investigated whether a precision medicine model using administrative data after outpatient mental health specialty visits could be developed to predict suicides among outpatients. We focused on male nondeployed Regular US Army soldiers because they account for the vast majority of such suicides. Four machine learning classifiers (naive Bayes, random forests, support vector regression and elastic net penalized regression) were explored. Of the Army suicides in 2004–2009, 41.5% occurred among 12.0% of soldiers seen as outpatient by mental health specialists, with risk especially high within 26 weeks of visits. An elastic net classifier with 10–14 predictors optimized sensitivity (45.6% of suicide deaths occurring after the 15% of visits with highest predicted risk). Good model stability was found for a model using 2004–2007 data to predict 2008–2009 suicides, although stability decreased in a model using 2008–2009 data to predict 2010–2012 suicides. The 5% of visits with highest risk included only 0.1% of soldiers (1047.1 suicides/100 000 person-years in the 5 weeks after the visit). This is a high enough concentration of risk to have implications for targeting preventive interventions. An even better model might be developed in the future by including the enriched information on clinician-evaluated suicide risk mandated by the VA/DoD CPG to be recorded.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1

Similar content being viewed by others

References

  1. Armed Forces Health Surveillance Center. Deaths by suicide while on active duty, active and reserve components, U.S. Armed Forces, 1998-2011. Med Surveill Monthly Rep 2012; 19: 7–10.

    Google Scholar 

  2. Nock MK, Deming CA, Fullerton CS, Gilman SE, Goldenberg M, Kessler RC et al. Suicide among soldiers: a review of psychosocial risk and protective factors. Psychiatry 2013; 76: 97–125.

    Article  Google Scholar 

  3. Smolenski DJ, Reger MA, Bush NE, Skopp NA, Zhang Y, Campise RL . Department of Defense Suicide Event Report. National Center for Telehealth & Technology 2013.

  4. Zamorski MA . Suicide prevention in military organizations. Int Rev Psychiatry 2011; 23: 173–180.

    Article  Google Scholar 

  5. Veterans Affairs/Dept of Defense. Assessment and management of patients at risk for suicide (2013). 2013.

  6. Dawes RM, Faust D, Meehl PE . Clinical versus actuarial judgment. Science 1989; 243: 1668–1674.

    Article  CAS  Google Scholar 

  7. Grove WM, Zald DH, Lebow BS, Snitz BE, Nelson C . Clinical versus mechanical prediction: a meta-analysis. Psychol Assess 2000; 12: 19–30.

    Article  CAS  Google Scholar 

  8. McCarthy JF, Bossarte RM, Katz IR, Thompson C, Kemp J, Hannemann CM et al. Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs. Am J Public Health 2015; 105: 1935–1942.

    Article  Google Scholar 

  9. Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, Bromet EJ et al. Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 2015; 72: 49–57.

    Article  Google Scholar 

  10. Ursano RJ, Colpe LJ, Heeringa SG, Kessler RC, Schoenbaum M, Stein MB et al. The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Psychiatry 2014; 77: 107–119.

    Article  Google Scholar 

  11. Kessler RC, Colpe LJ, Fullerton CS, Gebler N, Naifeh JA, Nock MK et al. Design of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res 2013; 22: 267–275.

    Article  Google Scholar 

  12. Schlesselman JJ . Case-Control Studies: Design, Conduct, Analysis, 1st edn. Oxford University Press: New York, 1982.

    Google Scholar 

  13. Brown GK, Beck AT, Steer RA, Grisham JR . Risk factors for suicide in psychiatric outpatients: a 20-year prospective study. J Consult Clin Psychol 2000; 68: 371–377.

    Article  CAS  Google Scholar 

  14. Brugnoli R, Novick D, Haro JM, Rossi A, Bortolomasi M, Frediani S et al. Risk factors for suicide behaviors in the observational schizophrenia outpatient health outcomes (SOHO) study. BMC Psychiatry 2012; 12: 83.

    Article  Google Scholar 

  15. Leadholm AK, Rothschild AJ, Nielsen J, Bech P, Ostergaard SD . Risk factors for suicide among 34,671 patients with psychotic and non-psychotic severe depression. J Affect Disord 2014; 156: 119–125.

    Article  Google Scholar 

  16. Simon GE, Hunkeler E, Fireman B, Lee JY, Savarino J . Risk of suicide attempt and suicide death in patients treated for bipolar disorder. Int J Bipolar Disord 2007; 9: 526–530.

    Article  Google Scholar 

  17. Simon GE, Rutter CM, Peterson D, Oliver M, Whiteside U, Operskalski B et al. Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv 2013; 64: 1195–1202.

    Article  Google Scholar 

  18. Simon GE, Savarino J, Operskalski B, Wang PS . Suicide risk during antidepressant treatment. Am J Psychiatry 2006; 163: 41–47.

    Article  Google Scholar 

  19. Bachynski KE, Canham-Chervak M, Black SA, Dada EO, Millikan AM, Jones BH . Mental health risk factors for suicides in the US Army, 2007—8. Inj Prev 2012; 18: 405–412.

    Article  Google Scholar 

  20. Bell NS, Harford TC, Amoroso PJ, Hollander IE, Kay AB . Prior health care utilization patterns and suicide among U.S. Army soldiers. Suicide Life Threat Behav 2010; 40: 407–415.

    Article  Google Scholar 

  21. Black SA, Gallaway MS, Bell MR, Ritchie EC . Prevalence and risk factors associated with suicides of Army soldiers 2001–2009. Mil Psychol 2011; 23: 433–451.

    Google Scholar 

  22. Gilman SE, Bromet EJ, Cox KL, Colpe LJ, Fullerton CS, Gruber MJ et al. Sociodemographic and career history predictors of suicide mortality in the United States Army 2004-2009. Psychol Med 2014; 44: 2579–2592.

    Article  Google Scholar 

  23. Hyman J, Ireland R, Frost L, Cottrell L . Suicide incidence and risk factors in an active duty US military population. Am J Public Health 2012; 102 (Suppl 1): S138–S146.

    Article  Google Scholar 

  24. Ireland RR, Kress AM, Frost LZ . Association between mental health conditions diagnosed during initial eligibility for military health care benefits and subsequent deployment, attrition, and death by suicide among active duty service members. Mil Med 2012; 177: 1149–1156.

    Article  Google Scholar 

  25. Schoenbaum M, Kessler RC, Gilman SE, Colpe LJ, Heeringa SG, Stein MB et al. Predictors of suicide and accident death in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS): results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 2014; 71: 493–503.

    Article  Google Scholar 

  26. Street AE, Gilman SE, Rosellini AJ, Stein MB, Bromet EJ, Cox KL et al. Understanding the elevated suicide risk of female soldiers during deployments. Psychol Med 2015; 45: 717–726.

    Article  CAS  Google Scholar 

  27. FDB Health. FDB First Databank, 2015. Available at: http://www.fdbhealth.com (accessed on 1 October, 2015).

  28. SAS Institute Inc. SAS/STAT Software. 9.3 for Unix edn.

  29. Rish I . An Empirical Study of the Naive Bayes Classifier. IBM Research Division: Yorktown Heights, NY, 2001.

    Google Scholar 

  30. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F, Chang CC et al Package 'e1071': misc functions of the Department of Statistics, Probability Theory Group, TU Wien. 1.5-7 edn 2015.

  31. Breiman L . Random Forests. Mach Learn 2001; 45: 5–32.

    Article  Google Scholar 

  32. Liaw A, Wiener M . Classification and regression by randomForest. R News 2002; 2: 18–22.

    Google Scholar 

  33. Smola AJ, Scholkopf B . A tutorial on support vector regression. Stat Comput 2004; 14: 199–222.

    Article  Google Scholar 

  34. Zou H, Hastie T . Regularization and variable selection via the elastic net. J R Stat Soc B 2005; 67: 301–320.

    Article  Google Scholar 

  35. Friedman J, Hastie T, Tibshirani R . Regularization paths for generalized linear models via coordinate descent. J Stat Softw 2009; 33: 1–22.

    Google Scholar 

  36. Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R et al. Health care contacts in the year before suicide death. J Gen Intern Med 2014; 29: 870–877.

    Article  Google Scholar 

  37. Large MM, Ryan CJ . Suicide risk categorisation of psychiatric inpatients: what it might mean and why it is of no use. Australas Psychiatry 2014; 22: 390–392.

    Article  Google Scholar 

  38. O'Connor E, Gaynes BN, Burda BU, Soh C, Whitlock EP . Screening for and treatment of suicide risk relevant to primary care: a systematic review for the U.S. Preventive Services Task Force. Ann Intern Med 2013; 158: 741–754.

    Article  Google Scholar 

  39. While D, Bickley H, Roscoe A, Windfuhr K, Rahman S, Shaw J et al. Implementation of mental health service recommendations in England and Wales and suicide rates, 1997-2006: a cross-sectional and before-and-after observational study. Lancet 2012; 379: 1005–1012.

    Article  Google Scholar 

  40. Berrouiguet S, Gravey M, Le Galudec M, Alavi Z, Walter M . Post-acute crisis text messaging outreach for suicide prevention: a pilot study. Psychiatry Res 2014; 217: 154–157.

    Article  Google Scholar 

  41. Valenstein M, Kim HM, Ganoczy D, McCarthy JF, Zivin K, Austin KL et al. Higher-risk periods for suicide among VA patients receiving depression treatment: prioritizing suicide prevention efforts. J Affect Disord 2009; 112: 50–58.

    Article  Google Scholar 

  42. LeardMann CA, Powell TM, Smith TC, Bell MR, Smith B, Boyko EJ et al. Risk factors associated with suicide in current and former US military personnel. JAMA 2013; 310: 496–506.

    Article  CAS  Google Scholar 

  43. Brenner LA, Ignacio RV, Blow FC . Suicide and traumatic brain injury among individuals seeking Veterans Health Administration services. J Head Trauma Rehabil 2011; 26: 257–264.

    Article  Google Scholar 

  44. Nowrangi MA, Kortte KB, Rao VA . A perspectives approach to suicide after traumatic brain injury: case and review. Psychosomatics 2014; 55: 430–437.

    Article  Google Scholar 

  45. Reger MA, Smolenski DJ, Skopp NA, Metzger-Abamukang MJ, Kang HK, Bullman TA et al. Risk of suicide among US military service members following operation enduring freedom or operation Iraqi freedom deployment and separation from the US military. JAMA Psychiatry 2015; 72: 561–569.

    Article  Google Scholar 

  46. Erdman HP, Greist JH, Gustafson DH, Taves JE, Klein MH . Suicide risk prediction by computer interview: a prospective study. J Clin Psychiatry 1987; 48: 464–467.

    CAS  PubMed  Google Scholar 

  47. Gustafson DH, Greist JH, Stauss FF, Erdman H, Laughren T . A probabilistic system for identifying suicide attemptors. Comput Biomed Res 1977; 10: 83–89.

    Article  CAS  Google Scholar 

  48. Gustafson DH, Tianen B, Greist JH . A computer-based system for identifying suicide attemptors. Comput Biomed Res 1981; 14: 144–157.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

Army STARRS was sponsored by the Department of the Army and funded under cooperative agreement number U01MH087981 with the US Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (NIH/NIMH). Dr Gilman’s participation in this work was supported by the Intramural Research Program of the Eunice Kennedy Shriver National Institute of Child Health and Human Development. We thank Kenneth L Cox for helpful comments on an earlier version of this paper. Although a draft of this manuscript was submitted to the Army and NIMH for review and comment before submission, this was with the understanding that comments would be no more than advisory.

Author contributions

The Army STARRS Team Co-Principal Investigators: Robert J Ursano (Uniformed Services University of the Health Sciences) and Murray B Stein (University of California San Diego and VA San Diego Healthcare System); Site Principal Investigators: Steven Heeringa (University of Michigan) and Ronald C Kessler (Harvard Medical School); NIMH collaborating scientists: Lisa J Colpe and Michael Schoenbaum; Army liaisons/consultants: COL Steven Cersovsky (USAPHC) and Kenneth Cox (USAPHC). Other team members: Pablo A Aliaga (Uniformed Services University of the Health Sciences); David M. Benedek (Uniformed Services University of the Health Sciences); Susan Borja (National Institute of Mental Health); Gregory G Brown (University of California San Diego); Laura Campbell-Sills (University of California San Diego); Catherine L Dempsey (Uniformed Services University of the Health Sciences); Richard Frank (Harvard Medical School); Carol S Fullerton (Uniformed Services University of the Health Sciences); Nancy Gebler (University of Michigan); Robert K Gifford (Uniformed Services University of the Health Sciences); Stephen E Gilman (Eunice Kennedy Shriver National Institute of Child Health and Human Development, Harvard School of Public Health); Marjan G Holloway (Uniformed Services University of the Health Sciences); Paul E Hurwitz (Uniformed Services University of the Health Sciences); Sonia Jain (University of California San Diego); Tzu-Cheg Kao (Uniformed Services University of the Health Sciences); Karestan C Koenen (Columbia University); Lisa Lewandowski-Romps (University of Michigan); Holly Herberman Mash (Uniformed Services University of the Health Sciences); James E McCarroll (Uniformed Services University of the Health Sciences); Katie A McLaughlin (Harvard Medical School); James A Naifeh (Uniformed Services University of the Health Sciences); Matthew K Nock (Harvard University); Rema Raman (University of California San Diego); Sherri Rose (Harvard Medical School); Anthony Joseph Rosellini (Harvard Medical School); Nancy A Sampson (Harvard Medical School); LCDR Patcho Santiago (Uniformed Services University of the Health Sciences); Michaelle Scanlon (National Institute of Mental Health); Jordan Smoller (Harvard Medical School); Michael L Thomas (University of California San Diego); Patti L Vegella (Uniformed Services University of the Health Sciences); Christina Wassel (University of Pittsburgh); and Alan M Zaslavsky (Harvard Medical School). We also thank John Mann, Maria Oquendo, Barbara Stanley, Kelly Posner and John Keilp for their contributions to the early stages of Army STARRS development.

Author information

Author notes

  1. R J Ursano, M B Stein, S Heeringa, R C Kessler, L J Colpe, M Schoenbaum, S Cersovsky, K Cox, P A Aliaga, D M Benedek, S Borja, G G Brown, L C Sills, C L Dempsey, R Frank, C S Fullerton, N Gebler, R K Gifford, S E Gilman, M G Holloway, P E Hurwitz, S Jain, T-C Kao, K C Koenen, L L Romps, H H Mash, J E McCarroll, K A McLaughlin, J A Naifeh, M K Nock, R Raman, S Rose, A J Rosellini, N A Sampson, L P Santiago, M Scanlon, J Smoller, M L Thomas, P L Vegella, C Wassel, A M Zaslavsky, J Mann, M Oquendo, B Stanley, K Posner and J Keilp: The list of Army STARRS Collaborators are specified in author contributions, before references.

Authors and Affiliations

  1. Department of Health Care Policy, Harvard Medical School, Boston, MA, USA

    R C Kessler, M V Petukhova, A J Rosellini, N A Sampson & A M Zaslavsky

  2. Departments of Psychiatry and Family and Preventive Medicine, University of California San Diego, La Jolla, CA, USA

    M B Stein

  3. VA San Diego Healthcare System, San Diego, CA, USA

    M B Stein

  4. Darla Moore School of Business, University of South Carolina, Columbia, SC, USA

    P Bliese

  5. Department of Veterans Affairs, Post Deployment Health Service, Washington, DC, USA

    R M Bossarte

  6. Department of Psychiatry and Behavioral Science, Stony Brook School of Medicine, Stony Brook, NY, USA

    E J Bromet

  7. Department of Psychiatry, Center for the Study of Traumatic Stress, Uniformed Services University School of Medicine, Bethesda, MD, USA

    C S Fullerton, J A Naifeh & R J Ursano

  8. Division of Intramural Population Health Research, Health Behavior Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development, Bethesda, MD, USA

    S E Gilman

  9. Departments of Epidemiology and Social and Behavioral Sciences, Harvard TH Chan School of Public Health, Boston, MA, USA

    S E Gilman

  10. US Army Medical Command, Behavioral Health Service Line, Bethesda, MD, USA

    C Ivany

  11. Institute for Social Research, University of Michigan, Ann Arbor, MI, USA

    L Lewandowski-Romps

  12. US Army Public Health Command, Aberdeen Proving Ground, MD, USA

    A Millikan Bell

  13. Department of Psychology, Harvard University, Cambridge, MA, USA

    M K Nock

  14. Children’s Hospital Boston and Harvard Medical School, Boston, MA, USA

    B Y Reis

Authors
  1. R C Kessler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M B Stein
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M V Petukhova
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. P Bliese
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. R M Bossarte
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. E J Bromet
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. C S Fullerton
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. S E Gilman
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. C Ivany
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. L Lewandowski-Romps
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. A Millikan Bell
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. J A Naifeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. M K Nock
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. B Y Reis
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. A J Rosellini
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. N A Sampson
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. A M Zaslavsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. R J Ursano
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

Army STARRS Collaborators

  • R J Ursano
  • , M B Stein
  • , S Heeringa
  • , R C Kessler
  • , L J Colpe
  • , M Schoenbaum
  • , S Cersovsky
  • , K Cox
  • , P A Aliaga
  • , D M Benedek
  • , S Borja
  • , G G Brown
  • , L C Sills
  • , C L Dempsey
  • , R Frank
  • , C S Fullerton
  • , N Gebler
  • , R K Gifford
  • , S E Gilman
  • , M G Holloway
  • , P E Hurwitz
  • , S Jain
  • , T-C Kao
  • , K C Koenen
  • , L L Romps
  • , H H Mash
  • , J E McCarroll
  • , K A McLaughlin
  • , J A Naifeh
  • , M K Nock
  • , R Raman
  • , S Rose
  • , A J Rosellini
  • , N A Sampson
  • , L P Santiago
  • , M Scanlon
  • , J Smoller
  • , M L Thomas
  • , P L Vegella
  • , C Wassel
  • , A M Zaslavsky
  • , J Mann
  • , M Oquendo
  • , B Stanley
  • , K Posner
  •  & J Keilp

Corresponding author

Correspondence to R C Kessler.

Ethics declarations

Competing interests

In the past 3 years, Dr Kessler has been a consultant for Hoffman-La Roche and Johnson & Johnson Wellness and Prevention. Dr Kessler has served on advisory boards for Mensante Corporation, Johnson & Johnson Services, Lake Nona Life Project and US Preventive Medicine. Dr Kessler is a co-owner of DataStat. Dr Stein has in the last 3 years been a consultant for Healthcare Management Technologies and had research support for pharmacologic imaging studies from Janssen. The other authors declare no conflict of interest.

Additional information

Disclosure

The contents are solely the responsibility of the authors and do not necessarily represent the views of the Department of Health and Human Services, NIMH, the Department of the Army or the Department of Defense.

Supplementary Information accompanies the paper on the Molecular Psychiatry website

Supplementary information

PowerPoint slides

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kessler, R., Stein, M., Petukhova, M. et al. Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Mol Psychiatry 22, 544–551 (2017). https://doi.org/10.1038/mp.2016.110

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/mp.2016.110

Search

Advanced search

Quick links