A validated predictive algorithm of post-traumatic stress course following emergency department admission after a traumatic stressor

Abstract

Annually, approximately 30 million patients are discharged from the emergency department (ED) after a traumatic event1. These patients are at substantial psychiatric risk, with approximately 10–20% developing one or more disorders, including anxiety, depression or post-traumatic stress disorder (PTSD)2,3,4. At present, no accurate method exists to predict the development of PTSD symptoms upon ED admission after trauma5. Accurate risk identification at the point of treatment by ED services is necessary to inform the targeted deployment of existing treatment6,7,8,9 to mitigate subsequent psychopathology in high-risk populations10,11. This work reports the development and validation of an algorithm for prediction of post-traumatic stress course over 12 months using two independently collected prospective cohorts of trauma survivors from two level 1 emergency trauma centers, which uses routinely collectible data from electronic medical records, along with brief clinical assessments of the patient’s immediate stress reaction. Results demonstrate externally validated accuracy to discriminate PTSD risk with high precision. While the predictive algorithm yields useful reproducible results on two independent prospective cohorts of ED patients, future research should extend the generalizability to the broad, clinically heterogeneous ED population under conditions of routine medical care.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Variable importance for the training set.

Data availability

All requests for raw and analyzed data and related materials, including programming code, will be reviewed by our legal departments (New York University Grossman School of Medicine and Emory University School of Medicine) to verify whether the request is subject to any intellectual property or confidentiality constraints. Any data and materials that can be shared will be released via a material transfer agreement for noncommercial research purposes. Request should be addressed to the corresponding author (K.S.) or the Principal Investigators of the two study sites (K.J.R. and I.R.G.-L.).

Code availability

The programming code is based on Scikit-learn (https://scikit-learn.org/stable/) and SHAP (https://github.com/slundberg/shap) and the core algorithm can be obtained from https://github.com/KSchultebraucks/DeepSuperLearner. Requests should be addressed to the corresponding author (K.S.).

References

  1. 1.

    DiMaggio, C. J., Avraham, J. B., Lee, D. C., Frangos, S. G. & Wall, S. P. The epidemiology of emergency department trauma discharges in the United States. Acad. Emerg. Med. 24, 1244–1256 (2017).

    PubMed  PubMed Central  Google Scholar 

  2. 2.

    Wiseman, T. A., Curtis, K., Lam, M. & Foster, K. Incidence of depression, anxiety and stress following traumatic injury: a longitudinal study. Scand. J. Trauma Resusc. Emerg. Med. 23, 29 (2015).

    PubMed  PubMed Central  Google Scholar 

  3. 3.

    Sullivan, E. et al. The association between posttraumatic stress symptoms, depression, and length of hospital stay following traumatic injury. Gen. Hosp. Psychiatry 46, 49–54 (2017).

    PubMed  Google Scholar 

  4. 4.

    Fakhry, S. M. et al. Continuing trauma: the unmet needs of trauma patients in the postacute care setting. Am. Surgeon 83, 1308–1314 (2017).

    PubMed  Google Scholar 

  5. 5.

    Shalev, A. Y. et al. Estimating the risk of PTSD in recent trauma survivors: results of the International Consortium to Predict PTSD (ICPP). World Psychiatry 18, 77–87 (2019).

    PubMed  PubMed Central  Google Scholar 

  6. 6.

    Rothbaum, B. O. et al. Early intervention following trauma may mitigate genetic risk for PTSD in civilians: a pilot prospective emergency department study. J. Clin. Psychiat. 75, 1380 (2014).

    Google Scholar 

  7. 7.

    Galatzer-Levy, I. R. et al. Early PTSD symptom trajectories: persistence, recovery, and response to treatment: results from the Jerusalem Trauma Outreach and Prevention Study (J-TOPS). PLoS ONE 8, e70084 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Shalev, A. Y. et al. Long-term outcome of early interventions to prevent posttraumatic stress disorder. J. Clin. Psychiat. 77, e580–e587 (2016).

    Google Scholar 

  9. 9.

    Shalev, A. Y. et al. Prevention of posttraumatic stress disorder by early treatment: results from the Jerusalem Trauma Outreach And Prevention study. Arch. Gen. Psychiatry 69, 166–176 (2012).

    PubMed  Google Scholar 

  10. 10.

    Roberts, N. P., Kitchiner, N. J., Kenardy, J. & Bisson, J. I. Early psychological interventions to treat acute traumatic stress symptoms. Cochrane Database Syst. Rev. 17, CD007944 (2010).

    Google Scholar 

  11. 11.

    Shalev, A. Y. & Barbano, A. C. PTSD: risk assessment and early management. Psychiatr. Ann. 49, 299–306 (2019).

    Google Scholar 

  12. 12.

    Galatzer-Levy, I. R., Karstoft, K. I., Statnikov, A. & Shalev, A. Y. Quantitative forecasting of PTSD from early trauma responses: a machine-learning application. J. Psychiatr. Res. 59, 68–76 (2014).

    PubMed  PubMed Central  Google Scholar 

  13. 13.

    Galatzer-Levy, I. R., Ma, S., Statnikov, A., Yehuda, R. & Shalev, A. Y. Utilization of machine learning for prediction of post-traumatic stress: a re-examination of cortisol in the prediction and pathways to non-remitting PTSD. Transl. Psychiatry 7, e1070 (2017).

    CAS  PubMed Central  Google Scholar 

  14. 14.

    Karstoft, K.-I., Galatzer-Levy, I. R., Statnikov, A., Li, Z. & Shalev, A. Y. Bridging a translational gap: using machine learning to improve the prediction of PTSD. BMC Psychiatry 15, 30 (2015).

    PubMed  PubMed Central  Google Scholar 

  15. 15.

    Shalev, A. Y. et al. A prospective study of heart rate response following trauma and the subsequent development of posttraumatic stress disorder. Arch. Gen. Psychiatry 55, 553–559 (1998).

    CAS  PubMed  Google Scholar 

  16. 16.

    Yehuda, R., McFarlane, A. & Shalev, A. Predicting the development of posttraumatic stress disorder from the acute response to a traumatic event. Biol. Psychiatry 44, 1305–1313 (1998).

    CAS  PubMed  Google Scholar 

  17. 17.

    Papini, S. et al. Ensemble machine-learning prediction of posttraumatic stress disorder screening status after emergency room hospitalization. J. Anxiety Disord. 60, 35–42 (2018).

    PubMed  PubMed Central  Google Scholar 

  18. 18.

    Ressler, K. J. Molecular signatures of stress and posttraumatic stress disorder: an overview. Biol. Psychiatry 83, 792–794 (2018).

    PubMed  Google Scholar 

  19. 19.

    Hinrichs, R. et al. Increased skin conductance response in the immediate aftermath of trauma predicts PTSD risk. Chronic Stress 3, 2470547019844441 (2019).

    PubMed Central  Google Scholar 

  20. 20.

    Heim, C., Schultebraucks, K., Marmar, C. R. & Nemeroff, C. B. in Post‐Traumatic Stress Disorder (eds Nemeroff, C. B. & Marmar, C.) 331 (Oxford Univ. Press, 2018).

  21. 21.

    Morris, M. C., Hellman, N., Abelson, J. L. & Rao, U. Cortisol, heart rate, and blood pressure as early markers of PTSD risk: a systematic review and meta-analysis. Clin. Psychol. Rev. 49, 79–91 (2016).

    PubMed  PubMed Central  Google Scholar 

  22. 22.

    Van Zuiden, M. et al. Glucocorticoid receptor pathway components predict posttraumatic stress disorder symptom development: a prospective study. Biol. Psychiatry 71, 309–316 (2012).

    PubMed  Google Scholar 

  23. 23.

    Schultebraucks, K. et al. Heightened biological stress response during exposure to a trauma film predicts an increase in intrusive memories. J. Abnorm. Psychol. 128, 645 (2019).

    PubMed  Google Scholar 

  24. 24.

    Michopoulos, V. et al. Association of prospective risk for chronic PTSD symptoms with low TNF-α and IFN-γ concentrations in the immediate aftermath of trauma exposure. Am. J. Psychiatry 2019, 19010039 (2019).

    Google Scholar 

  25. 25.

    Mellon, S. H., Gautam, A., Hammamieh, R., Jett, M. & Wolkowitz, O. M. Metabolism, metabolomics, and inflammation in posttraumatic stress disorder. Biol. Psychiatry 83, 866–875 (2018).

    CAS  PubMed  Google Scholar 

  26. 26.

    Michopoulos, V., Powers, A., Gillespie, C. F., Ressler, K. J. & Jovanovic, T. Inflammation in fear-and anxiety-based disorders: PTSD, GAD, and beyond. Neuropsychopharmacology 42, 254 (2017).

    CAS  PubMed  Google Scholar 

  27. 27.

    Nugent, N. R., Christopher, N. C. & Delahanty, D. L. Emergency medical service and in-hospital vital signs as predictors of subsequent PTSD symptom severity in pediatric injury patients. J. Child Psychol. Psychiatry 47, 919–926 (2006).

    PubMed  Google Scholar 

  28. 28.

    Shalev, A., Liberzon, I. & Marmar, C. Post-traumatic stress disorder. New Engl. J. Med. 376, 2459–2469 (2017).

    PubMed  Google Scholar 

  29. 29.

    Love, J. & Zatzick, D. Screening and intervention for comorbid substance disorders, PTSD, depression, and suicide: a trauma center survey. Psychiatr. Serv. 65, 918–923 (2014).

    PubMed  PubMed Central  Google Scholar 

  30. 30.

    Vermetten, E., Zhohar, J. & Krugers, H. J. Pharmacotherapy in the aftermath of trauma; opportunities in the ‘golden hours’. Curr. Psychiatry Rep. 16, 455 (2014).

    PubMed  Google Scholar 

  31. 31.

    Altman, D. G., Vergouwe, Y., Royston, P. & Moons, K. G. Prognosis and prognostic research: validating a prognostic model. Br. Med. J. 338, b605 (2009).

    Google Scholar 

  32. 32.

    Schultebraucks, K. & Galatzer-Levy, I. R. Machine learning for prediction of posttraumatic stress and resilience following trauma: an overview of basic concepts and recent advances. J. Trauma Stress 32, 215–225 (2019).

    PubMed  Google Scholar 

  33. 33.

    McLean, S. A. et al. The AURORA study: a longitudinal, multimodal library of brain biology and function after traumatic stress exposure. Mol. Psychiatry 25, 283–296 (2020).

    PubMed  Google Scholar 

  34. 34.

    Horwitz, L. I., Kuznetsova, M. & Jones, S. A. Creating a learning health system through rapid-cycle, randomized testing. New Engl. J. Med. 381, 1175 (2019).

    PubMed  Google Scholar 

  35. 35.

    American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Publishing, 2013).

  36. 36.

    Fein, J. A., Kassam-Adams, N., Vu, T. & Datner, E. M. Emergency department evaluation of acute stress disorder symptoms in violently injured youths. Ann. Emerg. Med. 38, 391–396 (2001).

    CAS  PubMed  Google Scholar 

  37. 37.

    Marmar, C. R, Weiss, D. S. & Metzler, T. J. The Peritraumatic Dissociative Experiences Questionnaire. in Assessing Psychological Trauma and PTSD 2nd edn (eds Wilson, J. P. & Kean, T. M.) 144–167 (Guilford Press, 2004).

  38. 38.

    Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30, 4765–4774 (2017).

    Google Scholar 

  39. 39.

    Blevins, C. A., Weathers, F. W., Davis, M. T., Witte, T. K. & Domino, J. L. The posttraumatic stress disorder checklist for DSM‐5 (PCL‐5): development and initial psychometric evaluation. J. Trauma. Stress 28, 489–498 (2015).

    PubMed  Google Scholar 

  40. 40.

    Schultebraucks, K., Wen, T., Kronish, I. M., Willey, J. & Chang, B. P. Post-traumatic stress disorder following acute stroke. Curr. Emerg. Hosp. Med. Rep. 8, 1–8 (2020).

    Google Scholar 

  41. 41.

    World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. J. Am. Med. Assoc. 310, 2191 (2013).

  42. 42.

    Foa, E. B., Riggs, D. S., Dancu, C. V. & Rothbaum, B. O. Reliability and validity of a brief instrument for assessing post‐traumatic stress disorder. J. Trauma. Stress 6, 459–473 (1993).

    Google Scholar 

  43. 43.

    Falsetti, S. A., Resnick, H. S., Resick, P. A. & Kilpatrick, D. G. The modified PTSD symptom scale: a brief self-report measure of posttraumatic stress disorder. Behav. Ther. 16, 161–162 (1993).

    Google Scholar 

  44. 44.

    Weathers, F. W., et al. The PTSD checklist for DSM-5 (PCL-5). Department of Verterans Affairs http://www.ptsd.va.gov (2013).

  45. 45.

    Ruglass, L. M., Papini, S., Trub, L. & Hien, D. A. Psychometric properties of the modified posttraumatic stress disorder symptom scale among women with posttraumatic stress disorder and substance use disorders receiving outpatient group treatments. J. Trauma. Stress Disord. Treat. https://doi.org/10.4172/2324-8947.1000139 (2014).

  46. 46.

    Weathers, F. W. Redefining posttraumatic stress disorder for DSM-5. Curr. Opin. Psychol. 14, 122–126 (2017).

    PubMed  Google Scholar 

  47. 47.

    Muthén, L. K. & Muthén, B. O. Mplus User’s Guide: Statistical Analysis with Latent Variables (Muthén & Muthén, 1998–2017).

  48. 48.

    van de Schoot, R., Sijbrandij, M., Winter, S. D., Depaoli, S. & Vermunt, J. K. The GRoLTS-checklist: guidelines for reporting on latent trajectory studies. Struct. Equ. Modeling 24, 451–467 (2017).

    Google Scholar 

  49. 49.

    Kuhn, M. & Johnson, K. Applied Predictive Modeling (Springer, 2013).

  50. 50.

    Young, S., Abdou, T. & Bener, A. Deep Super Learner: a deep ensemble for classification problems. in Advances in Artificial Intelligence. Canadian Conference on Artificial Intelligence (Canadian AI 2018) (eds Bagheri, E. & Cheung, J. C. K.) 84–95 (2018).

  51. 51.

    Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).

    Google Scholar 

  52. 52.

    Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).

    Google Scholar 

  53. 53.

    Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning, (Springer, 2001).

  54. 54.

    Ferri, C., Hernández-Orallo, J. & Modroiu, R. An experimental comparison of performance measures for classification. Pattern Recognit. Lett. 30, 27–38 (2009).

    Google Scholar 

  55. 55.

    Zhou, Z.-H. & Feng, J. Deep forest. Nat. Sci. Rev. 6, 74–86 (2018).

    Google Scholar 

  56. 56.

    Fawcett, T. ROC graphs: notes and practical considerations for researchers. Mach. Learn. 31, 1–38 (2004).

    Google Scholar 

  57. 57.

    Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 13, 1 (2015).

    PubMed  PubMed Central  Google Scholar 

  58. 58.

    Luo, W. et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J. Med. Internet Res. 18, e323 (2016).

    PubMed  PubMed Central  Google Scholar 

  59. 59.

    Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E. & Altman, D. Prognosis and prognostic research: what, why, and how? Br. Med. J. 338, b605 (2009).

    Google Scholar 

  60. 60.

    Štrumbelj, E. & Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 41, 647–665 (2014).

    Google Scholar 

Download references

Acknowledgements

K.S. was supported by the German Research Foundation (SCHU 3259/1–1). The study was also supported by K01MH102415 (I.R.G.-L.), R01MH094759 (C.B.N.) and R01MH094757 (K.J.R.).

Author information

Affiliations

Authors

Contributions

I.R.G.-L., K.J.R, C.B.N., A.Y.S., C.R.M., B.O.R., V.M., T.J. and K.S. substantially contributed to the design of the study and developed the study concept. S.-M.S., I.R.G.-L., V.M., J.S.S., J.L.M.-K., C.R.G. were involved in the data collection process. K.S. developed the data analytical plan and performed data analysis. G.A.B. and I.R.G.-L. provided supervision. K.S. wrote the first draft of the manuscript and all co-authors reviewed and revised the manuscript critically for important intellectual content. All co-authors approved the version of the manuscript to be published.

Corresponding author

Correspondence to Katharina Schultebraucks.

Ethics declarations

Competing interests

B.O.R. has funding from Wounded Warrior Project, Department of Defense Clinical Trial Grant No.W81XWH-10-1-1045, National Institute of Mental Health grant no. 1R01MH094757-01 and McCormick Foundation. B.O.R. also receives royalties from Oxford University Press, Guilford, APPI and Emory University and received advisory board payments from Genentech, Jazz Pharmaceuticals, Sophren, Nobilis Therapeutics, Neuronetics and Aptinyx. C.R.M. serves on the scientific advisory board and has equity in Receptor Life Sciences. He also serves on the PTSD advisory board for Otsuka Pharmaceutical. He receives support from the National Institute on Alcohol Abuse and Alcoholism, National Institute of Mental Health, Department of Defense, US Army Congressionally Directed Medical Research Program, the Steven & Alexander Cohen Foundation, Cohen Veterans Bioscience, Cohen Veterans Network, Home Depot Foundation, McCormick Foundation, Robin Hood Foundation and the City of New York. C.B.N. discloses the following: research/grants from National Institutes of Health and Stanley Medical Research Institute; consulting (last three years) at Xhale, Takeda, Taisho Pharmaceutical Inc., Bracket (Clintara), Fortress Biotech, Sunovion Pharmaceuticals Inc., Sumitomo Dainippon Pharma, Janssen Research & Development LLC, Magstim, Inc., Navitor Pharmaceuticals, Inc., TC MSO, Inc. and Intra-Cellular Therapies, Inc.; stockholder of Xhale, Celgene, Seattle Genetics, Abbvie, OPKO Health, Inc., Antares, BI Gen Holdings, Inc., Corcept Therapeutics Pharmaceuticals Company, TC MSO, Inc. and Trends in Pharma Development, LLC; scientific advisory boards of American Foundation for Suicide Prevention (AFSP), Brain and Behavior Research Foundation, Xhale, Anxiety Disorders Association of America (ADAA), Skyland Trail, Bracket (Clintara) and Laureate Institute for Brain Research Inc.; board of directors of AFSP, Gratitude America, ADAA and Xhale Smart, Inc.; income sources or equity of USD$10,000 or more from American Psychiatric Publishing, Xhale, Bracket (Clintara), CME Outfitters, Takeda, Intra-Cellular Therapies, Inc., Magstim and EMA Wellness; patents for the method and devices for transdermal delivery of lithium (US 6,375,990B1), the method of assessing antidepressant drug therapy via transport inhibition of monoamine neurotransmitters by ex vivo assay (US 7,148,027B2) and compounds, compositions, methods of synthesis and methods of treatment (CRF receptor binding ligand) (US 8,551,996 B2). K.J.R. performs consulting for Janssen, Verily, Alkermes and Biogen, Inc. on matters unrelated to this manuscript. I.R.G.-L. receives salary and stock options from AiCure. All other authors declare no competing interests.

Additional information

Peer review information Kate Gao was the primary editor on this article, and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Schematic overview of the study design.

Displayed are the basic steps of the model development and model validation.

Extended Data Fig. 2 Predictive performance in terms of discrimination and calibration.

In panel a, the ROC curve shows the specificity and the sensitivity of the predictions on the training set (blue line) and the external validation set (orange line) and is accompanied by a calibration plot for the predicted probabilities on the training set (orange line and blue bars) and the external validation set (red line and green bars) in panel b. The bars in the calibration plot in panel (b) displays the predicted probabilities in 10 bins [0, 10%], (10%, 20%],…, (90%, 100%], whereas the lines visualize the predicted probabilities in two bins (low vs. high probability).

Supplementary information

Supplementary Information

Supplementary Figs. 1–7, Supplementary Tables 1–10, Supplementary Discussion.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schultebraucks, K., Shalev, A.Y., Michopoulos, V. et al. A validated predictive algorithm of post-traumatic stress course following emergency department admission after a traumatic stressor. Nat Med 26, 1084–1088 (2020). https://doi.org/10.1038/s41591-020-0951-z

Download citation

Further reading