Abstract
Annually, approximately 30 million patients are discharged from the emergency department (ED) after a traumatic event1. These patients are at substantial psychiatric risk, with approximately 10–20% developing one or more disorders, including anxiety, depression or post-traumatic stress disorder (PTSD)2,3,4. At present, no accurate method exists to predict the development of PTSD symptoms upon ED admission after trauma5. Accurate risk identification at the point of treatment by ED services is necessary to inform the targeted deployment of existing treatment6,7,8,9 to mitigate subsequent psychopathology in high-risk populations10,11. This work reports the development and validation of an algorithm for prediction of post-traumatic stress course over 12 months using two independently collected prospective cohorts of trauma survivors from two level 1 emergency trauma centers, which uses routinely collectible data from electronic medical records, along with brief clinical assessments of the patient’s immediate stress reaction. Results demonstrate externally validated accuracy to discriminate PTSD risk with high precision. While the predictive algorithm yields useful reproducible results on two independent prospective cohorts of ED patients, future research should extend the generalizability to the broad, clinically heterogeneous ED population under conditions of routine medical care.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
All requests for raw and analyzed data and related materials, including programming code, will be reviewed by our legal departments (New York University Grossman School of Medicine and Emory University School of Medicine) to verify whether the request is subject to any intellectual property or confidentiality constraints. Any data and materials that can be shared will be released via a material transfer agreement for noncommercial research purposes. Request should be addressed to the corresponding author (K.S.) or the Principal Investigators of the two study sites (K.J.R. and I.R.G.-L.).
Code availability
The programming code is based on Scikit-learn (https://scikit-learn.org/stable/) and SHAP (https://github.com/slundberg/shap) and the core algorithm can be obtained from https://github.com/KSchultebraucks/DeepSuperLearner. Requests should be addressed to the corresponding author (K.S.).
References
DiMaggio, C. J., Avraham, J. B., Lee, D. C., Frangos, S. G. & Wall, S. P. The epidemiology of emergency department trauma discharges in the United States. Acad. Emerg. Med. 24, 1244–1256 (2017).
Wiseman, T. A., Curtis, K., Lam, M. & Foster, K. Incidence of depression, anxiety and stress following traumatic injury: a longitudinal study. Scand. J. Trauma Resusc. Emerg. Med. 23, 29 (2015).
Sullivan, E. et al. The association between posttraumatic stress symptoms, depression, and length of hospital stay following traumatic injury. Gen. Hosp. Psychiatry 46, 49–54 (2017).
Fakhry, S. M. et al. Continuing trauma: the unmet needs of trauma patients in the postacute care setting. Am. Surgeon 83, 1308–1314 (2017).
Shalev, A. Y. et al. Estimating the risk of PTSD in recent trauma survivors: results of the International Consortium to Predict PTSD (ICPP). World Psychiatry 18, 77–87 (2019).
Rothbaum, B. O. et al. Early intervention following trauma may mitigate genetic risk for PTSD in civilians: a pilot prospective emergency department study. J. Clin. Psychiat. 75, 1380 (2014).
Galatzer-Levy, I. R. et al. Early PTSD symptom trajectories: persistence, recovery, and response to treatment: results from the Jerusalem Trauma Outreach and Prevention Study (J-TOPS). PLoS ONE 8, e70084 (2013).
Shalev, A. Y. et al. Long-term outcome of early interventions to prevent posttraumatic stress disorder. J. Clin. Psychiat. 77, e580–e587 (2016).
Shalev, A. Y. et al. Prevention of posttraumatic stress disorder by early treatment: results from the Jerusalem Trauma Outreach And Prevention study. Arch. Gen. Psychiatry 69, 166–176 (2012).
Roberts, N. P., Kitchiner, N. J., Kenardy, J. & Bisson, J. I. Early psychological interventions to treat acute traumatic stress symptoms. Cochrane Database Syst. Rev. 17, CD007944 (2010).
Shalev, A. Y. & Barbano, A. C. PTSD: risk assessment and early management. Psychiatr. Ann. 49, 299–306 (2019).
Galatzer-Levy, I. R., Karstoft, K. I., Statnikov, A. & Shalev, A. Y. Quantitative forecasting of PTSD from early trauma responses: a machine-learning application. J. Psychiatr. Res. 59, 68–76 (2014).
Galatzer-Levy, I. R., Ma, S., Statnikov, A., Yehuda, R. & Shalev, A. Y. Utilization of machine learning for prediction of post-traumatic stress: a re-examination of cortisol in the prediction and pathways to non-remitting PTSD. Transl. Psychiatry 7, e1070 (2017).
Karstoft, K.-I., Galatzer-Levy, I. R., Statnikov, A., Li, Z. & Shalev, A. Y. Bridging a translational gap: using machine learning to improve the prediction of PTSD. BMC Psychiatry 15, 30 (2015).
Shalev, A. Y. et al. A prospective study of heart rate response following trauma and the subsequent development of posttraumatic stress disorder. Arch. Gen. Psychiatry 55, 553–559 (1998).
Yehuda, R., McFarlane, A. & Shalev, A. Predicting the development of posttraumatic stress disorder from the acute response to a traumatic event. Biol. Psychiatry 44, 1305–1313 (1998).
Papini, S. et al. Ensemble machine-learning prediction of posttraumatic stress disorder screening status after emergency room hospitalization. J. Anxiety Disord. 60, 35–42 (2018).
Ressler, K. J. Molecular signatures of stress and posttraumatic stress disorder: an overview. Biol. Psychiatry 83, 792–794 (2018).
Hinrichs, R. et al. Increased skin conductance response in the immediate aftermath of trauma predicts PTSD risk. Chronic Stress 3, 2470547019844441 (2019).
Heim, C., Schultebraucks, K., Marmar, C. R. & Nemeroff, C. B. in Post‐Traumatic Stress Disorder (eds Nemeroff, C. B. & Marmar, C.) 331 (Oxford Univ. Press, 2018).
Morris, M. C., Hellman, N., Abelson, J. L. & Rao, U. Cortisol, heart rate, and blood pressure as early markers of PTSD risk: a systematic review and meta-analysis. Clin. Psychol. Rev. 49, 79–91 (2016).
Van Zuiden, M. et al. Glucocorticoid receptor pathway components predict posttraumatic stress disorder symptom development: a prospective study. Biol. Psychiatry 71, 309–316 (2012).
Schultebraucks, K. et al. Heightened biological stress response during exposure to a trauma film predicts an increase in intrusive memories. J. Abnorm. Psychol. 128, 645 (2019).
Michopoulos, V. et al. Association of prospective risk for chronic PTSD symptoms with low TNF-α and IFN-γ concentrations in the immediate aftermath of trauma exposure. Am. J. Psychiatry 2019, 19010039 (2019).
Mellon, S. H., Gautam, A., Hammamieh, R., Jett, M. & Wolkowitz, O. M. Metabolism, metabolomics, and inflammation in posttraumatic stress disorder. Biol. Psychiatry 83, 866–875 (2018).
Michopoulos, V., Powers, A., Gillespie, C. F., Ressler, K. J. & Jovanovic, T. Inflammation in fear-and anxiety-based disorders: PTSD, GAD, and beyond. Neuropsychopharmacology 42, 254 (2017).
Nugent, N. R., Christopher, N. C. & Delahanty, D. L. Emergency medical service and in-hospital vital signs as predictors of subsequent PTSD symptom severity in pediatric injury patients. J. Child Psychol. Psychiatry 47, 919–926 (2006).
Shalev, A., Liberzon, I. & Marmar, C. Post-traumatic stress disorder. New Engl. J. Med. 376, 2459–2469 (2017).
Love, J. & Zatzick, D. Screening and intervention for comorbid substance disorders, PTSD, depression, and suicide: a trauma center survey. Psychiatr. Serv. 65, 918–923 (2014).
Vermetten, E., Zhohar, J. & Krugers, H. J. Pharmacotherapy in the aftermath of trauma; opportunities in the ‘golden hours’. Curr. Psychiatry Rep. 16, 455 (2014).
Altman, D. G., Vergouwe, Y., Royston, P. & Moons, K. G. Prognosis and prognostic research: validating a prognostic model. Br. Med. J. 338, b605 (2009).
Schultebraucks, K. & Galatzer-Levy, I. R. Machine learning for prediction of posttraumatic stress and resilience following trauma: an overview of basic concepts and recent advances. J. Trauma Stress 32, 215–225 (2019).
McLean, S. A. et al. The AURORA study: a longitudinal, multimodal library of brain biology and function after traumatic stress exposure. Mol. Psychiatry 25, 283–296 (2020).
Horwitz, L. I., Kuznetsova, M. & Jones, S. A. Creating a learning health system through rapid-cycle, randomized testing. New Engl. J. Med. 381, 1175 (2019).
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Publishing, 2013).
Fein, J. A., Kassam-Adams, N., Vu, T. & Datner, E. M. Emergency department evaluation of acute stress disorder symptoms in violently injured youths. Ann. Emerg. Med. 38, 391–396 (2001).
Marmar, C. R, Weiss, D. S. & Metzler, T. J. The Peritraumatic Dissociative Experiences Questionnaire. in Assessing Psychological Trauma and PTSD 2nd edn (eds Wilson, J. P. & Kean, T. M.) 144–167 (Guilford Press, 2004).
Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30, 4765–4774 (2017).
Blevins, C. A., Weathers, F. W., Davis, M. T., Witte, T. K. & Domino, J. L. The posttraumatic stress disorder checklist for DSM‐5 (PCL‐5): development and initial psychometric evaluation. J. Trauma. Stress 28, 489–498 (2015).
Schultebraucks, K., Wen, T., Kronish, I. M., Willey, J. & Chang, B. P. Post-traumatic stress disorder following acute stroke. Curr. Emerg. Hosp. Med. Rep. 8, 1–8 (2020).
World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. J. Am. Med. Assoc. 310, 2191 (2013).
Foa, E. B., Riggs, D. S., Dancu, C. V. & Rothbaum, B. O. Reliability and validity of a brief instrument for assessing post‐traumatic stress disorder. J. Trauma. Stress 6, 459–473 (1993).
Falsetti, S. A., Resnick, H. S., Resick, P. A. & Kilpatrick, D. G. The modified PTSD symptom scale: a brief self-report measure of posttraumatic stress disorder. Behav. Ther. 16, 161–162 (1993).
Weathers, F. W., et al. The PTSD checklist for DSM-5 (PCL-5). Department of Verterans Affairs http://www.ptsd.va.gov (2013).
Ruglass, L. M., Papini, S., Trub, L. & Hien, D. A. Psychometric properties of the modified posttraumatic stress disorder symptom scale among women with posttraumatic stress disorder and substance use disorders receiving outpatient group treatments. J. Trauma. Stress Disord. Treat. https://doi.org/10.4172/2324-8947.1000139 (2014).
Weathers, F. W. Redefining posttraumatic stress disorder for DSM-5. Curr. Opin. Psychol. 14, 122–126 (2017).
Muthén, L. K. & Muthén, B. O. Mplus User’s Guide: Statistical Analysis with Latent Variables (Muthén & Muthén, 1998–2017).
van de Schoot, R., Sijbrandij, M., Winter, S. D., Depaoli, S. & Vermunt, J. K. The GRoLTS-checklist: guidelines for reporting on latent trajectory studies. Struct. Equ. Modeling 24, 451–467 (2017).
Kuhn, M. & Johnson, K. Applied Predictive Modeling (Springer, 2013).
Young, S., Abdou, T. & Bener, A. Deep Super Learner: a deep ensemble for classification problems. in Advances in Artificial Intelligence. Canadian Conference on Artificial Intelligence (Canadian AI 2018) (eds Bagheri, E. & Cheung, J. C. K.) 84–95 (2018).
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).
Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning, (Springer, 2001).
Ferri, C., Hernández-Orallo, J. & Modroiu, R. An experimental comparison of performance measures for classification. Pattern Recognit. Lett. 30, 27–38 (2009).
Zhou, Z.-H. & Feng, J. Deep forest. Nat. Sci. Rev. 6, 74–86 (2018).
Fawcett, T. ROC graphs: notes and practical considerations for researchers. Mach. Learn. 31, 1–38 (2004).
Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 13, 1 (2015).
Luo, W. et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J. Med. Internet Res. 18, e323 (2016).
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E. & Altman, D. Prognosis and prognostic research: what, why, and how? Br. Med. J. 338, b605 (2009).
Štrumbelj, E. & Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 41, 647–665 (2014).
Acknowledgements
K.S. was supported by the German Research Foundation (SCHU 3259/1–1). The study was also supported by K01MH102415 (I.R.G.-L.), R01MH094759 (C.B.N.) and R01MH094757 (K.J.R.).
Author information
Authors and Affiliations
Contributions
I.R.G.-L., K.J.R, C.B.N., A.Y.S., C.R.M., B.O.R., V.M., T.J. and K.S. substantially contributed to the design of the study and developed the study concept. S.-M.S., I.R.G.-L., V.M., J.S.S., J.L.M.-K., C.R.G. were involved in the data collection process. K.S. developed the data analytical plan and performed data analysis. G.A.B. and I.R.G.-L. provided supervision. K.S. wrote the first draft of the manuscript and all co-authors reviewed and revised the manuscript critically for important intellectual content. All co-authors approved the version of the manuscript to be published.
Corresponding author
Ethics declarations
Competing interests
B.O.R. has funding from Wounded Warrior Project, Department of Defense Clinical Trial Grant No.W81XWH-10-1-1045, National Institute of Mental Health grant no. 1R01MH094757-01 and McCormick Foundation. B.O.R. also receives royalties from Oxford University Press, Guilford, APPI and Emory University and received advisory board payments from Genentech, Jazz Pharmaceuticals, Sophren, Nobilis Therapeutics, Neuronetics and Aptinyx. C.R.M. serves on the scientific advisory board and has equity in Receptor Life Sciences. He also serves on the PTSD advisory board for Otsuka Pharmaceutical. He receives support from the National Institute on Alcohol Abuse and Alcoholism, National Institute of Mental Health, Department of Defense, US Army Congressionally Directed Medical Research Program, the Steven & Alexander Cohen Foundation, Cohen Veterans Bioscience, Cohen Veterans Network, Home Depot Foundation, McCormick Foundation, Robin Hood Foundation and the City of New York. C.B.N. discloses the following: research/grants from National Institutes of Health and Stanley Medical Research Institute; consulting (last three years) at Xhale, Takeda, Taisho Pharmaceutical Inc., Bracket (Clintara), Fortress Biotech, Sunovion Pharmaceuticals Inc., Sumitomo Dainippon Pharma, Janssen Research & Development LLC, Magstim, Inc., Navitor Pharmaceuticals, Inc., TC MSO, Inc. and Intra-Cellular Therapies, Inc.; stockholder of Xhale, Celgene, Seattle Genetics, Abbvie, OPKO Health, Inc., Antares, BI Gen Holdings, Inc., Corcept Therapeutics Pharmaceuticals Company, TC MSO, Inc. and Trends in Pharma Development, LLC; scientific advisory boards of American Foundation for Suicide Prevention (AFSP), Brain and Behavior Research Foundation, Xhale, Anxiety Disorders Association of America (ADAA), Skyland Trail, Bracket (Clintara) and Laureate Institute for Brain Research Inc.; board of directors of AFSP, Gratitude America, ADAA and Xhale Smart, Inc.; income sources or equity of USD$10,000 or more from American Psychiatric Publishing, Xhale, Bracket (Clintara), CME Outfitters, Takeda, Intra-Cellular Therapies, Inc., Magstim and EMA Wellness; patents for the method and devices for transdermal delivery of lithium (US 6,375,990B1), the method of assessing antidepressant drug therapy via transport inhibition of monoamine neurotransmitters by ex vivo assay (US 7,148,027B2) and compounds, compositions, methods of synthesis and methods of treatment (CRF receptor binding ligand) (US 8,551,996 B2). K.J.R. performs consulting for Janssen, Verily, Alkermes and Biogen, Inc. on matters unrelated to this manuscript. I.R.G.-L. receives salary and stock options from AiCure. All other authors declare no competing interests.
Additional information
Peer review information Kate Gao was the primary editor on this article, and managed its editorial process and peer review in collaboration with the rest of the editorial team.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Schematic overview of the study design.
Displayed are the basic steps of the model development and model validation.
Extended Data Fig. 2 Predictive performance in terms of discrimination and calibration.
In panel a, the ROC curve shows the specificity and the sensitivity of the predictions on the training set (blue line) and the external validation set (orange line) and is accompanied by a calibration plot for the predicted probabilities on the training set (orange line and blue bars) and the external validation set (red line and green bars) in panel b. The bars in the calibration plot in panel (b) displays the predicted probabilities in 10 bins [0, 10%], (10%, 20%],…, (90%, 100%], whereas the lines visualize the predicted probabilities in two bins (low vs. high probability).
Supplementary information
Supplementary Information
Supplementary Figs. 1–7, Supplementary Tables 1–10, Supplementary Discussion.
Rights and permissions
About this article
Cite this article
Schultebraucks, K., Shalev, A.Y., Michopoulos, V. et al. A validated predictive algorithm of post-traumatic stress course following emergency department admission after a traumatic stressor. Nat Med 26, 1084–1088 (2020). https://doi.org/10.1038/s41591-020-0951-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41591-020-0951-z
This article is cited by
-
Awareness with paralysis and symptoms of post-traumatic stress disorder among mechanically ventilated emergency department survivors (ED-AWARENESS-2 Trial): study protocol for a pragmatic, multicenter, stepped wedge cluster randomized trial
Trials (2023)
-
Development and validation of a machine learning model using electronic health records to predict trauma- and stressor-related psychiatric disorders after hospitalization with sepsis
Translational Psychiatry (2023)
-
Uncovering expression signatures of synergistic drug responses via ensembles of explainable machine-learning models
Nature Biomedical Engineering (2023)
-
Natural language processing for mental health interventions: a systematic review and research framework
Translational Psychiatry (2023)
-
Applications of artificial intelligence−machine learning for detection of stress: a critical overview
Molecular Psychiatry (2023)