There is an ongoing revolution in computational behavioral analysis in business and government. Automated analysis of text is used to screen job applicants and score essays, and is applied to social media to influence individuals’ purchasing and voting choices. Automated face and emotion recognition is used for both surveillance and to supplement polygraph testing. Wearables are used to collect physiological data from athletes and astronauts, and increasingly for medical purposes.

Only recently have these computational approaches been applied in psychiatry to study disturbances in thought, emotion, and behavior, which traditionally have been assessed using only expert human appraisal, codified in standardized interviews and ratings, but labor-intensive and error-prone. Herein, we review a sample of ongoing lines of research, in respect to language, emotional expression and physiological parameters.

Automated speech analysis can characterize intoxication by different drugs of abuse with increased verbosity induced by methamphetamine, and increased semantic proximity to words such as friendship/rapport/support/intimacy characterizing intoxication from MDMA or ‘ecstasy’ (Bedi et al, 2014); acoustic features of speech are similarly discriminative (Agurto et al, 2017). This behavioral readout in speech of drug effects has implications for diagnosis, care and clinical trials, as well as for investigations of neural mechanisms of intoxication.

We have also used automated speech analysis to characterize the subtle language disturbance that precedes psychosis onset in schizophrenia (Bedi et al, 2015), identifying a highly accurate predictor of psychosis that comprises both decrease in sentence-level semantic coherence (indexing tangentiality), and decrease in syntactic complexity (indexing concreteness). Further, we have used automated metaphor identification to show a significantly higher rate of metaphor usage across stages of schizophrenia, including putative prodromal stages (Gutierrez et al, 2017). This technology is portable, inexpensive, and easy to implement, and can improve prognosis and understanding of mechanisms of thought disorder in schizophrenia.

Beyond words themselves, voice and face expression also provide important data. At the ACNP Annual Meeting in 2016, Dr Satrajit Ghosh presented data showing that voice acoustic features can be used to predict severity of depression and Parkinson’s disease (Ghosh, 2016). He introduced VoiceUp, his open source mobile platform for collection and analysis of voice data, a sensor into mental health feasible to track over time. In the same ACNP panel, Dr Justin Baker presented data on ‘face action units,’ which comprise specific movements of the mouth, cheeks, and eyes, as collected by video and analyzed using the OpenFace mobile platform (Baker, 2016). Specific face action unit abnormalities were associated with symptom severity, as was the extent of mutual gaze and vowel space in speech.

The smartphones in our pockets provide complex longitudinal in vivo data, much of it passively obtained, including spatial trajectories (GPS), physical movement, and sleep (accelerometer), and social networks and dynamics (phone communication logs); these ‘big data’ at the level of the individual can be used to promote precision psychiatry (Onnela and Rauch, 2016). Smartphones can record sleep patterns, respiration, and heart rate variability. Physiological data, along with language and facial data, can provide accurate and nuanced real-time readout in telepsychiatry, and lead to deep phenotyping that can be integrated with genetic and neuroimaging data.

Funding and disclosure

This research is funded by grants from the NIMH: R01MH107558 and R03MH108933. The authors declare no conflict of interest.