Fig. 1 | Nature Communications

Fig. 1

From: Real-time decoding of question-and-answer speech dialogue using human cortical activity

Fig. 1

Schematic of real-time speech decoding during a question (blue) and answer (red) task. a On each trial, participants hear a question and see a set of possible answer choices on a screen. b Participants are instructed to freely choose and verbally produce one of the answers when a green response cue appears on the screen. c Simultaneously, cortical activity is acquired from ECoG electrodes implanted across temporal and frontal cortex and then filtered in real-time to extract high gamma activity. d A speech detection model uses the spatiotemporal pattern of high gamma activity to predict whether a question is being heard or an answer is being produced (or neither) at each time point. e When the speech detection model detects a question event, that time window of high gamma activity is passed to a question classifier that uses phone-level Viterbi decoding to compute question utterance likelihoods. f The question with the highest likelihood is output as the decoded question. g To integrate questions and answers, the stimulus set was designed such that each answer was only likely for certain questions (context priors). h These context priors are combined with the predicted question likelihoods to obtain answer priors. i When the speech detection model detects an answer event, that time window of neural activity is passed to an answer classifier that uses phone-level Viterbi decoding to compute answer utterance likelihoods. j The context integration model combines these answer likelihoods with the answer priors to yield answer posterior probabilities (purple). k The answer with the highest posterior probability is output as the decoded answer. The answer choice icons shown in (a, b) were made by www.freepik.com from www.flaticon.com

Back to article page