Introduction

Perceptual learning often requires a significant amount of training and attention to the training task from the participant, particularly when the learning target cannot be perceived, as in auditory discrimination. It is not explicitly clear what and how to learn effectively. For example, native Japanese speakers are usually unable to perceive the difference between /l/ and /r/ sounds1,2,3,4 in English and thus a long duration is required for training. Similarly, much training is needed for native English speakers to discriminate Mandarin tones5. This difficulty can be observed in not only language education, but also other fields such as training people with hearing impairments or learning difficulties, musicians, or sound engineers. Although a recent study6 indicated that visual perceptual improvement was observed using decoded fMRI neurofeedback without stimulus presentation, this technique requires participants to discriminate the target in advance.

Many previous studies7,8,9,10,11 have shown that auditory discrimination ability improves with a significant amount of behavioral training, while mismatch negativity (MMN) becomes stronger as an index of sound discrimination accuracy. The MMN can be elicited by any discriminable auditory change and provides a separate objective measure of the discrimination accuracy for any dimension of auditory stimulation12,13. Interestingly, the MMN response can be detected in the absence of a conscious realization of the contrast14. Furthermore, an MMN can be elicited without the listener subjectively attending to the sound stimuli15,16.

In this study, we developed a novel neurofeedback method where the strength of participants' MMN, as a measure of perceptual discriminability, is presented as visual feedback to provide a continuous cue for learning. While focusing on the visual feedback, participants unconsciously achieved a significant improvement in auditory discrimination of the applied stimuli.

Results

In our experiment, the participants were randomly distributed into the neurofeedback (n = 8:P1,…,P8) and control groups (n = 8:P9,…,P16) to compare the effect of neurofeedback. The sequences of tones used for the neurofeedback group consisted of a standard stimulus (1000 Hz, 80% of tones) and a deviant stimulus (1008 Hz, 20% of tones). The two stimuli were presented in random order every 0.5 s (Fig. 1a). The average amplitude of the MMNs was calculated from responses to the previous 20 stimuli (16 standard and 4 deviant stimuli) and was represented by a solid green disc, the radius of which corresponded to the amplitude of the MMN (Fig. 1a). Participants were instructed to ignore the sounds played through their earphones and concentrate on making the solid green disc presented on the screen as large as possible. The radius of the green disc was fixed for the first 20 stimuli of the session, because these 20 sounds are needed to calculate the MMN. Following this, the radius of the disc was determined every 0.5 s by linearly mapping the amplitude of the MMN (see Methods).

Figure 1
figure 1

Procedure of the main experiment.

(a) The training procedure. (b) The four combinations of auditory stimuli used in the behavioral auditory discrimination (BAD) test. The vertical rectangles for the two tones are hardly discriminable from each other. (c) The trial procedure during the BAD test.

Full size image

Improvement in the behavioral auditory discrimination (BAD) test

To evaluate the improvement in participants' auditory perception, a BAD test was performed before the first day of training (pre-training test) and after each training day. In the BAD test, participants were asked whether two pure tones (the same 1000 Hz and 1008 Hz stimuli that were used for training) were different (Fig. 1b and 1c). Figure 2a shows the performance of all 8 participants in each BAD test. A one-way analysis of variance with repeated measures indicated a significant effect of training (F (5, 42) = 8.40, P < 0.0001). A post-hoc t-test comparing accuracies on subsequent training days revealed that the discrimination between the two pure tones significantly improved on all but the 3rd and 4th days [t (7) = 6.01, P < 0.01; t (7) = 5.01, P < 0.05; and t (7) = 5.49, P < 0.05, with Bonferroni correction for the 1st, 2nd and 5th day, respectively]. Overall, we found a gradual increase in the discrimination accuracy during each training day and an increase in the mean discrimination accuracy on subsequent training days (Fig. 2c). There was a significant improvement of 25.45 ± 3.09% (mean ± s.e.m. across participants) in discrimination accuracy on the final day of training when compared with the results of the pre-training test [t (7) = 8.23, P < 0.001 with a Bonferroni correction].

Figure 2
figure 2

Results of the BAD tests.

(a) The discrimination accuracy on each training day for individual participants in the neurofeedback group (P1,…,p8). (b) The discrimination accuracy on each training day for individual participants in the control group (P9,…P16). (c) The average improvement in discrimination accuracy in the neurofeedback group (red) and the control group (green). Significant differences in discrimination accuracy between the two groups appear after the first training day. (d) The average improvement in accuracy for the neurofeedback and control groups. The improvement in accuracy was obtained by subtracting the accuracy measured on the first BAD test from the last BAD test. Error bars represent s.e.m.* P < 0.05, ** P < 0.01 and *** P < 0.001.

Full size image

Although the participants were asked to ignore the auditory stimuli during training, we hypothesized that they might become accustomed to hearing the stimuli repeatedly and thus, learning might unconsciously occur and auditory discrimination performance might improve. Furthermore, a previous study in perceptual learning reported that repetitive pairing of reward and visual stimuli leads to performance improvement on that stimuli17. Therefore, there is a possibility that the size of the disc had worked as a reinforcement signal and repetitive pairing of this reinforcement signal and auditory stimuli led to the behavioral improvements. To test these possibilities, we performed a control experiment with 8 new participants (control groups: P9,…,P16) who were given the same stimuli and instructions as the neurofeedback group. Electrodes were also attached to the participants, but the sizes of the green discs they were shown did not correspond to their MMN responses. Instead, the sizes corresponded to the sequences of visual stimulus presented to participants in the neurofeedback group. It should be noted that the participants did not know whether they were in the neurofeedback or control group. We measured the performance of each participant in the control group using the BAD test (Fig. 2b). A one-way analysis of variance with repeated measures indicated that there was no significant improvement in performance (F (5, 42) = 0.16, P = 0.98; Fig. 2c). Furthermore, Figure. 2c indicates that there was no significant difference in performance on the pre-test between the neurofeedback and control groups. In addition, the score in the pre-test was not significantly different from chance (50% correct), as tested by a binomial test (the critical score of significant difference was 60.6%). However, we found that the average discrimination performance improved significantly in the neurofeedback group compared with the control group (t (7) = 7.72, P < 0.001) (Fig. 2d). Taken together, these results demonstrate that the improvement in discrimination performance observed in the neurofeedback group was not attributable simply to repeatedly hearing the sound stimuli (Fig. 2c).

Improvement in neural activity

We also assessed whether neural activity changed in the neurofeedback and control groups. Using the electroencephalography (EEG) data collected on each training day, we found that the average MMN amplitude on the last training day was significantly higher when compared with the first training day in the neurofeedback group (t (7) = 2.45, P = 0.044) (Fig. 3a). In addition, the average daily MMN peak latency was significantly shorter on the last day when compared with the first day in the neurofeedback group (t (7) = 2.88, P = 0.024) (Fig. 3b). Furthermore, all participants in the neurofeedback group showed significant changes in at least one of the neural measures (MMN amplitude or peak latency), whereas no significant changes were observed in either amplitude or latency of MMNs in the control group (see Supplementary Fig. S1 online). Figure 3c shows the group grand average MMN responses on the 1st and 5th training days in the neurofeedback and control groups, respectively. However, due to the difference in peak latency between participants, the result shown in fig. 3c differs somewhat from fig. 3a and 3b.

Figure 3
figure 3

Changes in MMN in the neurofeedback and control groups.

(a) The average MMN amplitude on the 1st and 5th training days in the neurofeedback (n = 8) and control (n = 8) groups. (b) The average MMN peak latency on the 1st and 5th training days in the neurofeedback and control groups. The error bars represent s.e.m. * P < 0.05. (c) The group grand average MMN responses on the 1st and 5th training days in the neurofeedback (n = 8) and control (n = 8) groups respectively.

Full size image

Discussion

A recent study has reported that visual perceptual learning can be achieved by inducing activity in the visual cortex that corresponds to orientation detection, using decoded functional magnetic resonance imaging (fMRI) neurofeedback, without stimulus presentation or the participants' subjective awareness of the aim of learning6. Our results support the view that changes in brain activity are associated with behavioral performance improvements, even when subjects are not conscious of them. To use the decoded fMRI technique, it would be necessary to determine which brain region relates to perceptual learning and how it is activated to achieve perceptual learning for a specific target. Therefore, perceptual learning using the decoded fMRI technique can only be achieved when the participants are able to perceive the target. In contrast, the method using MMN used here can affect these neural changes without identifying the specific brain activity pattern or the regions impacted by individual sound features. This is because the MMN has been widely used and is known as an index of sound discrimination accuracy and is elicited by any discriminable auditory change12,13, in the absence of a conscious realization of the contrast14. In addition, it can be detected without knowing the brain regions or specific brain patterns associated with low-level processes. Therefore, our method enables participants to perceive auditory differences that they could not previously perceive. Compared with expensive fMRI devices, an EEG device is more affordable and accessible and the MMN response is easy to record. Finally, EEG devices can easily be equipped with various advanced functions, such as stable active electrodes, fast fit caps and compact mobile setups.

Previous studies have shown that the MMN response is correlated with discrimination accuracy and persistence of sensory memory for sounds18,19,20,21. In our study the improvements in discrimination accuracy were confirmed by BAD tests; therefore, it is reasonable to conclude that our neurofeedback method improved discrimination accuracy. Several studies7,8,9,10,11 have shown greater amplitude, shorter latency, or both, of the MMN after behavioral training. In these studies, the MMN was not elicited initially by slight sound changes that the participants were unable to discriminate, but emerged in those participants who were trained behaviorally to discriminate stimulus changes (such as frequency, phoneme and syllable in speech sounds). Furthermore, another study22 has shown that the deficits of pitch discrimination ability and MMN are positively correlated with the degree of impairment in phonological skills, as reflected in reading errors of regular words and non-words. Interestingly, some data8 have shown that in the course of training, MMN might emerge before the improvement in stimulus-identification ability. Our results indicate that BAD performance can be improved by enhancing brain activity alone, even without behavioral discrimination training.

Interestingly, the participants of our study were not aware of the purpose of the experiment. When they were asked after the BAD test on the last training day how they made the disc size change, none of their responses were related to the acoustic stimuli of the experiment (see “cognitive strategies” in the Supplementary Information). Because our new neurofeedback method does not require learners to pay attention to the auditory stimuli13,14,23,24, or to be aware of the learning process, it can improve discrimination skills unconsciously.

Our results indicate that an adult can gain auditory discrimination abilities unconsciously without any behavioral training. Therefore, it is possible to learn to discriminate similar sounds that even cannot be perceived as different. Furthermore, our neurofeedback method is more helpful and powerful than behavioral training, because this method could provide learners with a continuous and accurate measure of their progress in learning to discriminate sounds, rather than just a binary feedback (correct/incorrect) on their behavioral output. Based on our findings, the method also has potential to be developed as an unconscious learning interface device, where users simply enjoy a brain-computer interface (BCI) game with the learning target sounds. In this case, a certain goal can be achieved by obtaining a large MMN while users unconsciously improve their listening ability.

Methods

Experimental details

Throughout each training session, the participants were seated in an antistatic chair in front of a 23-inch computer screen. Stimuli were presented binaurally via earphones. Event-related potential (ERP) responses were measured with the MP150 Data Acquisition System (BIOPAC Systems, Inc. Goleta, California USA). Responses were recorded with a sampling rate of 500 Hz. A band pass filter of 0.1–35 Hz was applied. Voltage variations caused by horizontal or vertical eye movements were monitored with an electrode attached to the outer canthus of the left eye. Recordings that contained voltage variations due to eye movements or other extracerebral artifacts exceeding ±40 μV were omitted.

Preliminary experiment

We conducted a preliminary experiment in which the MMN was calculated using an auditory stimulus sequence of 1000 Hz and 2000 Hz tones as the standard and deviant stimuli in the oddball paradigm. These two tones are easily discriminable from each other; therefore, the amplitude of the MMN for the two tones (1000 Hz and 2000 Hz) was used as the maximum value (preMAX) and the disc's maximum possible size was set to 4.97 deg radius6. The minimum size of the feedback disc is the size of the white fixation point (0.4 deg radius) presented in the center of the display. The calculation formula was: SIZE = 4.57 * MMN/preMAX + 0.4 (deg radius). The size could not become larger than the maximum possible size even if MMN was greater than the preMAX.

Behavioral auditory discrimination (BAD) test

In the BAD test, a two-alternative forced choice task was performed, in which the accuracy of behavioral auditory discrimination was assessed using simple sinusoidal tones of 1000 Hz and 1008 Hz. In each trial, two pure tones were presented as a stimulus set with a duration of 100 ms, including 5 ms rise and fall times, in one of four combinations (1000 Hz and 1000 Hz; 1000 Hz and 1008 Hz; 1008 Hz and 1000 Hz; and 1008 Hz and 1008 Hz) (Fig. 1b). The intensity of the stimuli was 85 dB. The stimulus onset asynchrony (SOA) was 500 ms. The order of presentation of the combinations was randomly determined and counterbalanced across trials and the number of trials for each combination was controlled to be equal. Throughout the task, participants were asked to fix their eyes on a solid green disc with a 0.8 deg radius in the center of the monitor. After each trial, a 2 s interval was inserted, consisting of 1 s of white noise as sound interference and 1 s of silence (Fig. 1c). During these intervals, the participants reported whether the two pure tones presented in a trial were different by pressing one of two buttons on a keyboard. A brief break period was provided after each run of 60 trials. The participants performed 300 trials on each day, except on the first day when 600 trials were performed (300 trials before and 300 after the training). Participants whose test accuracy rate exceeded 65% in the pre-training test were excluded and did not participate in any subsequent training or testing.

Participants

Sixteen of the 26 participants initially screened with the BAD test (an accuracy rate range of 50–63%) participated in the present study, including 9 men and 7 women. All participants were right-handed, monolingual speakers of Japanese (age, 22–38 years) with normal hearing. Participants were randomly distributed into the neurofeedback group (4 men, 4 women) and control group (5 men, 3 women). The participants gave written informed consent. The study protocol was approved by the local ethics research committee at Osaka University, Japan and all research was performed in accordance with the ethical standards described in the Declaration of Helsinki.

Training procedure

In the learning stage, an auditory stimulus sequence of 1000 Hz and 1008 Hz tones as the standard and deviant stimuli, respectively, was presented according to the oddball paradigm. The intensity and stimulus onset asynchrony (SOA) were the same as the BAD test. The total number of trials was 300 (1000 Hz, 240 trials; 1008 Hz, 60 trials) in each session. The stimuli were presented in a random order. ERPs were recorded with for 300 ms from stimulus onset at Fz (the International 10–20-system for EEG electrode placement). The MMN was calculated from the previous 20 trials (4 deviant stimuli and 16 standard stimuli). First, ERPs for the standard and deviant stimuli were averaged and then the MMN was obtained by subtracting the average standard ERP from the average deviant ERP. The MMN amplitudes were measured using the frontal (Fz) deviant-minus-standard differences as the peak value with a 100–250 ms interval. Data from the first 20 trials in the learning stage were used to compute the MMN while the size of the solid green disc was fixed. After the first 20 trials, the MMN was updated every 500 ms in parallel with the auditory stimuli. A single session consisted of a sequence of 300 sounds (0.5 s × 300 = 150 s) and 12 sessions were performed on each training day. Each participant underwent training for 5 days (completed in no more than 10 days), with at least 24 hours between training. The size of the disc (0.4–4.97 deg radius) was updated in response to the MMN amplitude every 500 ms.