Introduction

Recent reviews of cognitive remediation (CR) for people with schizophrenia have found moderate effect sizes on neurocognitive outcomes, and there has been growing evidence for generalized benefits to social role functioning when paired with rehabilitation interventions. McGurk et al.1 performed a meta-analysis of 26 RCTs and found effect sizes of 0.41 for cognitive outcomes and 0.36 for psychosocial functioning; Wykes et al.2 performed a meta-analysis of 40 studies and reported effect sizes of 0.45 for cognitive improvement and 0.42 for social functioning; and Katsumi et al.3 recently did an updated literature review and reported similar findings.

Our research group (Wexler & Bell4,5) and others (e.g. McGurk et al.6; Nuechterlein et al.7) have focused on CR as an augmentation to vocational rehabilitation (VR), reporting favorable employment outcomes. A meta-analysis of 9 studies from 5 different countries8 found that participants receiving CR showed 20% higher employment rate, worked more days and earned US $959 more than those who did not receive CR. They concluded that CR plus vocational services enhanced productivity and quality of life. Recently, Lystad et al.9 reported on a complex study in Norway that compared CR with Cognitive Behavior Therapy (CBT) for participants with schizophrenia spectrum disorders in VR programs over two years. They found that both groups improved on several neurocognitive domains, though results favored CR. Vocational outcomes did not differ by condition, although cognitive improvement in CR uniquely predicted hours worked.

As efforts to integrate CR into clinical and rehabilitation activities continue, it is important to learn as much as possible about its effectiveness across various rehabilitation programs in everyday practice. Wykes and Spaulding10 in a seminal paper argue that we need to know about the clinical significance of CR, its impact on functioning and what features of the intervention are most important and for whom.

The present report describes a study designed with some of these aims in mind. Data collection was begun in 2009 and completed in 2015. To our knowledge, it is the first study conducted on individuals with schizophrenia-spectrum illness designed to compare CR in the context of a range of vocational rehabilitation services with an active sham control comprised of portable “brain-games.” The primary aim was to compare a scientifically validated focused cognitive training procedure (Brain Fitness and Insight by Posit Science) performed in a cognitive training laboratory (Cog Lab) to non-specific “brain games” (Brain Age-2) that would serve as a rigorous sham control. We believed that Brain Age-2 was an excellent choice as a sham control because it had face valid exercises of executive function and provided user-friendly feedback in the form of a “Brain Age” score, that indicated progress toward “younger” (better) cognitive performance. However, we believed it was a sham because unlike the targeted training of Posit Science’s auditory (Brain Fitness) and visual (Insight) exercises, the Brain Age-2 had no training that addressed primary sensory processing and had very limited within-exercise adaptations to performance. No CR clinical trial had used such a face valid comparator at the time this study was proposed. Recently, a review of 18 commercial CR programs concluded that Posit Science had by far the strongest scientific evidence of effectiveness, that Brain Age-2 was in the second tier of evidence, while almost all others had either poor evidence or none at all11. Thus, it appears that Brain-Age 2 has some scientific support for cognitive benefits, at least in healthy adults12,13.

In addition, Brain Age-2 was on a portable, hand-held device (Nintendo DS), which allowed us to ask a second question regarding the impact of portability on adherence. While most previous studies including our own supported the benefits of targeted CR performed on site14, Brain Age-2 had the advantage that participants could use it outside of laboratory hours, and thus might train much more. Moreover, because portable training allows users to train at their convenience, it was less likely to conflict with job searching or establishing a new work schedule. At the time of this study, there were no tablet ready versions of Posit Science’s targeted CR exercises. Had there been, we might have focused our study on comparing portable training with Cog Lab training. Although we couldn’t do that comparison, it was still an important question to test whether portability would lead to more training adherence. For these reasons, we hypothesized that (1) scientifically supported CR training would be more effective in improving neurocognitive function than a face valid, sham control, and that these neurocognitive benefits would generalize to superior vocational outcomes and community functioning; and (2) portability of computer training would lead to more overall training activity.

Results

Sample Characteristics

Baseline demographic, illness and employment characteristics (Table 1) showed no significant differences between conditions, except for a greater number of participants with schizophrenia and schizoaffective disorder (when combined) for the Posit Science group (72% vs 50%, p < 0.05). Most of the participants were receiving SSDI, SSI or VA service-connected benefits (89.6%) and hadn’t worked on average in almost 11 years. Average earnings in the last 30 days was about $25.00. At baseline 30 (39%) had not begun their vocational rehabilitation service at intake but were in process; 24 (31%) had just begun working in Incentive Therapy; 15 (19.5%) were in SE though none were currently working; 6 (8%) had begun working in Compensated Work Therapy at the VA, and 2 (2.6%) were working casual jobs a few hours a week. All participants were eventually enrolled in some form of VR with slightly more than half in SE (54.5%). There was no difference between conditions in type of VR services used (Table 1).

Table 1 Baseline Characteristics.

The 13 participants from the CMHC sample did not differ on PANSS scores at baseline, but they did differ significantly on QoL scores (VA mean = 67.9 (17.07); CMHC mean = 54.92 (11.45); t (75) = 2.62, p < 0.01). However, there was no significant difference between the two conditions regarding what site they came from (Chi Sq = 0.74, p = ns). The 13 participants from CMHC represents only 16.8% of the whole sample, and when site was added as a covariate in subsequent analyses, it had no effect on findings.

Retention and Adherence

Retention was somewhat better for VR + CR at 6- (87% vs 74%) and 12- month follow-up (74% vs 71%) but not statistically different (Consort Flow diagram in Supplementary Material). Adherence to treatment did not differ by condition (F(1,74) = 1.19, p = 0.29). A few participants spent a lot of time on Brain Age-2, influencing a higher mean number of training hours in the VR + CG condition but with a very large standard deviation and range (VR + CG M = 63.5, SD = 65.12, Range 0–193.00). When categorized into 3 levels of training (less than 14 hours; 14 to 60 hours; more than 60 hours), the conditions also did not differ significantly (Χ2(2) = 3.87, p = 0.15). While both conditions showed generally good attendance at the weekly goal-setting group, VR + CR participation (mean = 19.27 (7.95) groups out of 26 weeks) was significantly greater than that of VR + CG (mean = 14.78 (9.90), t(72) = 2.15, p < 0.05).

Employment Outcome

Conditions did not differ by competitive employment rate at 12-month follow-up (VR + CG = 9/38, VR + CR = 9/36; Χ2(1) = 0.02, p = 0.89). However, non-parametric mixed-effects model of employment attainment at 2, 4, 6 and 12 months revealed significant condition (Χ2(1). = 4.73, p < 0.03) and time (Χ2(3) = 13.94, p < 0.01) effects favoring VR + CG, but no time-by-condition interaction (Χ2(1) = 4.25, p = 0.24). This analysis indicated that VR + CG led to significantly earlier job attainment during the 6-month active training period, but those receiving VR + CR caught up over the 6-month follow-up period (Figure 1).

Hours engaged in training was not significantly associated with employment attainment (b = 0.003, se = 0.005, p = 0.591, OR = 1.003). Non-parametric analysis revealed significant differences in employment hours and earnings at 6 months favoring VR + CG (Table 2), but no difference at 12 months. Mixed-models analyses of employment hours using all time points (2 months, 4 months, 6 months, 12 months), showed a condition effect favoring VR + CG for employment hours (df = 1, ATS = 6.21, p < 0.013), but no time or time-by-condition interaction. There was a significant decline in overall productive hours across groups by the end of 12 months (time effect: df = 2.62, ATS = 5.56, p < 0.001), but no time-by-condition interaction (Supplements S2 & S3.) The decline in hours over time was mostly due to the 6-month transitional work program, which ended around the 6-month assessment. Since most recipients of this service didn’t get employed, their productive hours declined sharply by 12 months. Of note, 41.2% of participants receiving Supported Employment (SE) attained employment over 12 months compared with 16.7% of those in transitional employment a significant difference (Χ2(1) = 3.94, p < 0.05).

Figure 1
figure 1

Employment rates over time by condition.

Table 2 Employment Outcomes.

Neurocognitive Outcomes (Table 3)

Mixed-models indicated no main effect for condition on baseline neurocognitive measures (Tables 3 and 4). A significant time-by-condition interaction in the Attention domain (p = 0.034) indicated significant increases in Attention over 12 months (p = 0.002) in VR + CR, and no appreciable change from baseline in VR + CG (p > 0.50; Supplement S4). No other between-condition effects were detected on neurocognitive outcomes.

Table 3 Descriptive Statisticsa for Quality of Life (QLS scale), Neurocognitive, and Employment Outcomes by Treatment Condition.
Table 4 Mixed Model Coefficients and Associated Standard Errors Testing Effects of VR + CR Versus VR + CG on Change in Neurocognitive Outcomes

Additional analyses support the relationship between cognitive training activity and cognitive outcomes across conditions and time. Hours of cognitive training by time predicted change in Working Memory (p = 0.010) and Processing Speed (trend level; p = 0.059) over 12-month. There were also significant time effects, unrelated to hours of training, for Attention (p = 0.002) and Visual Learning (p = 0.019), and for Global Cognitive Index (p = 0.006). Follow-up analysis of simple slopes indicated that improvements in Working Memory were obtained for individuals who trained 40 hours or more across the treatment period. Three-way interactions between condition, training hours, and time were non-significant.

Quality of Life (QLS) Outcomes (Table 3)

Mixed-models indicated no difference in baseline between conditions (Tables 3 and 5). Significant time-by-condition interactions for Total QLS (p < 0.02) and Instrumental Function (p < 0.01; Supplement S5) indicated that these QLS measures remained stable over time in VR + CR (p < 0.50) but declined significantly for VR + CG (QLS Total p < 0.01 and Instrumental Function p < 0.001). Training hours were associated with Instrumental Function ratings at the trend level (p = 0.053).

Table 5 Mixed Model Coefficients and Associated Standard Errors Testing Effects of VR + CR vs VR + CG on Change in Quality of Life Outcomes

Discussion

To our knowledge this is the first report comparing focused cognitive remediation in a “Cog Lab” to a sham control of cognitive games on a mobile device in the context of vocational services. Employment outcomes by 12 months in this chronically unemployed and disabled psychiatric sample do not indicate that either condition was superior. There may have been some advantage to having a portable device during the 6-month training phase as suggested by more rapid rate of employment and more hours of competitive work in the VR + CG condition. Furthermore, an upshift in employment in VR + CR following completion of cognitive training, reaching levels equivalent to VR + CG by 12 months, supports the interpretation that participants in this condition only fully engaged in job searching after completing the 6-month curriculum of cognitive training. Since hours spent training did not differ by condition and appeared unrelated to attaining employment by 12 months, it remains possible that coming into the “Cog Lab” rather than using a portable training system may have had an opportunity cost in terms of employment.

In a previous study that compared enhanced SE with 12-month cognitive training along with a social information processing group to enhanced SE alone5, we found that although the training condition led to better vocational outcomes at two-year follow-up, those in the “Cog Lab” took longer to obtain a job. The findings of overall superior employment outcome with cognitive training in that study may have been observed due to a longer treatment and follow-up period, which provided more time for the benefits of cognitive training to generalize to vocational function. Moreover, the enhancement of SE included an initial period of paid on-the-job training so that almost everyone began work within 3 months and had the benefit of synergy between cognitive training and work activity. McGurk and colleagues1,6 have similarly reported better vocational outcomes using cognitive training but all participants were engaged in SE. Unsurprisingly, in our study, what made the greatest difference in employment was whether participants were enrolled in SE. Those in SE had superior employment rates to those in transitional work programs, regardless of cognitive training condition.

Regarding neurocognitive outcomes, conditions did not differ on MATRICs global score, but VR + CR was superior to VR + CG in the Attention domain. This suggests that despite little overall improvement by either training method, there may have been some specific advantage for the Posit Science training over Brain Age-2 in this domain. We have long maintained that cognitive training produces the greatest benefits when performed in the context of other activating rehabilitation activities. Although, some participants obtained work in our study, most were looking for work or only working a few hours in transitional employment and were otherwise inactive. Cognitive training may be most effective when there are opportunities to generalize and reinforce its lessons through other rehabilitation activity. Nevertheless, we did detect a relationship between amount of training across condition and improvement on MATRICS Working Memory score, suggesting a dose response relationship regardless of the training method. Such findings are unlikely related to practice-effects because the period between assessments is large (6 months) and the MATRICs assessments were selected because they were ones that showed only small practice- effects at much shorter intervals.

Findings for community function as measured by QLS suggest that participants in the VR + CR experienced functional benefits that extended beyond the active intervention period, while those in VR + CG showed consistent decline from baseline over active and follow-up periods. Everyone began the study when they were beginning some type of vocational service, but after initial efforts it appears that the VR + CG group began to be less engaged in productive roles, despite a few more individuals obtaining employment. It may be that the systematic training that led to improvements in attention encouraged VR + CR participants to stay engaged in productive activity, even though it took them longer to get employed. It is also possible that a regular schedule evolved around “Cog Lab” training that created new opportunities to engage in community activity. Participating in more groups than VR + CR may also have increased their capacity for engagement, especially since goal setting included job searching and other forms of productive activity (such as helping family members). Although we do not know the mechanism for the relationship between VR + CR and QLS findings, it is a signal that cognitive remediation with goal-setting groups may be useful in augmenting rehabilitation efforts.

We hypothesized that a mobile device might lead to more training; however, both conditions showed good engagement in cognitive training with no significant difference in overall training hours. A few participants did use their portable device a great deal more than those who were limited to training during lab hours, but this was uncommon. There was, however, a significant difference in group attendance, suggesting that VR + CR participants may have been more likely to attend group since they were already attending “Cog Lab.” It is also possible that since VR + CG participants obtained employment sooner, going to groups conflicted with work activity. Now that many cognitive training programs including Posit Science’s Brain HQ are available on tablets, the option of portability might provide both the benefits of more focused training and times of training more compatible with individual schedules. It remains unknown, however, whether the assistance of a “Cog Lab” monitor and the added social experience of being with others in groups and in the “Cog Lab” offers unique benefits.

Limitations to these conclusions include the lack of a no-treatment control condition, the powerful disincentives to competitive employment in a sample where 90% are receiving disability payments, and the relatively small size of the sample. Only about half of the participants received SE, and since SE had the greatest impact on vocational outcomes its effects may have overshadowed other influences.

While we observed an advantage of laboratory-based over portable device training on cognitive outcomes, these gains did not appear to translate immediately to work outcome. Indeed, given that “Cog Lab” attendance may have impeded job seeking or attainment, the advantages of cognitive gains may come at the cost to other opportunities. Therefore, for those who identify work as a primary goal, we might recommend starting the job search first and offering cognitive training once the client is stable on the job and a need for cognitive remediation has been identified based on job performance or other client goals. Consistent with the supported employment model, the principles of “place then train” could apply appropriately to cognitive remediation interventions targeting vocational outcome.

Method

Participants

Seventy-seven participants including 64 Veterans at VA Connecticut Healthcare System (VACHS) and 13 non-veterans at Connecticut Mental Health Center (CMHC) with chronic and disabling psychotic illness (61% schizophrenia, 39% other psychosis) in transitional work services (45.5%) or supported employment (54.5) were included in the study. This was originally intended to be for Veterans only, but the study was modified in its third year to include non-Veterans from CMHC to meet recruitment goals. Inclusion criteria included willingness to engage in vocational rehabilitation services of some kind and presence of disabling psychotic illness. Additionally, vocational programs had some admission requirements including demonstrated need for services, referral from clinicians, and an interest in work rehabilitation. All participants were unemployed, though a few were earning money through informal work. Co-morbid substance use disorder was an exclusion with less than 30 days of abstinence. Known neurological disease, uncorrected sensory impairment, intellectual deficiency or premorbid IQ estimate under 70 were additional exclusion criteria.

Measures

Clinical, Quality of Life and Vocational Assessments

Structured Clinical Interview for DSM IV (SCID)15 was used for diagnostic evaluation at baseline. Positive and Negative Syndrome Scale (PANSS)16 was used for symptom assessment at baseline and quality of life was measured using Quality of Life Scale interviews (QLS)17. QLS has five subscales: Intrapsychic, Interpersonal, Instrumental Function and Objects and Activities. Our raters achieve good to excellent inter-rater reliabilities on all subscales on both instruments18,19. QLS was administered at baseline, 2-month, 4-month, 6-month and 12-month follow-ups. Vocational data was recorded at the same observation points and included job attainment, competitive hours worked and money earned, transitional hours worked and money earned, and volunteer work. These data were collected from the vocational program records which were available for Incentive Work Therapy (an in-house work placement that pays half minimum wage and has reduced expectations), Compensated Work Therapy (paid work placements through contracts) and SE. The very few participants who worked independently outside of SE reported their earnings at 6 and 12-month follow-up. Informal income was included in competitive earnings. Types of jobs obtained and reasons for termination were also recorded for qualitative purposes, but are not included in this report.

Neurocognitive Assessments

Premorbid intellectual functioning was assessed at baseline using the Wechsler Test of Adult Reading20. The MATRICS battery21 was used to assess changes in cognitive performance over time from baseline to 6-month and 12-month follow-up. Performance was evaluated based on age- and gender-corrected standard scores in domains of Processing Speed, Attention/Vigilance, Working Memory, Verbal Learning, Visual Learning, Reasoning and Problem Solving, and a global neurocognitive composite of these scores.

Procedures

All procedures were approved by the Institutional Review Boards at VA VACHS and Yale and were carried out in accordance with the relevant guidelines and regulations. Recruitment was done by clinician referral and by publicly displayed flyers. Following informed consent, participants had baseline assessments and if found eligible were randomized in blocks of 6 (3 to Cognitive Games (VR + CG) and 3 to laboratory-based Cognitive Remediation (VR + CR) in a 1:1 ratio by a statistician unrelated to the study. Blocks of 3 were used to ensure nearly equal numbers in both conditions while keeping assessors blind to condition. VR services were unrelated to randomization; individuals had free access to all available VR services. Trained master’s level or Ph.D. research staff blind to condition performed assessments. Training began in the “Cog Lab” located at the Learning Based Recovery Center (LBRC) at VACHS, but VR + CG participants were given the option of subsequently training at home following 3 sessions of instruction about how to use the portable devices. Goal-setting groups occurred weekly at the LBRC.

Vocational Rehabilitation (VR) Programs

VACHS and CMHC had a range of work services that included transitional programs and Individual Placement and Support22 SE programs. IPS programs at both institutions were independently evaluated for fidelity and were in the acceptable range. The transitional programs included paid placement in work therapy at VACHS or at a psychosocial clubhouse with staff support and guidance for job seeking.

Interventions

VR + CG (Nintendo Brain Age-2)

Participants received a hand-held Nintendo DS with “Brain Age-2” training games. They could train as much as they wished, but staff recommended training 1 hour each day, five days per week. They were also asked to do the “Brain-Age” assessment with each day of exercise. They could use the device at our “Cog Lab” or take it home with a $30 deposit. Daily training activity was logged by the Nintendo device and transferred to the research record at weekly goal-setting meetings.

VR + CR (Posit Science Brain Fitness and Insight)

Participants performed exercises that targeted auditory and visual discrimination and memory. This training occurred in the “Cog Lab”, available 7 hours a day, 5 days a week, following training schedules created around work and other appointments. A laboratory monitor provided a friendly and encouraging environment, recorded training activity, and gave technical assistance as needed. After a short time, most participants could work through the exercises with minimal assistance. If a participant became sleepy or frustrated during a task, the monitor suggested taking a break before continuing. The monitor, however, did not coach participants in strategies to improve performance. Participants were encouraged to attend 5 laboratory sessions per week, with exercise selection and duration of sessions determined by software. Training progress was logged into a laboratory manual and reviewed with participants at each session. The “Cog Lab” did not include any specific protocol for encouraging socializing or for “buddying up” among the participants; the environment was friendly but business-like, and each person trained on their own.

Goal-setting Group

All participants attended the same weekly group regardless of condition assignment. There was no discussion of cognitive training condition in these groups. These groups focused on setting VR goals, reviewing achievements, problem-solving barriers to achievement, and providing encouragement and support while celebrating successes. These goal setting groups were led by a vocational rehabilitation specialist. In addition, ball-draws for rewards for cognitive training participation (see Compensation) occurred in the group and was a source of light-hearted fun.

Compensation

Participants were compensated for participating in each assessment and could earn $235 in total for completing all assessments. There was no fixed reimbursement for training activity. To avoid giving participants the impression that CR was a paid job, they were not paid a specific hourly rate for training. Rather, training was incentivized using contingency management23, a procedure commonly used to encourage drug abstinence. This procedure involved drawing for monetary prizes like a lottery based on attendance. Draws were earned following blocks of 5 training sessions, with the number of draws increasing progressively as training advanced. Each draw had a 2-out-of-5 chance of a monetary award of $1 (1/5 odds) to $25 (1/50 odds).

Statistical Analyses

Employment outcomes were highly skewed and were analyzed using the nonparametric approach for repeated measures24. Data were ranked, then fitted using a mixed-effects model with an unstructured variance-covariance matrix and p-values adjusted for ANOVA-type statistics (ATS). Predictors included condition (VR + CG vs. VR + CR), time, and group-by-time. Employment rates were analyzed across time using a generalized linear model with a logit link function and random subject effects and included the same predictors as above. A logistic regression analysis tested differences between conditions in employment status at study end. Number of training hours was tested as a covariate.

Cognitive and QLS outcomes were analyzed using mixed-effects models that account for the correlation of repeated measures within participants while allowing for inclusion of cases with incomplete or missing data. Models included a random intercept and had condition, time, and group-by-time as predictors. MATRICS measures were collected at 3 time points (baseline, 6, and 12 months). QLS and employment, volunteer and transitional work hours and earnings were collected at 5 time points (baseline, 2, 4, 6, and 12 months). All tests-were two tailed with alpha set at 0.05.