Introduction

The exponential growth in the time we spend playing video games has motivated interest in exploring the potential use that these may have as screening tools for detecting cognitive and academic difficulties. In addition, the potential of video games as diagnostic tools for cognitive deficits has been poorly studied. Following the idea that a good performance in a video game requires a specific cognitive ability, it is expected that people who show poor performance in the video game may have a deficit in that skill (Tartaglione et al., 2014).

While cognitive tests traditionally are administered inside clinical contexts and required a trained professional to score them, video games can be administered automatically by staff with little training. Other advantages include the precise control of time for the stimulus presentation and the player response, the fact that they can be applied to a group of subjects simultaneously, and the capacity of recording a big number of variables. Also, cognitive tests are applied individually, whereas video games can be applied to a group of subjects. Finally, cognitive tests can take a long time to administer and even more time to score accurately in contrast to video games which can provide an assessment quickly and an accurate score immediately afterwards.

Lumosity and Evo constitute examples of research on video game platforms as cognitive tests (Hardy and Scanlon, 2009; Sternberg et al., 2013; Anguera et al., 2016). Furthermore, there are very few video games in Spanish supported by well-designed psychometric and standardization studies. One example is Towi, a neuropsychological screening in which a set of tests is presented as a cognitive battery under a fun narrative (Rosetti et al., 2017). This effort, however, is still lacking normative data.

The current work describes an effort to fill this void. The video game Chefmania was recently designed as a screening test to determine attention, memory, and impairment in executive functions, such as planning and inhibition, as well as some academic concepts such as number, space, time, and instructions follow-up in school-age children.

The present study aimed to obtain normative data and validity of Chefmania in a Mexican sample of 6–12 years old children.

Methods

The study comprised three stages. In the first stage, normative data were obtained, in the second stage (concurrent validity) Chefmania scores were compared with those obtained in standardized tests and in the third stage (discriminant validity) the Chefmania scores of school children were compared with those of children diagnosed with attention deficit hyperactivity disorder (ADHD), a psychiatric disorder which encompasses inattention and impairment in executive functions (Dajani et al., 2016; Carter-Leno et al., 2018). The study was approved by the internal review board of participant institutions, informed assent and consent were obtained from the subjects and their parents.

Sample characteristics

A sample of 6–12 years old children was recruited from public and private primary schools of two cities in Mexico. Those children with a history of neurological or psychiatric disorders, severe visual impairment, academic failure, or who were over-aged for their grade level were excluded.

Instruments

Chefmania

Chefmanía is a video game located in a restaurant setting. It consists of eight tasks aimed at exploring both cognitive (attention, memory, and executive functions) and basic academic skills (numeracy, space, time, proportionality, tracking instructions). Running the program is not complicated, only basic hardware (keyboard, mouse, screen) and basic computer skills are needed. The player can take the role of chef or waiter depending on the tasks, which are described as follows:

  • Table allocation: The waiter needs to direct the guests to the requested table, clicking on the table as appropriate (e.g. second table on the right) (Fig. 1a).

  • Orders: The waiter takes three orders selecting and dragging the food from several images, the second order will be set based on the first order, increasing or decreasing the number of dishes. The third order follows the same rules.

  • Cupcakes: A third customer orders cupcakes with toppings, the waiter will choose by clicking only on those that comply with the requested feature (Fig. 1b).

  • Committing an order to memory: The next order will be interrupted by another customer so the waiter will have to remember the order and perform the command after serving the customer with the urgent order.

  • Getting the bill: To deliver the bill the waiter will make a two-digit sum corresponding to the cost of the dishes and a multiplication depending on the number of guests. Later the customer will give a discount card (10%/20%/25%/50%) and the waiter should estimate the final amount.

  • Choosing ingredients: The chef has to identify whether the drawers contain food or not and click on the “yes” or “no” button as appropriate. The drawers show the ingredients and distractors in superimposed figures (Fig. 1c).

  • Chop: The chef player must quickly prepare a fruit salad, chopping everything except red fruits in the processor. There is a counter for the ingredients to be chopped as well as a time bar showing the remaining time (Fig. 1d).

  • Cooking: The chef should prepare three dishes considering that all must be finished at the same time and should not be under or overcooked.

Fig. 1: Examples of Chefmania tasks.
figure 1

(Panel a) Table allocation, examining orientation. The waiter and the customers are depicted. The player can observe tables on both sides of the screen, that are positioned from the closest as the first table, second table and so on. Following the instructions to seat the diners at a given table, the player will have to visualize himself from the waiter’s position by selecting the indicated. (Panel b) Cupcakes, examining selective attention. A number of cupcakes with different toppings can be observed, a cupcake is shown to the player, who has to choose all the cupcakes with the same topping in a certain time shown at the bottom of the screen. (Panel c) Choosing ingredients, examining visual recognition. Black silhouettes of different objects such as tools, kitchen utensils of food will appear. The player will analyze the information to distinguish each figure separately. On the sides they will have two buttons with the answers ✓ or ✗. The player will select ✓ in case of recognizing the figure of a food. (Panel d) Chop, examining attention and inhibition. An image of a processor is displayed with the “chop” button in the center. Different foods will appear inside the processor, respondents are required to push “Chop” when the food is not red. To accomplish the task the player will have to pay attention to each food shown and avoid incorrect responses. The player will observe the number of foods left to chop on the left and on the other the remaining time on the right.

Full size image

The total test time is 20 min, but children can complete it earlier. Chefmania was designed based on standardized cognitive tests, the cognitive correlate, foundation, and scoring system of each task are shown in Table 1.

Table 1 Chefmania video game components.
Full size table

Chefmania also provides the following global scores: Time (total execution time), Global Hits (GH, total hits), Global Misses (GM, total misses), Efficacy (E, GH−GM) and the Impulsivity control index (ICI, obtained by the formula ICI = (E/(GH + GM)×100).

Child neuropsychological assessment (Evaluación Neuropsicológica Infantil, ENI)

The ENI is a comprehensive neuropsychological battery developed specifically for Spanish-speaking children which was created based on paper and pencil tests. It provides an assessment of several cognitive domains including constructional abilities, memory, perceptual abilities, oral language, metalinguistic awareness, spatial skills, attention, concept formation, and executive functions. Normative data were collected from 800 children in Mexico and Colombia, aged 5–16 years (Rosselli et al., 2010). In the present study the superimposed figures, mental calculation, and right–left orientation understanding subtests were used.

NEUROPSI attention and memory (NEUROPSI)

This neuropsychological battery was developed based on paper and pencil tests to assess a wide spectrum of cognitive functions including orientation, attention, memory, and executive functions. Normative data were collected from Spanish-speaking individuals, aged 6–85 years (Ostrosky-Solís et al., 2007). In the present study, the following subtests were used: wordlist, digit forward span, digit backward span, spatial forward span, visual search, digit detection, Stroop, word list (free recall, cued recall, recognition).

Battery for the assessment of executive functions and frontal lobes (BANFE)

This neuropsychological battery was developed based on paper and pencil tests to assess cognitive functions related to the prefrontal cortex, including planning, working memory, inhibition. Normative data were collected from Mexican Spanish-speaking individuals, aged 6–55 years (Flores and Ostrosky-Solís, 2008). In the present study, only the mazes subtest was used.

Procedure

The study design is shown in Fig. 2. Subjects for the standardization were asked to play Chefmania. A subsample was then evaluated by trained psychologists for the obtention of concurrent validity data. For discriminant validity, a sample of children diagnosed with ADHD was recruited at a public mental health facility in Mexico City and asked to play chefmania.

Fig. 2: Study design.
figure 2

Each panel shows the procedures and the number of participants in each stage of the study.

Full size image

Statistical analysis

Statistical analysis was carried out using the Statistical Package for Social Science (SPSS 21 for Windows). Only the data for those participants who completed all tasks were included. Performance measures were transformed into z scores for each age group and task. Subsequently, these were transformed into a scalar score according to the Wechsler Intelligence Scale standardized version for Mexico (Wechsler, 2003) with mean = 10 and standard deviation = 3, that is, a scalar score of 10 suggests an average performance within a specific age group, while scalar scores of 7 and 13 reflect a performance one standard deviation below and above the mean, respectively. To examine the effect of age on the performance we divided the sample into three groups (6–7, 8–9, and 10–12 years old) and performed an ANOVA. Pearson correlation tests were used for concurrent validity. Discriminant validity was obtained comparing the scores of children with ADHD vs. a paired sample from the standardization study using Student’s t-test, hypothesizing that the former group would show lower scores. Significance was set at p < 0.05 with 95% confidence intervals.

Results

A sample of 334 children was recruited, 62 did not complete the game and 6 were excluded because they were over-aged for their grade level. The final analysis included data from 266 children distributed as follows: 81 (30.5%) in the 6–7 years old group (first and second grades), 45 (16.9%) in the 8–9 years old group (third and fourth grades), and 140 (52.6%) in the 10–12 years old group (fifth and sixth grades).

The mean execution time of the total sample was 13.78 ± 4.66 min. An age effect was observed for the Chefmania subtests and Impulsivity control index, where older children showed better performance (Table 2). No sex differences were found.

Table 2 Age differences in the obtained scores for Chefmania subtests.
Full size table

The evaluation of concurrent validity showed significant and moderate correlations between Chefmania scores and those obtained in standardized tests. The highest correlations were obtained between Getting the bill and ENI mental calculation, Chop misses and NEUROPSI Stroop, and between Chop hits and BANFE mazes. All correlations are shown in Table 3. The evaluation of discriminant validity showed that subjects with ADHD obtained lower scores (Table 4).

Table 3 Correlations between Chefmania and standardized tests scores.
Full size table
Table 4 Comparison of Chefmania scores between control and ADHD groups.
Full size table

Discussion

The present study aimed to obtain normative data and to determine the validity of Chefmania in a Mexican sample of 6–12 years old. children. The results allowed to obtain a profile in orientation, spatial abilities, attention, short-term memory, arithmetical calculation, visual analysis, inhibitory control and planning as well as basic academic skills (numeracy, space, time, proportionality, and following instructions) based on the subject’s age.

The scores were transformed into scalar scores according to the WISC-IV, in order to obtain a cognitive profile coherent with the age and school grade. This scoring system allows the determination of the correspondence between a child’s academic performance and his or her age.

In this sense, Chefmania could be considered as a cognitive screening test for children attending elementary school. The observed age differences in the scores resemble previous reports of a better cognitive function in older children (Zelazo et al., 2003; Chelune and Baer, 1986; Luciana and Nelson, 1998; Welsh et al., 1991). In addition, the 95% CI of the Impulsivity control index for each age group allowed to classify the academic performance in one of the following categories: early elementary years with poor performance, early elementary, mid-elementary, late elementary, and late elementary with high performance. This, in turn, can provide information to determine whether a child is performing according to his or her age, which makes Chefmania a good screening tool for assessing the academic level in countries where the students may not reach the expected academic level at the end of their basic education (Gurria, 2016). Finally, the lack of sex differences is in line with previous reports which examined normative patterns of development according to sex (Martin and Ruble, 2010).

Regarding concurrent validity, Chefmania scores showed moderate correlations with NEUROPSI spatial forward span, visual search, digit backward span, Stroop, ENI mental calculation, and BANFE mazes. As reported in other studies, when comparing screening video games to standardized pencil and paper tests, only moderate correlations can be expected given the differences in the manner of application, the motivational issues, and stimuli visualization (Rosetti et al., 2017).

Chefmania showed a good discriminant validity, since children with ADHD, showed lower global scores in ordinality, orders, choosing ingredients, chop and in the impulsivity control index than the school group. These findings are consistent with the difficulties in attention and impulsivity, which are characteristics of ADHD (Ter-Stepanian et al., 2017).

Limitations and perspectives

Despite the limitations such as a different number of children in the three age groups and the lack of rural samples, which could affect the normative data, Chefmania presents an efficient form of videogame-based assessment. Among the advantages of Chefmania is the combined evaluation of academic achievements and cognitive skills through a digital tool, which makes it possible to assess a large number of children in a short time frame. In the future, the program could implement the use of artificial intelligence or neural networks that adapt the game to different types of users, so the use of Chefmania could be expanded to other age ranges or to subjects with particular cognitive impairments.