Article | Open

EEG-Based Cognitive Control Behaviour Assessment: an Ecological study with Professional Air Traffic Controllers

  • Scientific Reports 7, Article number: 547 (2017)
  • doi:10.1038/s41598-017-00633-7
  • Download Citation
Received:
Accepted:
Published online:

Abstract

Several models defining different types of cognitive human behaviour are available. For this work, we have selected the Skill, Rule and Knowledge (SRK) model proposed by Rasmussen in 1983. This model is currently broadly used in safety critical domains, such as the aviation. Nowadays, there are no tools able to assess at which level of cognitive control the operator is dealing with the considered task, that is if he/she is performing the task as an automated routine (skill level), as procedures-based activity (rule level), or as a problem-solving process (knowledge level). Several studies tried to model the SRK behaviours from a Human Factor perspective. Despite such studies, there are no evidences in which such behaviours have been evaluated from a neurophysiological point of view, for example, by considering brain activity variations across the different SRK levels. Therefore, the proposed study aimed to investigate the use of neurophysiological signals to assess the cognitive control behaviours accordingly to the SRK taxonomy. The results of the study, performed on 37 professional Air Traffic Controllers, demonstrated that specific brain features could characterize and discriminate the different SRK levels, therefore enabling an objective assessment of the degree of cognitive control behaviours in realistic settings.

Introduction

The efficiency of humans in coping with complex situations is largely due to the availability of a large repertoire of different mental representations of the environment from which rules to control behaviour can be generated ad-hoc1,2,3. Nowadays, several models defining the different types of cognitive human behaviour are available4,5,6. For this study, we have selected the Skill, Rule and Knowledge (SRK) model proposed by Rasmussen in 19837. Accordingly with basic different ways of representing the constraints in the cognitive human behaviour, three typical levels emerge: skill-, rule-, and knowledge-based level7, 8.

The skill-based behaviour represents sensory-motor performance during acts or activities which, following a statement of an intention, take place without conscious control as smooth, automated, and highly integrated patterns of behaviour. In most skilled sensory-motor tasks, the body acts as a multivariable continuous control system, synchronizing movements depending on the surrounding environment. Performance is based on feedforward control and depends on a very flexible and efficient dynamic internal world model. Characteristically, skilled performance rolls along without conscious attention or control. The total performance is smooth and integrated, and sense input is not selected or observed: the senses are only directed towards the aspects of the environment subconsciously needed to update and orient the internal map. In general, human activities can be considered as a sequence of such skilled acts or activities composed for the actual occasion. The flexibility of skilled performance is due to the ability to compose, from a large repertoire of automated subroutines, the sets suited for specific purposes.

At the level of rule-based behaviour, the composition of such a sequence of subroutines in a familiar working situation is typically controlled by a stored rule or procedure, which may have been empirically derived during previous occasions, or it may be prepared in correspondence of conscious problem-solving and planning. In this case, the performance is goal-oriented but structured by “feedforward control” through stored rules. Very often, the goal is not even explicitly formulated, but it is implicitly found from previous successful experiences. Feedback correction during performance will require functional understanding and analysis of the current response of the environment, which may be considered as an independent concurrent activity at the higher level (knowledge-based).

The boundary between skill-based and rule-based performance is not very distinct, and much depends on the level of training and the attention of the person. In general, the skill-based performance rolls along without the person’s conscious attention, and he/she will be unable to describe how he/she controls and on what information he/she bases the performance. The higher level, rule-based coordination, is generally based on explicit know-how, and the adopted rules can be reported by the person. During unfamiliar situations, facing with an environment for which no know-how or rules for control are available from previous experiences, the control of the performance must move to a higher conceptual level, in which the performance is goal-controlled and knowledge-based. In this situation, the goal is explicitly formulated, based on an analysis of the environment and the overall aims of the person. If we consider that in real life people have a varying degree of expertise when performing a task depending on variations and disturbances, the correspondence with the three cognitive control behaviours in the present context is clear. One aspect of the categorization of human performance in skill/rule/knowledge-based behaviour is the role of the information derived from the environment, which is basically different in the various categories. This is the case of unfamiliar situations, when same information or events could be perceived in many different ways depending on the operator’s expertise level8. As described by Reason9, each level has its own pros and cons, and it is related to specific types of performance and error: for instance, as the skill level is characterised by a low level of control, the corresponding behaviour is fast and effortless, but it is also related to slips and lapses errors. While performing at the rule-based level, it is possible to quickly and effectively face familiar and anticipate unfamiliar events, but rule-based mistakes can happen, such as the misapplication of right rules or the application of wrong rules. Finally, the knowledge-based behaviour leads to the resolution of new problems, to the generation of new rules and, in general, to innovation and creativity; on the other hand, this behaviour is slow, cognitively demanding and very prone to error. The reason of the SRK model choice was that the proposed study has been carried out for safety critical systems, with particular interest in the Air Traffic Management (ATM) domain, where the Human Factor (HF) bearing has been deeply investigated also from a neurophysiological point of view10,11,12,13,14,15,16,17,18,19,20. The SRK framework is broadly known and applied in many operational environments, as aviation and healthcare, for the design21 and generation of the human error taxonomy22. A second reason for our choice was related to the origin of this model, as it has not been developed from cognitive psychology studies, but from the direct observation of operators performing their working tasks in complex systems. The use of a neurophysiological-based model, rather than a psychological one, poses some challenges. In fact, in the scientific literature there are no direct links with the well-known cognitive psychology concepts, and this study can contribute to better highlight the potentiality of the SRK model and the proposed methodology. As mentioned above, the SRK framework is still lacking of a quantitative, reliable and validated way to be quantified. Assessing the level of the cognitive control behaviour of professional personnel in operational environments would enable a deeper comprehension of the cognitive processes actually activated in real work activities.

Toward an integration between srk and neurophysiological variables

Several interesting issues about the neuroanatomical and functional models may be related to the level of attentional control of SRK framework7 for a possible integration between SRK and neurophysiological variables. One of the first models of attentional control has been proposed by Posner and Petersen23 and it consists of two separate, but closely interrelated, systems: an anterior attentional system (linked to environmental monitoring and detection of target stimuli), and a posterior attentional system (linked to the orientation of attention). More recent versions of this model describe three systems24,25,26. In particular, the Alert/vigilance system (connected to frontal and parietal regions), the Orienting system (consisting in posterior parietal, frontal cortex, temporo-parietal junction (TPJ), thalamic nuclei, and superior colliculus), and the Executive system (consisting in prefrontal medial cortex, anterior cingulate cortex, and supplementary motor area). Furthermore, recent studies27 have proposed that cortical control of attention is splitted into two functionally-divided but interacting systems:

  • Dorsal frontoparietal network (top-down, endogenous attention) is guided by cognition, involved in control of goal-driven attention, and preparation and application of relevant stimuli selection.

  • Ventral frontoparietal network (bottom-up, exogenous attention) is guided by perception and it is a stimulus-driven attentional system. It is involved in disengagement and re-orienting of attention towards salient or unexpected stimuli.

The SRK framework can be connected to the attentional model mentioned above towards the investigation of neurophysiological correlates of Rasmussen’s taxonomy. In fact, among the cognitive neuropsychologists, in terms of limited capacity, it has been hypothesized a mechanism with the function of programming and controlling cognitive processes in relation to priorities, goals and external conditions. This mechanism, called Central Executive, provides a higher-level processing in a hierarchical organization. Since the central executive has a limited capacity, many routine operations should be delegated to mechanisms that operate automatically, regardless of the voluntary attentional costs in terms of effort. An automatic process28 consists of the activation of a learned sequence of elements without the conscious control. A controlled process instead requires continuous effort and monitoring by the subject. It works in serial mode, drawing on the stock of the short-term memory. The automation is facilitated by practice, which makes the process faster, parallel, with less mental load demand. However, it should be remembered that attention does not always coincide with a controlled process: the automatic process also focuses attention on certain stimuli, selecting among other not-relevant stimuli for the task, and keeps focusing on them for the necessary time. Neumann29 pointed out that the automatic process is not completely uncontrolled, but rather control is below the threshold of consciousness. Therefore, it seems appropriate to distinguish between non-intentional attentional processes, involving minimal effort and self-perception of the subject, and intentional attention, conscious and characterized by a limited capacity. Several empirical studies30,31,32,33 demonstrated how information not consciously receiving attention can be processed in relation to their meaning, and in absence of intentional control. Since attentional mechanisms can control the access to awareness34, 35, different models tend to identify the function of attentional control with the conscience, such as central operative system of activation, and self-monitoring. According to Allport30 and Schmidt36, intentionality, attention, control and awareness are the constitutive dimensions of consciousness. To sum up, the attentional level is very important for the neurophysiological characterization and discrimination of the SRK behaviours in the ATM domain4, 37, since skill-based behaviour generally rolls along without the person’s conscious attention, whereas higher levels, rule- and knowledge-based behaviours, are generally based on explicit attentional control. In other words, higher activations of the Electroencephalographic (EEG) features correlated to the attentional level will represent higher attention demand, therefore relative to rule- and knowledge-based behaviours.

Hence, the objective of this study was to assess whether it was possible to differentiate the three degrees of cognitive controls proposed by the SRK model7 by the analysis of the Air Traffic Controllers’ brain activity during the execution of a high realistic ATM scenario. In particular, we aimed to characterize the three cognitive control behaviours in terms of “pure” cognitive engagement by considering only brain features linked to cognitive processes, such as information processing, decision making, working memory, and attention. In fact, we have not considered motor brain features (e.g. sensory-premotor cortex) in order to do not differentiate the SRK behaviours on the basis of different amount of movements (hands or feet).

Material and Methods

Cognitive functions and related brain features correlated to the SRK levels

In order to define a complete overview of the considered topic, a multidisciplinary approach has been used. Such approach is summarised in Fig. 1.

Figure 1
Figure 1

Simplified illustration of the multi-disciplinary approach used in the study. Three levels of analysis have been considered: human factor concepts, realism of the experimental task, and neurophysiological evidences related to the main cognitive phenomena linked to the SRK behaviours.

Firstly, a literature review has been specifically done to find out HF concepts relevant for and used in the ATM field. Then, such HF concepts have been translated into psychophysiological functions to finally identify the brain features underlying cognitive processes involved in the SRK behaviours (green arrows). In other words, we sought the main cognitive processes involved in the three levels of cognitive control behaviours, and then we selected the EEG rhythms by which objectively describe such different behaviours. At the same time, an experimental ATM environment has been set up in order to induce the behaviours we wanted to assess in the Controllers (light blue arrows). Finally, experiments have been conducted in order to validate our hypothesis and verify that the selected brain features were capable of discriminating the three levels of cognitive control behaviour.

By definition, the three behaviours involve different cognitive processes and levels of automatism, and they correspond to operational conditions which require different amount of cognitive effort. For example, as quoted previously, the skill behaviour needs automatic actions. During rule behaviour, the user has to identify the right procedures and then apply them. In other words, the rule behaviour can be roughly defined as long-term memory access + automatic actions (skill behaviour). During knowledge behaviour, the user has to understand what is going on, organize and process the information, plan a strategy, find out the right procedures, and finally to apply them. This behaviour can be seen as information processing + decision making + procedures selection (rule behaviour) + automatic actions (skill behaviour). Therefore, the cognitive control behaviours, especially the rule and knowledge ones, are slightly overlapped to each other, and even if they were designed as pure as possible, they would always show some contributes derived from the others. In order to investigate the possibility to measure and quantify SRK events in realistic settings, we properly inserted the SRK events within ATM conditions of different complexity, where the difficulty profile of the task changed over time to simulate realistic air-traffic situations. Therefore, the issue could be to not being able to discriminate SRK behaviours when the difficulty of the faced condition changes (i.e. different workload demand). In this regard, the mental workload is a complex concept and it involves many cognitive processes, such as stress, attention, time pressure, task difficulty, emotional status, mental effort, physical demand, memory, etc38. For such a reason, the proposed work aimed to find out brain features able to characterize the three cognitive control behaviours (Skill, Rule, and Knowledge) even if the task difficulty (or workload demand) was changing. Brain activations are generally correlated to specific cognitive phenomena39. In fact, literature evidences show that an increase of EEG activity, especially in the theta band over the frontal cortex has been observed when the demand of executive control (attention and working memory)40,41,42, the activation of decision-making processes (like resolution of conflicts and error detection43), problem solving demand44, mental workload request38, 45,46,47,48,49,50,51, and the task complexity are high16, 17, 19, 52. Additionally, theta rhythms, engaged in the hippocampus/PFC (prefrontal cortex), potentially interplays in the consolidation of memory and empirical results reports the effect of enhanced theta oscillations on memory consolidation53,54,55,56,57, and induction of long-term plasticity58,59,60. Therefore, we hypothesized that enhanced parietal theta activation supports exchange of information between the hippocampus and neocortical areas, hence it could be considered as an indicator of memory consolidation61, 62.

Concerning the alpha EEG rhythm, the alpha-band frequency range has been usually associated with attention63,64,65. A measure of support for this notion comes from imaging studies reporting attentional modulations in the pulvinar nucleus66,67,68 and in the lateral geniculate nucleus69, 70. In the work of Neuper and Pfurtscheller71, it has been shown that the event-related desynchronization (ERD) of EEG activity in the alpha band reflected an increased excitability level of neurons in the involved cortical areas, which could be related to an enhanced information transfer in thalamo-cortical circuits. In contrast, event-related synchronization (ERS) of alpha activity (i.e. increases in alpha activity) is thought to reflect a reduced state of active information processing in the underlying neuronal networks72, reflecting the inhibitory aspect of alpha-band oscillations73, 74.

Klimesch75 argued that the two fundamental functions of attention, suppression and selection, enable selective access to the Knowledge System (KS) and operate accordingly to the proposed inhibition timing function of alpha-band activity76, 77. The KS is a storage system comprising not only traditional Long-Term Memory (LTM), but any type of knowledge, including procedural and implicit perceptual knowledge. Perception, encoding, and recognition are processes that are guided by attention and are closely related to access of information in the KS78, 79. Access to the KS may be an event-related or a continuous process. In the latter case, the alpha band frequency range reflects continuous ‘semantic orientation’, which represents the ability to be consciously oriented in time, space, and with respect to the meaning of all entities surrounding the individual80. This ability requires a certain kind of attention and it is knowledge-based, which means that semantic orientation provides individuals with the ability to selectively access stored information that represents the meaning of sensory information and ‘higher order information’, such as language, mathematics, and geography81,82,83. Selective access to the KS is thought to depend largely on inhibiting task-irrelevant memory entries. Thus, periods of prolonged access should be associated with ERS, reflecting increased (i.e. synchronization) alpha band activity84,85,86.

As a result, we hypothesized to discriminate the SRK behaviours by evaluating the degree of information processing, working memory (frontal theta EEG rhythm)16, 38, 45, 46, procedural memory (parietal theta EEG rhythm)43, 53, and attention (frontal alpha EEG rhythm)51, 64, 69, 75 along the execution of the SRK events. The brain activations supposed to characterize the SRK levels of cognitive control have been summarized in Table 1.

Table 1: Hypothesized SRK characterization by means of the selected EEG rhythms. The SRK behaviours were described in terms of variation from a reference condition: synch – synchronization; desynch – desynchronization.

Experimental Subjects

Informed consent for both study participation and publication of identifying information and images was obtained from a group of 15 ATC Experts (40.41 ± 5.54 years old), and a group of 22 ATC Students (23 ± 1.95 years old) in the final year of the course from the École Nationale de l’Aviation Civile (ENAC) of Toulouse (France) after the explanation of the study. In total, thirty-seven ATCOs (Air Traffic Controllers) were involved in the experiment. The ATCOs have been selected in order to have homogeneous experimental groups in terms of age and expertise. The experiment was conducted following the principles outlined in the Declaration of Helsinki of 1975, as revised in 2000. The study protocol received the favourable opinion from and has been approved by the Ethical Committee of the Sapienza University of Rome. The study involved only healthy participants, recruited on a voluntary basis. The ATCOs were free to accept or not to take part to the experimental protocol, and all of them accepted to participate to the study. Only aggregate information has been released while no individual information were or will be diffused in any form.

Air Traffic Management Simulation

The experimental setup has been designed by HF and ATCO Experts in order to provide a high level of realism in terms of similarity to the activities and context experienced by ATCOs in their daily operational environment. The ATCOs have been asked to perform an Air Traffic Management (ATM) scenario using a research simulator hosted at ENAC (Fig. 2). Also, at the end of the simulation, the controllers who participated to the experiment rated the realism of the ATM scenario as “Very high”. Two Pseudo-Pilots have also been involved in the experiments to simulate real pilots, flying the aircrafts under ATCOs control, and communicating with them through proper radio sector frequencies. Although there were only two pseudo-pilots, cabin noise and slight deformation of the pilots’ voice were chosen and associated to each aircraft by a voice recognition module developed by ENAC87. As a result, the ATCOs heard different voices and different cabin atmospheres for each assumed aircraft. The ATM simulation scenario has been designed from real traffic samples normally used in the training program at ENAC, and it has been modified according to the objectives and needs of the experiment. The adjustments mostly impacted the routes and air-traffic within the air-space sector (adding or changing them), while the borders of the sectors were not changed. The air-traffic adjustments consisted in flights rescheduling and flight levels (FLs) changing, according to the Number of aircraft, Traffic geometry, and Number of conflicts, with the aim to define different air-traffic complexity levels, Easy (E), Medium (M) and Hard (H) on average (but with a changing shape over time), and simulate realistic transitions between such ATC conditions. The total duration of the scenario was 45 minutes, while the E, M and H levels lasted 15 minutes each.

Figure 2
Figure 2

Experimental setup: ATCO working positions developed and hosted at ENAC (Toulouse, France). The ATCO’s brain activity has been recorded continuously during the execution of the ATM scenario, and the SRK events have been marked in order to recognize them within the EEG recording.

SRK Events

The ATM simulation scenario included specific SRK events designed to generate the corresponding Skill, Rule and Knowledge-based behaviours. A Subject Matter Expert (SME) from the Ente Nazionale di Assistenza al Volo (ENAV, Rome, Italy) has been involved in order to create realistic and not disruptive SRK events during the simulation (to be limited in time and not changing the realism of the scenario). Two triplets of SRK events (S1, R1, K1, and S2, R2, K2, 6 events in total) have been inserted into the ATM scenario in a randomized sequence, as reported in Fig. 3, in order to avoid any habituation and expectation effects.

Figure 3
Figure 3

Distribution of the SRK events along the considered ATM scenario. The triplets of SRK events have been inserted in a randomized sequence in order to avoid any habituation and expectation effects. No SRK events have been inserted in the Hard condition in order to do not generate unrealistic ATC conditions.

In particular, the SRK events have been inserted into task phases with different difficulty levels (3 SRK events in the E condition, and 3 events in the M condition) with the aim to check the capability of the selected brain features of characterizing the three cognitive control behaviours in realistic ATM settings. In fact, since the ATM scenario has been designed with the aim to achieve high air-traffic realism and conditions, the task difficulty and complexity kept varying along the time and, as a consequence, the different SRK events happened within different difficulty levels. Therefore, if the chosen brain features were, for example, related only to the task difficulty or workload, it could be not possible to discriminate the SRK events. The events have not been inserted into the H part of the ATM scenarios, since the introduction of external events could increase the complexity to an unacceptable level, generating too much disruption on the controlling activity and possibly invalidating part of the recording and of the realism of the task. The design of the SRK events has been the following. The Skill (S) events were basically interactions with the interface, during the task execution. Controllers were asked, by showing them a sheet with the instructions, to visualize the distance between two aircrafts (Distance event) or to display the Flight Plan Level (FPL) trajectory of each aircraft present in the controlled sector (Display FPs). The Rule (R) events were mainly control-tasks and conflicts-resolutions, during which Controllers were also performing skill-events (interaction with the interface). In the “Conflict event”, Controllers had to detect and solve a conflict by using the menu of the interface, assign new altitudes and headings, and eventually to call the pilots asking information about their FPLs. The hypothesis was that conflict detection task represented a familiar situation for ATCOs. Therefore, Controllers should recognize the correct procedures and usual solutions and then apply them to solve the conflict. The Knowledge (K) events integrated in the ATM scenario represented more unfamiliar and unusual situations. They generated a certain level of uncertainty, leading the Controllers to require time to analyse the situation and to find out the proper procedure to cope with the unexpected event. In other words, the ATCOs initially had to analyse the situation, identify the unusual situations, and then come back to the rule-based level for adopting the right procedures. In the first knowledge-based event, an aircraft deviated from the route filled in the initial FPL. An alarm was displayed on the Radar interface with the aim to make the ATCO focusing on the aircraft (a/c), checking its manoeuvres, and detecting that something was happening. Once contacted, the pilot claimed to be on the right FPL. Controllers needed to understand if there was a problem with the system or a pilot’s error. In the second knowledge-based event (Unidentified Flying Object - UFO), the Pseudo-Pilot reported an unknown-traffic detected by the Traffic Collision Avoidance System (TCAS) and a TCAS resolution advisory to avoid a mid-air collision. This unknown aircraft has not been displayed on the Controller’s radar image, who had been supposed to understand the situation and to ask additional information to the pilot of the considered aircraft. After the avoidance manoeuvre (e.g. descent), the pseudo-pilot had to ask for his previous flight-level, which never changed on Controller’s HMI because the pseudo-pilots were instructed to act during this event. The Controller could not observe neither the aircraft responsible of the TCAS advisory nor the implementation of the avoidance manoeuvre. In Table 2, we reported detailed descriptions of the six SRK events, in the proposed order, to show how couples of same type of events were not exactly the same, in terms of requested actions to perform, in order to make the results more robust to possible confounds on brain activations while accomplishing the SRK events. Finally, along the execution of the SRK events, SMEs (sat behind the ATCOs) checked if the Controllers performed correctly the considered events. The SMEs confirmed that all the involved Controllers, both Experts and Students, did execute correctly the proposed SRK events.

Table 2: Detailed descriptions of the six SRK events designed to induce Skill, Rule and Knowledge behaviours in the Controllers during the execution of the ATM scenario.

Physiological Signals Recording and Pre-Processing

The neurophysiological signals have been recorded using the digital monitoring BEmicro system (EBNeuro system, Italy). The nine EEG channels (FPz, F3, Fz, F4, AF3, AF4, P3, Pz and P4) have been collected with a sampling frequency of 256 (Hz). All the EEG electrodes have been referenced to both the earlobes, grounded to the mastoids, and the impedances of the electrodes were kept below 10 (kΩ). The acquired EEG signals have been digitally band-pass filtered by a 5th order Butterworth filter (low-pass filter cut-off frequency: 30 (Hz), high-pass filter cut-off frequency: 1 (Hz). The eye-blink artifacts have been removed from the EEG using the Reblinca method88. In particular, with respect to other regressive algorithms (e.g. Gratton method89) the Reblinca method presents the advantages to preserve EEG information in blink-free signal segments by using a specific threshold criterion that automatically recognize the occurrence of an eye-blink, and only in this case the method corrects the EEG signals. If there is not any blink, the method does not introduce any changes into the EEG signal. In addition, the Reblinca method does not require any EOG signal(s). The EEG signal has been then segmented in 2 second-epochs, shifted of 0.125 (sec), with the aim to have both a high number of observations in comparison with the number of variables, and to respect the condition of stationarity of the EEG signal90. In fact, the latter one is necessary in order to proceed with the spectral analysis of the signal. For other sources of artifacts, specific procedures of the EEGLAB toolbox have been applied91. In particular, three methods have been used: the threshold criterion, the trend estimation and the sample-to-sample difference. In the threshold criterion an EEG epoch has been marked as “artifact” if the EEG amplitude was higher than ±100 (μV). In the trend estimation, the EEG epoch has been interpolated in order to check the slope of the trend within the considered epoch. If such slope was higher than 3 (µV/epoch) (non-physiological variation), the considered epoch has been marked as “artifact”. The last check calculated the difference between consecutive EEG samples. If such difference, in terms of amplitude, was higher than 25 (μV), it meant that an abrupt variation (non-physiological) happened, thus it was marked as “artifact”. At the end, the EEG epoch marked as “artifact” have been removed with the aim to have a clean EEG dataset from which estimating the brain parameters for the different analyses. The Power Spectral Density (PSD) has then been estimated by using the Fast Fourier Transform (FFT) in the EEG frequency bands defined for each subject by the estimation of the Individual Alpha Frequency (IAF) value92. By segmenting the EEG signal, we achieved an average number of 224 epochs, for a 30 second-long event, with an average rejection rate of 17.3 ± 13.6%. In addition, by using 2 second-long Hanning windows for the PSD estimation, we obtained a frequency resolution of 0.5 (Hz). Thus, according to the definition of theta [IAF-6 ÷ IAF-2] and alpha [IAF-2 ÷ IAF + 2] EEG bands, we obtained 17 frequency bins, i.e. 17 PSD values for each EEG channel. Furthermore, the Baseline (brain activity during a minute of rest conditions, that is, closed eyes - OC) and the Reference (ATCOs looked at the radar screen without reacting for 3 minutes, where two no-colliding airplanes have been presented - REF) conditions have been recorded before starting with the ATM simulations. The OC condition has been used for the IAF estimation, while the REF condition has been used to evaluate the PSDs variations, with respect to the considered experimental condition, within the S, R and K events in terms of z-score93 values. The results proposed in the following sections will be represented as z-score values, thus values (y-axis) of zero mean that there are no differences, in terms of PSD, between the considered experimental condition (S, R and K) with respect to the REF one. Positive z-score values mean synchronization of the considered EEG rhythm (PSD increment), while negative z-score values mean desynchronization of the considered EEG rhythm (PSD decrement).

SRK Neurophysiological Characterization

The Power Spectral Density (PSD) has been estimated for the different brain features (frontal and parietal theta, and frontal alpha EEG rhythms), and the analysis of their spectral information has been performed in order to assess if they could be used to achieve the proposed goal. One-way repeated measures ANOVAs have been performed for each brain rhythm with the SRK, (3 levels: Skill, Rule and Knowledge) as within factor, and with the PSD as independent variable.

SRK Discrimination: Machine-Learning Analysis

The classification algorithm automatic stop StepWise Linear Discirminant Analysis (asSWLDA)17, 49 has been used to select the most relevant brain spectral features. In particular, the frontal and parietal theta, and frontal alpha EEG rhythms have been used for the calibration of the asSWLDA94 to characterize and discriminate the different cognitive control levels (SRK) during the execution of the ATM simulation. Therefore, the asSWLDA algorithm has been calibrated by using brain features (i.e. PSD epochs) extracted within the considered spectral domain (frontal and parietal theta, and frontal alpha) from one triplet of SRK events (S1, R1, K1), and then tested on the remaining triplet (S2, R2, K2), and vice-versa. For each testing triplet, the Area Under Curve (AUC) values have been calculated of the Receiver Operating Characteristic (ROC)95 by considering couples between SRK distributions. The AUC values related to the discrimination accuracy between the three couples of conditions (S vs R, S vs K, R vs K) have been calculated and analysed for each ATCO. In order to test the effectiveness of the algorithm, and that the SRK discrimination would not be due to chance, for each couple of conditions (S vs R, R vs K, S vs K) we have compared the AUC distributions obtained from the experimental data of the ATCOs (Measured AUC), with a random distribution calculated for each specific couple (Random AUC), situation corresponding to the random classification. In particular, we performed all the possible permutations along both the calibrating and testing dataset96, 97. In this regard we shuffled the SRK labels for the ATCO Experts related data, and then we calculated the AUC values between couples of conditions (e.g. S vs R) considering two cases: one in which the right labels have been used (Measured AUC), and the other in which the labels have been randomly set (Random AUC). The Random AUC distributions have been compared with the Measured AUC by using three two tailed student t-tests (α = 0.05), in order to demonstrate the reliability of the algorithm. For the ATCO Students, we were unable to perform a three-class analysis, because most of them missed the first knowledge-based event (Deviation). In this regard, we have considered only the rule and skill conditions. In particular, the asSWLDA has been calibrated by using one couple (S1, R1) and the discrimination accuracy (AUC values) has been tested on the remaining couple (S2, R2), and vice-versa. As consequence, the AUC distributions (Measured AUC, and Random AUC) have been calculated only for the “S vs R” comparison.

Comparison between ATCO Experts and Students

Two-tailed unpaired t-tests (α = 0.05) have been performed to compare the discrimination accuracy (i.e. AUC) between the skill and the rule conditions (the knowledge event was missed as quoted above) between the two groups (ATCO Experts and ATCO Students). The hypothesis was that the ATCO Students were not as skilled as the ATCO Experts, so that the skill-based events (that should be characterised by completely automated activities) could require a brain activation and attention as for the rule-based events. In this regard, the asSWLDA has been calibrated by using one couple of conditions (S1, R1), and the AUC values have been calculated on the remaining couple (S2, R2) and vice-versa, both for the ATCO Experts and Students. In addition, one-way ANOVAs have been performed on the frontal and parietal theta, and frontal alpha PSDs with the aim to assess eventual differences between the two groups (between factor RANK; 2 levels: Experts and Students). Before every statistical analysis, the z-score transformation98 has been used to normalize the data. Furthermore, the most common brain features selected by the asSWLDA, within the single EEG band and in total, have been analysed and compared between the two groups. In particular, we assigned “1” to the selected features, and then we summed up the values over the cross-validation and ATCOs, separately for the Experts and Students, to finally calculate the average rate by which each feature was picked for the SRK events discrimination. For example, if a frequency bin, within the frequency range “theta + alpha band”, was selected 5 times for all the ATC Experts, it was then divided by the total number of cross-validations (2 cross-validations) multiplied by the number of the considered ATCOs (15 Experts). Therefore, the final percentage corresponding to such a bin was (5/(2 * 15)) * 100 = 16.67%. Two-tailed unpaired t-tests have then been performed to assess possible differences between the ATCO groups in terms of selected brain features.

Results

SRK Neurophysiological Characterization

The results of the ANOVA on the frontal theta (Fig. 4) PSDs showed that this brain features changed significantly (F(2, 64) = 17.01; p < 10−5) among the S, R and K events, thus it could be used for the SRK discrimination. In particular, the post-hoc test reported high discriminability of the skill event with respect the other two (p < 0.00006), and the difference between the knowledge and the skill events (p < 0.0002). On the contrary, the rule and the knowledge did not differ significantly (p = 0.3).

Figure 4
Figure 4

The figure reports the results of the ANOVA analysis on the frontal theta PSD with the factor “SRK” of 3 levels (Skill, Rule and Knowledge). The results showed that such brain feature could be used as SRK discriminant brain feature, as its PSD values were significantly different (p = 0.000001) between the S, R and K levels.

The ANOVA performed on the parietal theta PSD (Fig. 5), demonstrated that it significantly changed across the SRK levels (F(2, 64) = 4.11; p = 0.021). The post-hoc showed that the parietal theta PSD could be used to discriminate accurately the skill from the knowledge level (p < 0.002). On the contrary, the parietal theta PSD did not change significantly between the rule – knowledge, and skill – rule levels (p > 0.25).

Figure 5
Figure 5

The figure reports the results of the ANOVA analysis (CI = 0.95) on the parietal theta PSD with the factor “SRK” of 3 levels (Skill, Rule and Knowledge) The results showed that such brain feature could be used as SRK discriminant brain feature, as its PSD values were significantly different (p = 0.02092) between the S, R and K levels.

The ANOVA on the frontal alpha PSD (Fig. 6) reported a significant effect in the SRK discrimination (F(2, 64) = 11.48; p < 10−4), and the post-hoc test showed that the frontal alpha rhythm could be used to differentiate either the skill from the rule (p = 0.0003) and knowledge levels (p = 0.0002). On the contrary, the frontal alpha PSD did not change significantly between the knowledge and rule levels (p > 0.8).

Figure 6
Figure 6

The figure reports the results of the ANOVA analysis (CI = 0.95) on the frontal alpha PSD with the factor “SRK” of 3 levels (Skill, Rule and Knowledge). The results showed that such brain feature could be used as SRK discriminant brain feature, as its PSD values were significantly different (p = 0.00006) between the S, R and K levels.

SRK Discrimination: Machine-Learning

As mentioned above, the asSWLDA was calibrated by using one triplet (S1, R1, K1) and the discrimination accuracy (AUC) values were calculated by testing it on the remaining triplet (S2, R2, K2), and vice-versa. In particular, for each couple of conditions (S vs R, R vs K, S vs K) we shuffled the SRK labels and then we calculated the AUC values considering two cases: one in which the right labels have been used (Measured AUC), and the other in which the labels have been randomly set (Random AUC). Referring on the results reported in the previous sections, the frontal and parietal theta, and frontal alpha EEG rhythms were defined as the frequency domain in which the asSWLDA classification model had to select the most significant features to discriminate the three cognitive control behaviours (SRK) from the chance level. Figure 7 reports the Measured AUC (blue bars) from the Random AUC (orange bars) for each ATCO (Experts on the top part of the image) averaged across all the possible labels combinations.

Figure 7
Figure 7

Bars plot of the averaged Measured AUC (blue bars) across all the possible labels combinations with respect to the Random AUC (orange bars) for each ATCO. In particular, the AUC values of the Experts are reported on the top part of the image, while those of the Students are reported on the bottom.

ATCO Experts

The two-tailed paired t-tests (α = 0.05) showed that all the Measured AUC (blue bars) distributions were significantly higher (left side of Fig. 8) with respect to the Random AUC (orange bars) distributions (all p < 0.002).

Figure 8
Figure 8

Bars plot related to the zscore-normalized Measured AUC (blue bars) distributions and the Random AUC (red bars) distributions, achieved by the ATCO Experts (left side) and Students (right side), referred to the discrimination accuracy between the three couples of conditions (S vs R, S vs K, R vs K) for Experts, and S vs R for Students. The three cognitive control behaviours could be significantly discriminate for both the groups (p < 0.05). In addition, the results show how the S and R behaviours were significantly (p < 0.05) more discriminable for the Experts than for the Students (p < 0.05).

ATCO Students

The two-tailed paired t-test (α = 0.05) showed that also for the ATCO Students all the Measured AUC (blue bars) distributions were significantly higher (right side of Fig. 8) than the Random AUC (orange bars) distributions (all p < 0.05).

Comparison between ATCO Experts and ATCO Students

Figure 8 shows also the differences between the two ATCO groups in terms of zscore-normalized AUCs. In particular, the ATCO Experts exhibited a significant higher (p < 0.05) discrimination accuracy with respect to the ATCO Students. The trend may indicate how the cognitive control behaviours were better discriminable for Experts than for Students.

In addition, the results performed on PSD values showed that the considered brain features could be used to distinguish the level of expertise (Expert vs Student) with significant reliability (all p < 0.03). In fact, the one-way ANOVAs showed significant differences between the group of ATCO Experts and ATCO Students in terms of brain activations (Figs 9, 10 and 11).

Figure 9
Figure 9

The figure reports the results of the ANOVA analysis (CI = 0.95) on the frontal theta PSD with the factor “RANK” of 2 levels (Experts and Students). The results showed that the two groups were statistically different (p = 0.005) in terms of activation of the frontal theta rhythm when facing the same SRK events.

Figure 10
Figure 10

The figure reports the results of the ANOVA analysis (CI = 0.95) on the parietal theta PSD with the factor “RANK” of 2 levels (Experts and Students). The results showed that the two groups were statistically different (p = 0.0023) in terms of activation of the parietal theta rhythm when facing the same S, R and K events.

Figure 11
Figure 11

The figure reports the results of the ANOVA analysis (CI = 0.95) on the frontal alpha PSD with the factor “RANK” of 2 levels (Experts and Students). The results showed that the two groups were statistically different (p = 0.024) in terms of activation of the frontal alpha rhythm when facing the same S, R and K events.

In particular, the ATCO Students showed both higher frontal and parietal theta synchronization, respectively (F(1, 32) = 9.14; p = 0.005) and (F(1, 32) = 10.97; p = 0.0023), and a lower (F(1, 32) = 5.62; p = 0.024) frontal alpha desynchronization than the ATCO Experts.

Selected Brain Features

In Figs 12 and 13, the selected brain features for the ATCO Experts and ATCO Students are respectively reported. The results showed that, on average, the selected brain features were 5.15 for the ATCO Experts, and 5.6 for the ATCO Students with no general statistical difference between them (p = 0.59). However, by the inspection of the figures, it can be highlighted that the selected features over the frontal areas of the Experts were lower than for the Students. Thereafter, two two-tailed unpaired t-tests have been performed separately on the features selected within the frontal EEG channels, and within the parietal EEG channels. The results showed a clear trend (even if not strictly significant p = 0.07) in the number of selected features over the frontal areas, since they were significantly higher for the ATCO Students (Fig. 13) than for the ATCO Experts (Fig. 12), while no difference were found among the selected parietal features (p = 0.63).

Figure 12
Figure 12

Percentages related to the brain features most commonly selected by the asSWLDA across the ATCO Experts for the SRK-based cognitive control behaviours discrimination. White color means that brain features have not been selected at all. On the contrary, the red colors means that the brain features were selected more frequently.

Figure 13
Figure 13

Percentages related to the brain features most commonly selected by the asSWLDA across the ATCO Students for the SRK-based cognitive control behaviours discrimination. White color means that brain features have not been selected at all. On the contrary, the red colors means that the brain features were selected more frequently.

Discussion

Several studies28, 33, 99,100,101 tried to model the different information processing procedures based on the Skill, Rule and Knowledge cognitive control behaviours from a HF perspective. Despite such studies, there are not evidences in which these cognitive control levels have been taken into account from a neurophysiological point of view, for example, by considering the EEG variations between the different SRK levels. The multi-disciplinary approach102 applied in this work started with an extensive literature review that identified the main cognitive processes related to the S, R and K-based behaviours. They were characterized considering only cognitive processes linked to the level of automatism when dealing with the different events. In other words, we sought features linked to information processing, decision making, working memory, and attention, without considering motor brain features, to avoid SRK behaviours discrimination depending on different amount of movements (hands and/or feet) during the execution of the ATC activities. Furthermore, the ATM scenario was designed as realistic as possible, and validated by ATCO and HF experts. Additionally, all the SRK events were inserted within different difficulty levels continuously changing over time to investigate whether the selected brain features were appropriate to discriminate the cognitive control behaviours despite the task difficulty or workload variations. The considered brain features for the SRK discrimination were the frontal and parietal theta, and the frontal alpha EEG rhythms. In fact, the frontal theta rhythm is linked to decision–making processes and working memory, therefore it is directly correlated to the amount of information and stimuli to process. Hence, the more complex the task is, the higher the theta synchronization over the frontal brain areas is. The second one, the parietal theta, is correlated to the procedural memory, and the more sustained the memory consolidation is, the higher its synchronization over the parietal brain areas is. The third EEG rhythm, frontal alpha, is correlated to the attentional level, and it has been demonstrated that the higher the attention is, the more the desynchronization of the alpha activity over the frontal brain areas is. The analysis of the frontal and parietal theta, and frontal alpha PSDs across the SRK levels confirmed that these brain features changed significantly across the SRK levels (Figs 4, 5 and 6). In fact, as expected, the frontal theta PSD increased with the complexity of the task, in particular especially in correspondence of the knowledge than in the skill events. The parietal theta PSD, representative of memory consolidation, increased less during automatic activities (i.e. skill events) than during more procedural or planning activities, that is knowledge and rule events. Similarly, the frontal alpha EEG rhythm, correlated to attentional level, desynchronised more during automatic actions than during activities where access to Knowledge System (KS) was required, as rule and knowledge events.

We would like to underline that by the analysis of the PSDs was not possible to discriminate all the SRK levels, but only the K, R behaviours from the S one, and not the K from the R ones. On the contrary, by using a machine-learning approach, we were able to discriminate significantly all the three cognitive control behaviours. In addition, the considered EEG features allowed to highlight the differences between the ATCO Experts and ATCO Students groups, in terms of expertise (Figs 7, 8, 9, 10 and 11). In this regard, the ATCO Experts showed higher memory consolidation and both lower attentional request and demand on executive controls than the ATCO Students. Such brain features were then used to calibrate the asSWLDA classification algorithm, to investigate the possibility to discriminate the three cognitive control levels during the execution of the realistic ATM scenario. The results confirmed that the asSWLDA was able to discriminate significantly the SRK conditions, as all the measured AUC distributions were significantly higher (p < 0.05) than the chance level (Fig. 7). Additionally, the analysis on the amount of brain features selected by the asSWLDA for each group demonstrated how the ATCO Students showed higher frontal brain activation than the ATCO Experts in dealing with the same SRK events (Figs 12 and 13). Therefore, we may conclude that such difference supports the higher information processing and working memory activity for the Students, with respect to the Experts, attributable to the different degree of expertise.

A limitation of the study has also to be discussed. In fact, because of the high realism of the ATM scenario, it was not possible to strictly design “pure” SRK events. Therefore, in combination with the SRK events, ATC activities were also performed. In order to investigate the possibility to measure and quantify SRK events in realistic settings, the HF and ATCO experts properly inserted the SRK events within ATM conditions of different complexity, where the difficulty profile of the task changed over time to simulate realistic air-traffic situations. Therefore, we tested if we were able to discriminate the SRK behaviours when the difficulty of the faced condition changed (i.e. different workload demand). Consequently, in the next studies we shall design experimental protocols by using controlled tasks in order to better simulate and induce pure skill, rule and knowledge-based behaviours into the users. This design will decrease the realism of the simulation, but it could allow to confirm and refine the promising findings of this work.

Conclusions

In this study, it has been demonstrated that it was possible to assess, with a high reliability, the ATCOs’ cognitive control behaviour (SRK) while performing realistic Air-Traffic-Management (ATM) scenario by means of specific brain features. Such brain features were selected with the aim to characterize the different SRK levels in terms of “cognitive automatism”, minimizing possible influences of motor activity (hands or feet movements). The results showed that i) the proposed neurometric (frontal and parietal theta, and frontal alpha EEG rhythms) could be valid metrics to investigate and objectively analyse the user’s cognitive control behaviour (SRK); ii) the asSWLDA was able to discriminate significantly the cognitive control behaviour for both the ATCOs groups (Experts and Students); and iii) the proposed methodology could be applied to assess differences in expertise between groups.

The proposed method could be potentially used as a Human Factors tool during the design phase of new solutions, as a Training tool to evaluate the level of learning and expertise gained by the user, and as a Monitoring tool to be used during operations (e.g. crew monitoring on a/c or ATC rooms). Additionally, it could provide objective information for a more precise prediction of the ways the operators may possibly fail in achieving their goals. This is particularly important for the Safety and Human Factors fields, whose goals are enhancing people performance, avoiding error-prone situations, and neuroergonomics designing of environments4, 103.

Additional Information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Rasmussen, J. The definition of human error and a taxonomy for technical system design. New technology and human error 23–30 (1987).

  2. 2.

    Rasmussen, J. & Vicente, K. Coping with Human Errors Through System Design: Implications for Ecological Interface Design. Int J Man-Mach Stud 31, 517–534 (1989).

  3. 3.

    Vicente, K. J. & Rasmussen, J. Ecological interface design: theoretical foundations. IEEE Trans. Syst. Man Cybern 22, 589–606 (1992).

  4. 4.

    Giraudet, L., Imbert, J.-P., Bérenger, M., Tremblay, S. & Causse, M. The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures. Behav. Brain Res. 294, 246–253 (2015).

  5. 5.

    Gluck, K. A. & Pew, R. W. Modeling Human Behavior With Integrated Cognitive Architectures: Comparison, Evaluation, and Validation. (Psychology Press, 2006).

  6. 6.

    Kahneman, D. Thinking, Fast and Slow. (Macmillan, 2011).

  7. 7.

    Rasmussen, J. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans. Syst. Man Cybern. SMC-13, 257–266 (1983).

  8. 8.

    Rasmussen, J. Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. (Elsevier Science Inc., 1986).

  9. 9.

    Reason, J. The Human Contribution: Unsafe Acts, Accidents and Heroic Recoveries. (CRC Press, 2008).

  10. 10.

    Pietro Arico, G. B. Reliability Over Time of EEG-Based Mental Workload Evaluation During Air Traffic Management (ATM) Tasks. EMBC 2015 25th–29th August Milan Italy 7242–7245 (2015).

  11. 11.

    Borghini, G. et al. A neurophysiological training evaluation metric for Air Traffic Management. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 3005–3008, doi:10.1109/EMBC.2014.6944255 (2014).

  12. 12.

    Brookings, J. B., Wilson, G. F. & Swain, C. R. Psychophysiological responses to changes in workload during simulated air traffic control. Biol. Psychol. 42, 361–377 (1996).

  13. 13.

    Hilburn, B. Cognitive complexity in air traffic control: A litera-ture review (EUROCONTROL Experimental Centre Note No. 04/04). Brussels, Belgium: EUROCONTROL.

  14. 14.

    Aricò, P. et al. Air-Traffic-Controllers (ATCO) - Neurophysiological analysis of training and workload ATCO. Ital. J. Aerosp. Med. 12, 18–35 (2015).

  15. 15.

    Borghini, G. et al. Quantitative Assessment of the Training Improvement in a Motor-Cognitive Task by Using EEG, ECG and EOG Signals. Brain Topogr, doi:10.1007/s10548-015-0425-7 (2015).

  16. 16.

    Toppi, J. et al. Investigating Cooperative Behavior in Ecological Settings: An EEG Hyperscanning Study. PLOS ONE 11, e0154236 (2016).

  17. 17.

    Aricò, P. et al. A passive Brain-Computer Interface (p-BCI) application for the mental workload assessment on professional Air Traffic Controllers (ATCOs) during realistic ATC tasks. Prog. Brain Res. Press (2016).

  18. 18.

    Borghini, G. et al. Avionic Technology Testing by using a Cognitive Neurometric Index: A Study with Professional Helicopter Pilots. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2015, 6182–6185 (2015).

  19. 19.

    Vecchiato, G. et al. Investigation of the effect of EEG-BCI on the simultaneous execution of flight simulation and attentional tasks. Med. Biol. Eng. Comput. 54, 1503–1513 (2016).

  20. 20.

    Flumeri, G. D. et al. In Symbiotic Interaction (eds. Blankertz, B., Jacucci, G., Gamberini, L., Spagnolli, A. & Freeman, J.) 45–56 (Springer International Publishing, 2015), doi:10.1007/978-3-319-24917-9_5.

  21. 21.

    Burns, C. M. & Hajdukiewicz, J. Ecological Interface Design. (CRC Press, 2004).

  22. 22.

    Reason, J. Human error: models and management. BMJ 320, 768–770 (2000).

  23. 23.

    Posner, M. I. & Petersen, S. E. The Attention System of the Human Brain. Annu. Rev. Neurosci. 13, 25–42 (1990).

  24. 24.

    Kochanska, G., Murray, K. T. & Harlan, E. T. Effortful control in early childhood: continuity and change, antecedents, and implications for social development. Dev. Psychol 36, 220–232 (2000).

  25. 25.

    Posner, M. I. Orienting of attention. Q. J. Exp. Psychol. 32, 3–25 (1980).

  26. 26.

    Raz, A. & Buhle, J. Typologies of attentional networks. Nat. Rev. Neurosci. 7, 367–379 (2006).

  27. 27.

    Corbetta, M. & Shulman, G. L. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215 (2002).

  28. 28.

    Shiffrin, R. M. & Schneider, W. Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychol. Rev. 127–190 (1977).

  29. 29.

    Neumann, O. In Cognition and Motor Processes (eds. Prinz, W. & Sanders, A. F.) 255–293 (Springer Berlin Heidelberg, 1984).

  30. 30.

    Allport, A. In Consciousness in Contemporary Science (eds. Marcel, A. J. & Bisiach, E.) (Oxford University Press, 1988).

  31. 31.

    Kellogg, R. T. Is conscious attention necessary for long-term storage? J. Exp. Psychol. [Hum. Learn.] 6, 379–390 (1980).

  32. 32.

    Marcel, A. J. Conscious and unconscious perception: Experiments on visual masking and word recognition. Cognit. Psychol. 15, 197–237 (1983).

  33. 33.

    Norman, D. A. & Shallice, T. In Consciousness and Self-Regulation (eds. Davidson, R. J., Schwartz, G. E. & Shapiro, D.) 1–18 (Springer US, 1986).

  34. 34.

    Baars, B. J. Some Essential Differences Between Consciousness and Attention, Perception, and Working Memory. Conscious. Cogn. 6, 363–371 (1997).

  35. 35.

    LaBerge, D. Attentional Processing: The Brain’s Art of Mindfulness. (Harvard University Press, 1995).

  36. 36.

    Schmidt, R. A., Lange, C. & Young, D. E. Optimizing summary knowledge of results for skill learning. Hum. Mov. Sci. 9, 325–348 (1990).

  37. 37.

    Causse, M., Imbert, J.-P., Giraudet, L., Jouffrais, C. & Tremblay, S. The Role of Cognitive and Perceptual Loads in Inattentional Deafness. Front. Hum. Neurosci. 344, doi:10.3389/fnhum.2016.00344 (2016).

  38. 38.

    Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D. & Babiloni, F. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci. Biobehav. Rev 44, 58–75 (2014).

  39. 39.

    Brain Mapping. MIT Technology Review Available at: https://www.technologyreview.com/s/526501/brain-mapping/. (Accessed: 27th May 2016).

  40. 40.

    Gevins, A. & Smith, M. E. Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci 4, 113–131 (2003).

  41. 41.

    Jensen, O., Kaiser, J. & Lachaux, J.-P. Human gamma-frequency oscillations associated with attention and memory. Trends Neurosci. 30, 317–324 (2007).

  42. 42.

    Sterman, M. B., Mann, C. A., Kaiser, D. A. & Suyenobu, B. Y. Multiband topographic EEG analysis of a simulated visuomotor aviation task. Int. J. Psychophysiol. Off. J. Int. Organ. Psychophysiol 16, 49–56 (1994).

  43. 43.

    Beulen, M. A. The role of theta oscillations in memory and decision making. (2011). Available at: http://dspace.library.uu.nl/handle/1874/206352. (Accessed: 31st May 2016).

  44. 44.

    Sandkühler, S. & Bhattacharya, J. Deconstructing insight: EEG correlates of insightful problem solving. PloS One 3, e1459 (2008).

  45. 45.

    Borghini, G. et al. Frontal EEG theta changes assess the training improvements of novices in flight simulation tasks. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2013, 6619–6622 (2013).

  46. 46.

    Borghini, G. et al. Assessment of mental fatigue during car driving by using high resolution EEG activity and neurophysiologic indices. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Conf. 2012, 6442–6445 (2012).

  47. 47.

    Arico, P. et al. Towards a multimodal bioelectrical framework for the online mental workload evaluation. in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 3001–3004 doi:10.1109/EMBC.2014.6944254 (2014).

  48. 48.

    Maglione, A. et al. Evaluation of the workload and drowsiness during car driving by using high resolution EEG activity and neurophysiologic indices. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2014, 6238–6241 (2014).

  49. 49.

    Aricò, P. et al. Adaptive Automation Triggered by EEG-Based Mental Workload Index: A Passive Brain-Computer Interface Application in Realistic Air Traffic Control Environment. Front. Hum. Neurosci. 539, doi:10.3389/fnhum.2016.00539 (2016).

  50. 50.

    Kong, W., Lin, W., Babiloni, F., Hu, S. & Borghini, G. Investigating Driver Fatigue versus Alertness Using the Granger Causality Network. Sensors 15, 19181–19198 (2015).

  51. 51.

    Toppi, J. et al. Time varying effective connectivity for describing brain network changes induced by a memory rehabilitation treatment. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 6786–6789, doi:10.1109/EMBC.2014.6945186 (2014).

  52. 52.

    Gevins, A. et al. Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Hum. Factors 40, 79–91 (1998).

  53. 53.

    Albouy, G. et al. Both the hippocampus and striatum are involved in consolidation of motor sequence memory. Neuron 58, 261–272 (2008).

  54. 54.

    Borghini, G. et al. Quantitative Assessment of the Training Improvement in a Motor-Cognitive Task by Using EEG, ECG and EOG Signals. Brain Topogr, doi:10.1007/s10548-015-0425-7 (2015).

  55. 55.

    Borghini, G. et al. Neurophysiological Measures for Users’ Training Objective Assessment During Simulated Robot-Assisted Laparoscopic Surgery. 38th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBC’16 (2016).

  56. 56.

    Vecchiato, G. et al. An electroencephalographic Peak Density Function to detect memorization during the observation of TV commercials. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2014, 6969–6972 (2014).

  57. 57.

    Taya, F. et al. Training-induced changes in information transfer efficiency of the brain network: A functional connectome approach. in 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER) 1028–1031, doi:10.1109/NER.2015.7146802 (2015).

  58. 58.

    Chauvette, S., Seigneur, J. & Timofeev, I. Sleep oscillations in the thalamocortical system induce long-term neuronal plasticity. Neuron 75, 1105–1113 (2012).

  59. 59.

    Kropotov, J. D. Quantitative EEG, Event-Related Potentials and Neurotherapy. (Academic Press, 2010).

  60. 60.

    Sauseng, P., Klimesch, W., Schabus, M. & Doppelmayr, M. Fronto-parietal EEG coherence in theta and upper alpha reflect central executive functions of working memory. Int. J. Psychophysiol. Off. J. Int. Organ. Psychophysiol 57, 97–103 (2005).

  61. 61.

    Gruzelier, J. A theory of alpha/theta neurofeedback, creative performance enhancement, long distance functional connectivity and psychological integration. Cogn. Process. 10(Suppl 1), S101–109 (2009).

  62. 62.

    Gruzelier, J., Egner, T. & Vernon, D. Validating the efficacy of neurofeedback for optimising performance. Prog. Brain Res. 159, 421–431 (2006).

  63. 63.

    Crick, F. Function of the thalamic reticular complex: the searchlight hypothesis. Proc. Natl. Acad. Sci. USA 81, 4586–4590 (1984).

  64. 64.

    LaBerge, D. Attention, consciousness, and electrical wave activity within the cortical column. Int. J. Psychophysiol. Off. J. Int. Organ. Psychophysiol 43, 5–24 (2001).

  65. 65.

    Compston, A. The Berger rhythm: potential changes from the occipital lobes in man, by E.D. Adrian and B.H.C. Matthews (From the Physiological Laboratory, Cambridge). Brain 1934: 57; 355–385. Brain 133, 3–6 (2010).

  66. 66.

    LaBerge, D. & Buchsbaum, M. S. Positron emission tomographic measurements of pulvinar activity during an attention task. J. Neurosci. Off. J. Soc. Neurosci. 10, 613–619 (1990).

  67. 67.

    Petersen, S. E. & Posner, M. I. The Attention System of the Human Brain: 20 Years After. Annu. Rev. Neurosci. 35, 73–89 (2012).

  68. 68.

    Petersen, S. E., Robinson, D. L. & Morris, J. D. Contributions of the pulvinar to visual spatial attention. Neuropsychologia 25, 97–105 (1987).

  69. 69.

    O’Connor, D. H., Fukui, M. M., Pinsk, M. A. & Kastner, S. Attention modulates responses in the human lateral geniculate nucleus. Nat. Neurosci. 5, 1203–1209 (2002).

  70. 70.

    Vanduffel, W., Tootell, R. B. & Orban, G. A. Attention-dependent suppression of metabolic activity in the early stages of the macaque visual system. Cereb. Cortex N. Y. N 1991 10, 109–126 (2000).

  71. 71.

    Neuper, C. & Pfurtscheller, G. Event-related dynamics of cortical rhythms: frequency-specific features and functional correlates. Int. J. Psychophysiol. Off. J. Int. Organ. Psychophysiol 43, 41–58 (2001).

  72. 72.

    Neuper, C. & Klimesch, W. Event-Related Dynamics of Brain Oscillations. (Elsevier, 2006).

  73. 73.

    Jensen, O., Bonnefond, M. & VanRullen, R. An oscillatory mechanism for prioritizing salient unattended stimuli. Trends Cogn. Sci. 16, 200–206 (2012).

  74. 74.

    Händel, B. F., Haarmeier, T. & Jensen, O. Alpha oscillations correlate with the successful inhibition of unattended stimuli. J. Cogn. Neurosci 23, 2494–2502 (2011).

  75. 75.

    Klimesch, W. α-band oscillations, attention, and controlled access to stored information. Trends Cogn. Sci. 16, 606–617 (2012).

  76. 76.

    Klimesch, W., Sauseng, P. & Hanslmayr, S. EEG alpha oscillations: the inhibition-timing hypothesis. Brain Res. Rev. 53, 63–88 (2007).

  77. 77.

    Klimesch, W. Evoked alpha and early access to the knowledge system: the P1 inhibition timing hypothesis. Brain Res. 1408, 52–71 (2011).

  78. 78.

    Summerfield, C. & Egner, T. Expectation (and attention) in visual cognition. Trends Cogn. Sci. 13, 403–409 (2009).

  79. 79.

    Friston, K. The free-energy principle: a rough guide to the brain? Trends Cogn. Sci. 13, 293–301 (2009).

  80. 80.

    Raichle, M. E. et al. A default mode of brain function. Proc. Natl. Acad. Sci. 98, 676–682 (2001).

  81. 81.

    Tononi, G. Consciousness as integrated information: a provisional manifesto. Biol. Bull. 215, 216–242 (2008).

  82. 82.

    Tononi, G. An information integration theory of consciousness. BMC Neurosci. 5, 42 (2004).

  83. 83.

    Klimesch, W. The structure of long-term memory: a connectivity model of semantic processing. (Lawrence Erlbaum, 1994).

  84. 84.

    Benedek, M., Bergner, S., Könen, T., Fink, A. & Neubauer, A. C. EEG alpha synchronization is related to top-down processing in convergent and divergent thinking. Neuropsychologia 49, 3505–3511 (2011).

  85. 85.

    Jauk, E., Benedek, M. & Neubauer, A. C. Tackling creativity at its roots: Evidence for different patterns of EEG alpha activity related to convergent and divergent modes of task processing. Int. J. Psychophysiol. 84, 219–225 (2012).

  86. 86.

    Zanto, T. P., Rubens, M. T., Thangavel, A. & Gazzaley, A. Causal role of the prefrontal cortex in top-down modulation of visual processing and working memory. Nat. Neurosci. 14, 656–661 (2011).

  87. 87.

    Imbert, J.-P., Hurter, C. & Jestin, Y. Think different: how we completely changed the visualization of Pseudo-Pilots. in (2015).

  88. 88.

    Di Flumeri, G., Aricò, P., Borghini, G., Colosimo, A. & Babiloni, F. A new regression-based method for the eye blinks artifacts correction in the EEG signal, without using any EOG channel. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. (2016).

  89. 89.

    Gratton, G., Coles, M. G. & Donchin, E. A new method for off-line removal of ocular artifact. Electroencephalogr. Clin. Neurophysiol. 55, 468–484 (1983).

  90. 90.

    Elul, R. Gaussian behavior of the electroencephalogram: changes during performance of mental task. Science 164, 328–331 (1969).

  91. 91.

    Delorme, A. & Makeig, S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21 (2004).

  92. 92.

    Klimesch, W. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Brain Res. Rev. 29, 169–195 (1999).

  93. 93.

    Zhang, J. H., Chung, T. D. & Oldenburg, K. A Simple Statistical Parameter for Use in Evaluation and Validation of High Throughput Screening Assays. J. Biomol. Screen. 4, 67–73 (1999).

  94. 94.

    Aricò, P., Borghini, G., Di Flumeri, G. & Babiloni, F. Metodo per stimare uno stato mentale, in particolare un carico di lavoro, e relativo apparato (A Method for the estimation of mental state, in particular of the mental workload and its device). (2015).

  95. 95.

    Bamber, D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J. Math. Psychol. 12, 387–415 (1975).

  96. 96.

    Combrisson, E. & Jerbi, K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J. Neurosci. Methods 250, 126–136 (2015).

  97. 97.

    Billinger, M. et al. In Towards Practical Brain-Computer Interfaces (eds. Allison, B. Z., Dunne, S., Leeb, R., Millán, J. D. R. & Nijholt, A.) 333–354 (Springer Berlin Heidelberg, 2012), doi:10.1007/978-3-642-29746-5_17.

  98. 98.

    Zhang, J.-H., Chung, T. D. Y. & Oldenburg, K. R. A Simple Statistical Parameter for Use in Evaluation and Validation of High Throughput Screening Assays. J. Biomol. Screen. 4, 67–73 (1999).

  99. 99.

    Petersen, S. E., van Mier, H., Fiez, J. A. & Raichle, M. E. The effects of practice on the functional anatomy of task performance. Proc. Natl. Acad. Sci. USA 95, 853–860 (1998).

  100. 100.

    Rasmussen, J. Information processing and human-machine interaction: an approach to cognitive engineering. (North-Holland, 1986).

  101. 101.

    Roy, R. N., Bonnet, S., Charbonnier, S. & Campagne, A. Mental fatigue and working memory load estimation: interaction and implications for EEG-based passive BCI. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2013, 6607–6610 (2013).

  102. 102.

    Bekhet, A. K. & Zauszniewski, J. A. Methodological triangulation: an approach to understanding data. Nurse Res. 20, 40–43 (2012).

  103. 103.

    Parasuraman, R. & Rizzo, M. Neuroergonomics: The Brain at Work. (Oxford University Press, 2008).

Download references

Acknowledgements

This work is co-financed by the European Commission by Horizon2020 projects Sesar-01-2015 Automation in ATM, “STRESS”, GA n. 699381 and Sesar-06-2015 High Performing Airport Operations, “MOTO”, GA n. 699379. The grant provided by the Italian Minister of University and Education under the PRIN 2012, GA n. 2012WAANZJ scheme to F.B. is also gratefully acknowledged.

Author information

Author notes

  1. Gianluca Borghini, Pietro Aricò and Gianluca Di Flumeri contributed equally to this work.

Affiliations

  1. Dept. of Molecular Medicine, Sapienza University of Rome, Piazzale Aldo Moro, 5, 00185, Rome, Italy

    • Gianluca Borghini
    • , Pietro Aricò
    • , Giulia Cartocci
    •  & Fabio Babiloni
  2. BrainSigns srl, via Sesto Celere, 00152, Rome, Italy

    • Gianluca Borghini
    • , Pietro Aricò
    • , Gianluca Di Flumeri
    • , Giulia Cartocci
    •  & Fabio Babiloni
  3. IRCCS Fondazione Santa Lucia, Neuroelectrical Imaging and BCI Lab, Via Ardeatina, 306, 00179, Rome, Italy

    • Gianluca Borghini
    • , Pietro Aricò
    •  & Gianluca Di Flumeri
  4. Dept. of Anatomical, Histological, Forensic & Orthopedic Sciences, Sapienza University of Rome, Piazzale Aldo Moro, 5, 00185, Rome, Italy

    • Gianluca Di Flumeri
    •  & Alfredo Colosimo
  5. DeepBlue srl, Piazza Buenos Aires 20, 00185, Rome, Italy

    • Stefano Bonelli
    • , Alessia Golfetti
    •  & Simone Pozzi
  6. École Nationale de l’Aviation Civile, 7 Avenue Edouard Belin, 31000, Toulouse, France

    • Jean Paul Imbert
    • , Géraud Granger
    •  & Railane Benhacene

Authors

  1. Search for Gianluca Borghini in:

  2. Search for Pietro Aricò in:

  3. Search for Gianluca Di Flumeri in:

  4. Search for Giulia Cartocci in:

  5. Search for Alfredo Colosimo in:

  6. Search for Stefano Bonelli in:

  7. Search for Alessia Golfetti in:

  8. Search for Jean Paul Imbert in:

  9. Search for Géraud Granger in:

  10. Search for Railane Benhacene in:

  11. Search for Simone Pozzi in:

  12. Search for Fabio Babiloni in:

Contributions

Gianluca Borghini, Pietro Aricò, and Gianluca Di Flumeri: EEG recordings, data analysis and paper writing. Gianluca Borghini, Giulia Cartocci, Alessia Golfetti, and Stefano Bonelli: SRK study and neurophysiological characterization, experimental protocol design. Railane Benhacene, Jean-Paul Imbert, and Géraud Granger: design and validation of the ATM scenario and SRK events, recruiting of professional Controllers, providing experimental facilities. Alfredo Colosimo, Simone Pozzi, Fabio Babiloni: experimental protocol desing and paper check.

Competing Interests

The authors declare that they have no competing interests.

Corresponding author

Correspondence to Gianluca Borghini.

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Creative Commons BY

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.