Uncovering the fast, directional signal flow through the human temporal pole during semantic processing

Tiesinga, P.; Platonov, A.; Pelliccia, V.; LoRusso, G.; Sartori, I.; Orban, G. A.

doi:10.1038/s41598-023-33318-5

Download PDF

Article
Open access
Published: 26 April 2023

Uncovering the fast, directional signal flow through the human temporal pole during semantic processing

P. Tiesinga¹,
A. Platonov²,
V. Pelliccia³,
G. LoRusso³,
I. Sartori³ &
…
G. A. Orban²

Scientific Reports volume 13, Article number: 6831 (2023) Cite this article

796 Accesses
1 Citations
Metrics details

Subjects

Abstract

The temporal pole (TP) plays a central role in semantic memory, yet its neural machinery is unknown. Intracerebral recordings in patients discriminating visually the gender or actions of an actor, yielded gender discrimination responses in the ventrolateral (VL) and tip (T) regions of right TP. Granger causality revealed task-specific signals travelling first forward from VL to T, under control of orbitofrontal cortex (OFC) and neighboring prefrontal cortex, and then, strongly, backwards from T to VL. Many other cortical regions provided inputs to or received outputs from both TP regions, often with longer delays, with ventral temporal afferents to VL signaling the actor’s physical appearance. The TP response timing reflected more that of the connections to VL, controlled by OFC, than that of the input leads themselves. Thus, visual evidence for gender categories, collected by VL, activates category labels in T, and consequently, category features in VL, indicating a two-stage representation of semantic categories in TP.

Right frontal HD-tDCS reveals causal involvement of time perception networks in temporal processing of concepts

Article Open access 03 October 2023

Fast temporal dynamics and causal relevance of face processing in the human temporal cortex

Article Open access 31 January 2020

The human endogenous attentional control network includes a ventro-temporal cortical node

Article Open access 15 January 2021

Introduction

Previous neuropsychology and imaging studies have established that the temporal pole (TP), located at the rostral end of the human temporal lobe, plays a crucial role in semantic memory¹, but how the TP achieves this function remains unknown^2,3,4, due to the lack of direct electrical recordings in TP. However, a growing number of studies have shown that TP is not an homogeneous region, but that it contains multiple cytoarchitectonic areas⁵, which have been grouped functionally using functional connectivity measured with fMRI^3,4,6,7. This heterogeneity of TP allows some questions about its operations to be formulated. One question concerns the flow of information in TP. Since some of the subparts are unimodal and others multimodal, one could expect signals to flow first from uni- to multimodal regions. Another question concerns the precise anatomical identity of the semantic hub: it is unclear whether the hub corresponds to the whole of TP or even the two TPs¹, or to one or two subparts of TP⁸. If the hub were to correspond to a single subpart, this raises the problem of how a small brain region can represent the wide range of semantic knowledge. To begin to address these questions, we build on our previous stereo EEG study⁹ (Methods, in “Patients”–”Standard statistical analysis” sections) showing that TP, more particularly the right TP, was specifically activated when patients visually discriminated the gender rather than the action performed by an actor (Fig. 1A,B). Indeed object concepts, and semantic categories in general, are basic elements of semantic memory^10,11, and deciding whether a visual test (here the image of a person) belongs to a semantic category (here male or female, subordinate levels of humans) is a typical way to test representations of such semantic categories¹⁰. Thus we used the gender discrimination task of Platonov et al.⁹ to investigate the representation of the semantic category gender in human TP. Hence, we concentrated on those patients tested with the Platonov et al. tasks⁹, in which right TP was well explored by stereo EEG recordings, to investigate the broadband gamma/high gamma responses (50–150 Hz, referred to as the broadband gamma in the following) in the different parts of TP. Importantly, we complemented this analysis by computation of Granger causality^12,13 between recorded leads to trace the directional flow of information within the TP and between TP and outside cortical regions.

Results

Functional heterogeneity of TP

The stimuli and tasks were exactly the same as in⁹ (see Methods, in “Behavioral testing” section). Patients viewed a screen on which a video, picturing a female or male actor performing one of two manipulative actions, was shown preceded by a static presentation of the first frame for a variable time (275–875 ms). In a two by two design either the complete video or a truncated one was shown (see Methods, in “Behavioral testing”), and patients either decided about the gender (using the static presentation) or discriminated the action performed (requiring the video). In a large group of 31 patients (Methods, in “Patients”), in which stereo EEG recordings (Methods, Section “StereoEEG data recording” section) in TP were available, 102/176 (58%) leads in right TP were responsive in the gender task (Fig. 1A, B), compared to 31/76 (39%) in left TP, confirming⁹. We therefore focused the present analysis on 19 out of these 31 patients with responsive leads in right TP, and on the trials in which a complete video of a manipulative action was shown. The 19 patients (8 females, age 21–49 y, Tables S1, S2) performed the gender decision task well (median accuracy 98%, median RT 827 ms after static onset). As in⁹, we found that TP leads show in the gender task brief, task-specific increases in broadband gamma (see Methods, in “Standard statistical analysis” section), which depended little on the duration of the static stimulus (Fig. 1C, top panels).

To investigate the functional heterogeneity of TP, we grouped the different architectonic areas of TP (5) into four subregions (Fig. 1D), following⁶. Specifically, we define a dorsal (Ds) subregion, matching the anterior TA cytoarchitectonic area and connected with auditory and somatosensory cortex⁶. We, however, considered cytoarchitectonic areas TG and anterior TE as separate subregions, labelled Tip (T) and ventrolateral (VL) respectively, because (1) TG in⁶ was the only subregion connected to all other TP subregions, indicating multimodality, while anterior TE was clearly visual in nature, as demonstrated by the connections with visual pulvinar, and (2) TG is dysgranular, while anterior TE is granular⁵. Finally, we grouped the remaining areas, rhinal cortex (areas 35, 36), enthorinal cortex, and TI, which all are agranular or dysgranular⁵, into a single ‘Medial’ (Md) subregion (Fig. 1D). The location and extent of these four TP subregions (Fig. 1D) was estimated from the location of the corresponding TP parts on the coronal sections of TP, shown in Fig. 1 of reference⁶. Of these four subregions, T is inferred from⁶ to be multimodal and VL only visual, something we cannot discriminate with the visual task we used. The proportions of responsive leads varied significantly (mixed effects, see Methods, in “Statistical tests” section and Suppl Text 1) between the four TP parts, being largest in T (85%), second in VL (57%), third in Md (42%) and smallest in Ds (8%, Figs. 1E, S1-1). While responsive leads are simply active during the static presentation in the gender task, a subset of these responsive leads is labelled selective when they are active during the static presentation but not during the video presentation in the gender task, nor during the static presentation in the action task. The proportion of selective leads showed similar significant (mixed effects, see Suppl Text 2) variations across the four TP parts, ranging from 58% of tested leads (T) to 8% (Ds, Fig. S1-1).

Responses in T were modestly stronger and longer, and had slightly shorter latencies than those in VL (see Table 1). In agreement with⁹, we found the RT to increase with response latency in VL and T rather similarly, but the correlation reached significance only in VL (Fig. 1F, G; VL: r = 0.80, p < 0.02, T: r = 0.51, p > 0.05). Although the number of patients contributing to the relationship for Md was small, this relationship was radically different in Md, with RT decreasing with increasing latency (RT = 939 ms-0.95 Lat), and the negative slope of Md falling outside the confidence interval of the slope for VL. To visualize the suppression in the action task we averaged (Fig. S1-2) the time courses of the leads of each TP part⁹, and observed only in T and VL a long lasting suppression in the action task, including both the static interval and the whole video presentation (Fig. S1-2). Thus our results bear out the functional heterogeneity of TP, with T and VL contributing heavily to the gender task, and T being more responsive and VL more related to the behavior. Although a substantial fraction of the tested TP leads (Black bar, 71/170 = 42%, Fig. S1-3A) were located in the epileptic zone (EZ), this impacted little our results, as a similar proportion of EZ leads occurred in the four regions, and both the proportion of responsive leads and the response strength were similar in and outside the EZ (Fig. S1-3B).

Table 1 Median (across patients) response properties of four subregions.

Full size table

In the videos both the face and the body of the actor were visible. Hence, to determine the source of the TP gender signals, we masked either the face or the hand (Fig. S2-1A) in four of the 19 patients (Table S1, Method in “Behavioral testing” section), and compared the responses in the gender task to that without masking. TP lead responses (averaging over VL and T leads) were similar in strength in all three conditions (Fig. 2A), but responses were delayed only when the face was masked, this delay correlating significantly (r = 0.79, p < 0.02) with an increase in RT (Fig. 2B), consistent with the increase in RT with latency reported earlier for the unmasked videos (Fig. 1F, G). In one patient (patient 2) we could verify that masking effects were similar in T and VL (Fig. S2-1B). Thus both body (hand) and face signals contributed to the gender decision responses in TP, face signals being faster than hand signals.

Intrinsic functional connections of TP

We have so far shown that T and VL are the two main TP regions involved in the use of visual information to decide about the gender of the actor. Next, we used Granger causality^12,13 (see Methods, in “Granger causality analysis” section ) to trace the information flow within and between these two regions, and compared them to their functional connections with the dorsal region. This analysis was performed in 7 of the 19 patients (Table S1). These included the four more recently recorded patients, also tested with masking, because the quality of their recordings reduced the need for artefact rejections (see Methods, in “Granger causality analysis” section). The 3 remaining patients were part of the previous study⁹, but had large numbers of leads in TP (13 or more). When computing the unconditioned Granger causality (uGC) in the time domain (see Methods, in “Granger causality analysis” section), we observed both tonic and phasic components (Fig. 3A, S3-1), carried by broadband gamma frequency bands (Fig. S3-2). We used z-scoring to isolate the phasic components (Figs. 3B, S3-3), and restricted all further analysis to these phasic Granger causality components. To cover the range of possible connections, and their corresponding range of temporal delays, we investigated in addition to these uGC calculated for a short delay (sampling rate 1000 Hz, with n = 4 lags in the GC calculation, i.e. maximally 4 ms, referred to as NS1, see Methods, in “Granger causality analysis” section) also uGC at longer delays (sampling rate 250 Hz, also at n = 4 lags, hence now maximally 16 ms, denoted as NS4), and observed a similar but coarser time course than the short delays (Fig. 3C, S3-4). Given the phasic nature of the uGC changes, we defined the strength of the uGC as the extreme value in the interval 0.1 to 0.4 s after static onset. Both uGC strength distributions display a longer positive tail in the gender than action task (Fig. 3D,E, S3-5), but showed little (albeit significant) correlation, indicating largely independent measurements (Fig. S3-5).The NS1-uGC depended less on low frequencies (below 50 Hz, 1–10 Hz) than the NS4-uGC, at least for the gender task (Table S3). Both uGC depended little on the power of the source or target leads (Tables S4, S5, see Methods, in “Granger causality analysis” section), indicating they are an independent measurement. In the gender task, connections within T and between T and VL were clearly stronger than connections of T or VL with Ds (Fig. 3F, mixed effects, see Suppl text 3). Furthermore, both NS1 and NS4 connections within T and between T and VL were significantly stronger in the backward than the forward direction, and overall connections in gender task were stronger than in action (mixed effects see Suppl text 4, Fig. S3-6). Very similar results were observed for the strength of the NS4-uGCs (Fig. 3G, S3-6).

We next investigated how the strength of the uGCs at the two delays combined for a pair of TP leads. Given the skewness of the uGC distributions (Fig. S3-5), we used a threshold of 5 to define strong connections of either sign, and considered a value of 3 as threshold of medium connections. Combining positive strong connections for NS1 and NS4 yielded 3 types (disappear, maintain and appear, Fig. 3D), while leads showing a strong negative uGC for at least one delay were grouped in a fourth type, labelled negative. These four types differed between the five connections between TP regions (within-T, T-VL, within-VL, VL-Ds and T-Ds with both directions combined): the uGC types with strong NS1 components (disappear and maintain) dominated in the within-T and the VL-T connections (mixed effects, see Suppl Text 5, Fig. S3-6). They also differed between the two directions of the within-T, T-VL and within-VL connections (mixed effects, see Suppl Text 6). This suggests that especially in the backward direction, a substantial fraction of these connections between and within T and VL are direct connections.

The previous analysis indicated the strength of the VL-T connections, linking the two regions most responsive in the gender task. We next investigated the timing of the T-VL NS1 uGCs. This could only be investigated in 5 of the 7 patients with leads in both VL and T and with strong enough uGCs between the two regions (we required 3 strong connections in at least one direction). Remarkably, the time course of the NS1-uGC between VL and T (Fig. 3H) was very similar to that of the TP activation, and equally task dependent. Furthermore, in all four patients for which data were available (Fig. S3-8), the NS1-uGC increase in the forward (VL to T) direction started significantly earlier than that in the backward direction (Fig. 3I), even if the backward uGC was stronger (Fig. 3F). Thus visual signals flow first, as predicted, from unimodal VL to multimodal T, but then quickly and strongly in the opposite direction. This flow could not be resolved by the latency of the responses in VL and T, which were very similar.

Next, to investigate to what extent the uGC between VL and T represented direct connections, we tested whether conditioning with activity of a third lead (see Methods, in “Granger causality analysis” section) could reduce the NS1 or NS4 uGC of strong VL-T connections to values below 3 (less than medium strength connections). This analysis was performed on the same five patients used for the timing of the VL-T connections. For conditioning leads outside TP we defined ‘via-leads’ as those leads connected to both the TP source and target (source to via-lead and via-lead to target uGCs > 5). Non-via-leads are the remaining leads for which one or both of the uGC scores did not exceed 5. NS1 uGC in the forward VL to T connection were conditioned by a few leads within VL, but by many more outside TP (Fig. 4A). Those included via-leads, all in the PFC and OFC, and non-via leads centered on the insula and the opercular regions (Fig. 4C), all of which exerted an outside control on the short-delay VL to T connection. For the NS4-uGC in the same direction (Fig. 4B), the number of conditioning leads were more balanced between VL and outside TP, but the latter were all non-via and scattered over the cortex, including in temporal and parietal cortex. In the backward direction (T to VL) significantly (mixed effect, see Suppl Text 7 & 7bis) fewer leads outside TP had a conditioning effect on the NS1-uGC, compared to the forward direction, and very few of these were via leads (Fig. 4A). For the NS4-uGC backward connection more outside leads had a conditioning effect, and for both delays these outside leads were located in prefrontal and temporal cortex (Fig. 4C, D, see Methods in “Visualization of lead location” section). These results are consistent with the view that internal NS1 and NS4 uGCs represent chiefly direct and indirect connections, respectively. It is noteworthy that in the backward direction for both delays conditioning leads within the TP source had a much more widespread effect on uGCs than the outside leads (Fig. 4E, F). For the maintain leads, which exhibit both strong NS1 and NS4 connections, we could compare the leads having a conditioning effect. We found that only a minority such leads matched for the two delays (11% and 18% in forward and backward directions respectively), providing further evidence of the independence of these two functional connectivity measurements. Also connections within a subregion (backwards within Tip, Fig. S5-1) were conditioned by outside TP leads, for both NS1 and NS4 uGCs, to a degree similar to T to VL connections, but this could be investigated only in 3 patients. Thus for intrinsic TP connections, uGCs at short delays were most prominent and for these delays, the two directions of the VL-T connections differed widely, the forward direction being earlier and more under outside control, and the backward direction much stronger.

Functional connections of TP with outside cortical regions

We next investigated the in- and output connections of the leads outside TP with both the VL and T regions in the 5 patients in which VL-T timing was investigated (Table S1). Indeed, in these patients electrodes targeted the two TP regions, and timing of intrinsic and extrinsic connections could be compared. Each patient had over 100 leads outside TP and globally the 619 outside leads in the five patients covered the whole cortex well (Fig. S5-1), with the exception of occipital cortex, implanted in only one patient, and parietal cortex, explored only in its rostro-ventral part (Table S6). Being interested in the functional connections of outside leads with T or VL, we averaged the uGC over the different VL or T leads available in a given patient. Again we considered uGC at two delays (Fig. S5-2): 4 ms (NS1) and 16 ms (NS4), which showed phasic increases synchronously with the TP activation, as illustrated for the four NS1 uGCs in Fig. 5A, B. These uGC distributions for the two delays were more similar than for the intrinsic connections, but showed again little correlation (Fig. 5C, S5-3, r = 0.254 (p < 3 10^–36) for gender and 0.355 (p < 2 10^–71) for action). For both delays the extrinsic uGC, unlike the intrinsic ones, depended on the source, target power and their product to a moderate degree (Tables S4, S5). On the other hand as for the intrinsic uGC, the extrinsic ones depended mainly on high frequencies for short delays in the gender task, and on both high and low temporal frequencies for the long delays (Table S3). Not surprisingly, the uGCs with outside leads, in contrast to the intrinsic uGCs, were weak (Fig. S5-4): only for short delays in the gender task some uGCs averaged over 1.5, yielding significant task dependency only for the short delays (mixed effect, see Suppl Text 8, 9). Hence, we used only the strong (> 5) uGCs to describe the outside connections of VL and T, using the four types defined above for internal TP connections (Fig. 5D). Although ‘disappear’ connections dominated all strong connections in the gender task, they reached significantly larger proportion in Tipin (from outside to T) connections than in the other 3 anatomical classes of outside connections (Tipout = from T to outside, VLin = from outside to VL, VLout = from VL to outside; Mixed effects, see Suppl Text 10). Compared to the intrinsic strong connections, the outside ones were significantly (mixed effects, see Suppl Text 11) less frequently ‘maintain’, and more frequently ‘appear’, indicating more segregation of the two delays in extrinsic connections (compare Figs. 5D and S3-7). The NS1 component of strong outside connections displayed two important properties: synchrony with the TP activation (onset latencies of the 4 strong outside uCG were on average 2 to 13 ms earlier than those of TP activation) and clear task dependence (Figs. 5A, S5-5, S5-6).

Just over a quarter of outside cortical leads were positively and strongly connected with the T/VL parts of TP in either direction: median proportion inward (26%, range 16–47%), and median proportion outward (26%, range 22–40%). Interestingly, although average values were very similar in the two directions, the proportions in the two correlated little across patients (R² = 0.09). This indicates that the uGCs reflect functional links and not the implantation pattern idiosyncratic of each patient. The outside leads connected with TP were located in all parts of temporal and frontal cortex, including OFC and insula (Fig. 6A, B). OFC stood out as the region most connected with TP (Tables S7, S8): median proportion inward reached 83% (range 33–100%) and median proportion outward 67% (range 33–100%), both effects being significant (mixed effect S Text 12, 13). In both directions, ventral temporal cortex (TC) was the second most connected region, and dorsolateral frontal cortex was the least connected region, all these effects being significant (mixed effects Suppl. Text 12, 13). Both the inward and outward external leads could be connected to both VL and T, or to only one of those regions. Proportionally fewer leads were connected only to VL amongst the output leads compared to input leads (Mixed effects, see Suppl Text 14), indicating that T dominates the output from TP. The temporal leads providing input to TP were segregated into dorsal and ventral regions, connected predominantly with T and VL, respectively (Fig. 6A, Tables S9, S10). Both effects of functional connectivity were significant, see mixed effects S Text 15, 16. While the ventral temporal (VT) leads connected with TP in the input direction were predominantly linked to VL, those VT leads connected in the output direction with TP were predominantly linked to T (Table S11), again a significant effect (mixed effect Suppl Text 17). Outside cortical leads negatively connected with TP showed a more restricted spatial pattern than positive leads (Fig. S6-1), but they were observed predominantly in a single patient (P1). Overall, both the positively and negatively connected leads responded in the gender task in a task dependent way, but response strength and task effect were smaller than in TP leads (Fig. 6C-E, S6-2, S6-3). Also these outside leads responded on average synchronously with the TP leads but individual leads showed large variation in onset latency (Fig. S6-4). Two VT regions providing input to VL are noteworthy: FFA in fusiform cortex (black arrow in Fig. 6A), and a region in lingual gyrus, labelled LG1 (green arrow in Fig. 6A). In agreement with⁹, the FFA leads (2 patients, Fig. S6-5B, C) responded with short latency to the static presentation in the two tasks equally. The LG1 leads (4 patients, Fig. S6-5A–D) showed a weaker and later response that was weakly task dependent. Across patients the latency of LG1 responses varied little with RT, unlike the VL leads (Fig. S6-5E, F). Remarkably, the VLin uGCs from both LG1 and FFA showed strong task dependency (Fig. S6-6A-D), and the LG1 NS1 uGCs correlated significantly (r = 0.73, p < 0.005) with the power of LG1 leads, unlike overall VLin NS1 uGCs (Table S5). The NS4-uGCs from LG1 were variable, ranging from absent in P1 to very strong in P3 (Fig. S6-7A-D), and differed from those in the other direction (Fig. S6-7E–H). The NS4-uGCs from FFA also differed in the two directions (Fig. S6-7F, G).

What determines the TP gender decision responses?

While the outside leads sending signals to TP are responsive and task dependent (task dependency was significant for leads with positive or negative, inward or outward connections, mixed effects, see Suppl Text 18), they are less so than the TP leads (Fig. S6-2, S6-3). On the other hand, the strong uGCs linking these leads to TP show clear phasic, task dependent rises (Figs. 5A, S-5, S6-6). Hence the unfolding in time of TP responses seems to reflect largely the temporal properties of the functional connections, in particular the short delay uGCs. These latter were anatomically specific (Fig. 5E): both for VLin and Tipin leads the connection to Ds (3 patients) were reduced to less than 30% of those to VL or T (Fig. S6-8)) and depended little on stimulus duration (Fig. 5F). Also, the latencies of VLin NS1-uGC and of VL power correlated significantly (r = 0.95, p < 0.02, Fig. 7A), as did those of Tipin NS1 uGCs and T power (r = 0.99, p < 0.01, Fig. 7B). Further analysis confirmed the importance of the VLin NS1-uGCs for the time course of VL power. First, the proportion of Vlin (r = − 0.92, p < 0.05) but not Tipin NS1-uGCs (r = − 0.53, p > 0.3) correlated negatively with RTs across the five patients (Fig. 7C, D). This effect of VLin proportions implies an effect on VL response latency (which correlates with RT), and explains the difference between VL and T in the equations linking uGC and TP response latencies (Fig. 7A, B): the strong negative intercept in the VLin-VL latency equation (latency VL = − 78 ms + 1.45 latency VLin NS1 uGC) indirectly reflects the effect of the proportion of NS1 uGCs, which decreases with latency, given a slope exceeding one. This effect of the proportion of VLin NS1-uGCs on latencies of VL responses, by the negative intercept, also sheds some light on the paradoxical observation that for patients with short RTs (patients 1, 2, 4), the VL latencies are shorter than those of the VLin leads. The second support for the role of VLin timing on TP responses comes from the effects of stimulus duration on the duration of TP responses and of NS1-uGCs increases. In agreement with⁹, when considering the five stimulus durations, the duration of VL responses hardly (< 10%) increased with stimulus duration, an effect mainly due to a single patient (Fig. S7-1). This independence of VL responses from stimulus duration was confirmed when dividing the trials into long and short halves: across patients duration of VL responses for long trials depended weakly (shallow slope and small r) on duration for short trials (Fig. S7-2). A very similar relationship (with the intercept 20 ms shorter) was observed for the VLin NS1-uGCs (Figs. 7E, S7-3), but not the Tipin NS1-uGCs (Figs. 7F, S7-3) nor Tip responses (Fig. S7-2). These Tip inputs and responses both exhibited a slope close to 1 and large r, reaching significance for Tip responses (r = 0.97, p < 0.01), setting them apart from the VL inputs and responses. It is noteworthy that the VLin lead responses also showed a relationship with near 1 (0.92) slope and large r (0.85, p > 0.05), further underscoring the differences between VL and VLin lead responses. When considering the duration of responses for short and long stimuli separately, yielding 10 data points across the 5 patients, the duration of VL responses correlated significantly (r = 0.81, p < 0.005) with that of VLin NS1-uGCs (Fig. 7G). This correlation, however did not hold (r = 0.42, p > 0.2) for the relation between T responses and Tipin NS1-uGCs (Fig. 7H), again setting Tip inputs and responses apart from VL inputs and responses. Thus not only does the VL response time course match that of the VLin NS1 uGCs, but this match between afferent connections and response sets VL apart from Tip.

Control of the inputs to TP

If the timing of the VL responses reflects that of the short delay VLin uGCs, this calls for a region controlling the onset of these functional links. As VLin and VL leads are largely synchronous (Fig. S6-4), this control region should provide input to VL in parallel to input to Vlin leads, and thus belong to the VLin leads, in addition to being connected to the other VLin leads and being task dependent. The OFC VLin leads, located mainly in the rostral part of right OFC (Figs. 8A, S8-1C), as those responsive in the gender task (Fig. S8-1A), were indeed connected to most other VLin leads (Figs. 8B, S8-2, S8-3). In addition, the OFC leads responsive in the gender task, are known to be task dependent⁹. OFC VLin leads met two further requirements for a region controlling VLin NS1 uGCs’ timing: (1) their uGC to VL (Fig. 8C, arrow 1) had a significantly shorter onset latency (Fig. 8E, mixed effects, see Suppl Text 19) than the uGCs from the remaining VLin leads (arrow 2 in Fig. 8C), (2) the NS1 uGC from the OFC-VLin leads to the other VLin leads (Fig. 8C, arrow 3) had a shorter onset latency than their VLin NS1 uGCs (Fig. 8C, arrow 2): indeed their difference was significantly different from zero (Fig. 8F, mixed effects, see Suppl Text 20).

One final issue concerns the intrinsic VL-to-T uGCs, which were the earliest event we observed in the right TP. These functional links might arise from a small group of VL neurons that escape detection by the coarse LFPs¹⁴, and receive FFA input, as indicated by some FFA-VL uGCs having early onset latency (Fig. S6-6H). These ‘early’ VL neurons may then activate the OFC leads conditioning the VL-to-T connection (Fig. 8C), which overlap with the OFC-VLin leads (Fig. S8-1D). In one patient these OFC leads indeed showed face responses (Fig. S8-4), and their location is consistent with that of the lateral OFC face patch in the monkey¹⁵.In addition to these short delay uGCs, the ‘early’ VL neurons may also drive the long delay VL to T connections (Fig. S3-6B).

Discussion

Our results showed that (1) the T and VL parts of right TP were most responsive in the gender task, but VL responses correlated with the RT of the gender task. These results were unaffected by the location of a substantial number of leads in the EZ. (2) The TP responses are based on visual signals from both the face and the hand of the actor. (3) The VL and T parts are linked in both directions with the VL to T flow earlier and under more external control than the stronger T to VL flow. (4) While the cortical input to TP targets mainly VL, originating predominantly in OFC and VT cortex, output arises more from T, returning to a range of outside regions, including OFC and VT cortex. (5) The time course of the TP responses reflect more the timing of the afferent links than the time course of the responses in afferent leads.

Initial studies of human temporal lobe responses to faces were investigated with event related potentials¹⁶, but subsequent studies used a variety of gamma band ranges: 50-150 Hz¹⁷, 30-70 Hz¹⁸, 30-100 Hz¹⁹, 40–90 Hz²⁰. The latter authors reported FFA responses to faces with a time course very similar to the one reported here. Interestingly they failed to find a relationship between the FFA gamma band power and behavioral measures in a gender discrimination task similar to ours. We used the broad band gamma range 50–150 Hz because of its relationship with neuronal activity (see²¹). Using the broad band gamma range, the present gender task activates not only FFA but also TP⁹.

The computation of Granger causality^12,13 between pairs of stereo-EEG leads allowed us to trace the functional connections between these recording sites. Granger causality computation yielded both strong phasic and tonic connectivity within right TP, but in the present study we concentrated on the phasic changes in an attempt to understand the remarkable properties of the phasic TP responses in the gender task⁹. Importantly, this computation allowed to trace also the direction of these transient connections and their time course, the latter being impossible with fMRI^6,7 given the long time constant of the hemodynamic response. Indeed we were able to show that the connections between VL and T differ significantly in strength and latency in the two directions (Fig. 3). These findings are in agreement with those of single cell and electrocorticographic studies in non-human primates^{22,23,24,25,26}. Furthermore we computed the unconditioned GC for two delays (4 ms and 16 ms), which proved to be approximately two independent measurements. Indeed there was little correlation between the uCG for the two delays in the intrinsic and outside connections of TP in either task (R² ranging from 0.06 to 0.25). In addition, the conditioning leads for the VL-T connections rarely matched for the two delays. These two delays of the uGC imply functional connections operating on a very short time scale. Yet, previous intracranial evidence of multi-stage temporal processes underlying visual cognition^27,28 demonstrated feedforward and backward dynamics with ~ 100 ms separations between activations in distant areas. This is different from the present study, in which we used Granger causality to investigate functional connectivity between two simultaneously active, neighboring TP subregions (T and VL), which, in addition, are active only very briefly. Hence it is not surprising to find 4-16 ms delays for the functional connections between these subregions, as the effects of the functional connections have to fit within the brief simultaneous activations. Furthermore our results are in line with single cell data from non-human primates. Indeed, consistent with the 8 to 13 ms delay obtained for the coupling of gamma frequencies in single cell studies of attention effects in V4 and FEF²², the uGCs at 16 ms delay may correspond to connections with more distant regions as shown by the location of the conditioning leads for NS4 uGCs between VL and T (Fig. 4). But this is not necessarily always so, since strong NS4-uGCs were measured between the two neighboring regions VL and T. The explanation for this apparent paradox resides in the fact that these uGCs do not necessarily indicate direct connections. In single cell studies it has been argued that frequency-resolved granger causality only applies to direct connections²⁹, but it is unclear whether this generalizes to our study using time resolved granger causality applied to local field potentials. In absence of anatomical information, as was the case here, a direct connection can be inferred only if conditioning of the GC between a pair of leads is independent of the activity of all leads outside the pair, which is impossible to assess with certainty given the limits to the within-patient sampling of recording sites in this study. If the leads with conditioning effect are located in the same region as the source of the uGC, as we observed (Fig. 4), this could be considered a convergent projection from the source region, implying a direct anatomical connection between the two regions in which the leads of the conditioned pair are located.

The proportions of leads responsive in the gender task confirm that right TP is a heterogeneous structure^5,6,7, with the VL region, in which more than half (58%) of the tested leads were responsive, being the entry stage for visual signals, as it occupies the most rostral part of TE. The Md subregion includes rhinal cortex which is known to host many visually responsive neurons²⁴, consistent with the relatively large proportion of responsive leads (42%) observed in this subregion. Anatomical connections^5,6,7 indicate that in addition to VL, Ds and some parts of Md are also unimodal subregions, linked to specific sensory systems, while T, receiving from all 3 unimodal regions⁶, is multimodal^3,30. As semantic categories are multimodal, they should therefore be represented in T. Since the functional link VL to T is the first event in TP, it follows that category labels in T are activated by the visual evidence collected by VL in the signals carried by the afferents from the temporal VLin regions. The masking experiments showed that these VL signals arise, be it with different latency, both from the actors’ face and body, which remain segregated at lower levels^31,32. This combination of face and body signals in human TP is consistent with our earlier prediction³³, but difficult to compare with the face patch in monkey TP, where neurons are selective for familiar faces but not familiar bodies (³⁴, Fig. S2-1). Consistent with the masking results, the temporal VLin regions included not only FFA, but also lingual regions processing color and texture information^35,36,37,38. Hence, the visual evidence collected by VL concerns both the geometry and material properties of face and body, i.e. the physical appearance of the actor. The lingual regions processing color and texture had a longer and more variable latency than FFA leads, consistent with the increase in latency of the TP responses when masking the face. While the location of the VLin leads informs on the nature of the signals reaching VL, the time courses of VL responses reflect more those of the VLin connections than those of the VLin leads: latency, duration, strength of response and its task dependency all are more similar between VL leads and short delay VLin connections than between VL and VLin leads.

Our results also clarify the efferent projections from multimodal region T, the connectivity of which has been difficult to study with imaging^3,4. Indeed, they indicate that the output from TP arises mainly from T or from T and VL combined, i.e. the regions representing semantic categories. Indeed T and VL are known to share many functional connections⁶. According to our results these TP outputs target a range of regions (Fig. 8D): (1) the VT cortex which are the sensory regions providing input to VL, i.e. the spokes of the model, consistent with⁸, and including feedback to the VLin leads (Fig. S8-3, Tables S10, S11); (2) dorsal temporal cortex i.e. possibly representing the lexicon, a path useful for naming tasks³⁹; (3) the PFC and OFC control regions (although to a reduced number of leads compared to the TP input leads, Fig. S8-4, Table S7, S8) to adjust the control⁴⁰, and (4) to inferior parietal lobule (IPL) decision regions⁴¹ for the semantic decisions in the present task. Although both T and VL provide output to rostral IPL (Fig. 6B), the significant correlation between response latency in VL and RT suggests that the visual evidence collected by VL is the main source of information for the semantic decision.

Jackson et al.⁸ investigated mechanisms for semantic memory by comparing a number of model architectures in terms of the degree of (1) conceptual abstraction they achieved, that is the ability to discern conceptual similarity among items denoted by images, words or other attributes, and (2) context sensitivity, in which a context-specific subset of properties of a concept needed to be generated, both of which are critical components of the task our subjects performed. They find that the best-performing model had a multimodal hub (their hidden layer 2 – HL2), that received input from multiple modality specific hubs (hidden layer 1 – HL1) that each integrate over spokes of modality-specific brain regions, some of which also sent and receive shortcut connections to the multimodal hub (these thus bypass HL1). Our findings concerning the direction of information flow within and through TP fit well with this model. Indeed, the VLin-VL-T sequence matches the optimal hub and spokes architecture⁸, with VLin leads corresponding to the spokes layer, VL to the hidden layer 1 (HL1), and T to the multimodal hidden layer 2 (HL2), with the short cuts between spokes and HL2 being instantiated by the few ventrolateral temporal leads connected directly to T (Fig. 6A). It is noteworthy that the transition from unimodal to multimodal TP regions corresponds to cytoarchitectonic distinctions⁵ between cortices with a thick layer 4 (VL) and thin layer 4 (T). Our results, however, clarify the model⁸ in several ways, specifically:

First, it proposes a solution to the problem of a small hub having to represent many categories. Our results suggest that this is solved by T (HL2) using VL (HL1) as annex for the category representation. Indeed, if many of the categories have to be localized in T, the category representation must be very reduced, limited to an abstract label, i.e. a simple index. Given the many T leads responding in a task using only very basic categories, the category label is likely distributed over large numbers of neurons, unlike what has been reported for declarative memory⁴². The automatic return activation of VL by the strong T to VL connection suggests that the abstract category label in T is strictly linked to the main visual category features in VL, possibly corresponding to the defining features of categories¹⁰. The link from T to the spokes (see above) may be more flexible, possibly emulating the characteristic features of semantic categories¹⁰. Thus T corresponds to the semantic hub in the strict sense, and its index representation solves or alleviates the problem of hosting many categories into a relatively small region, even if population codes can be computationally very efficient⁴³. Such a representation may also facilitate the establishment of links between related categories. T receives input not only from VL and PFC/OFC control regions, but also from dorsal TC, which may provide input from the lexicon⁴⁴, thereby establishing a one to one relation between abstract category labels and words. Thus categories could be activated by a different path than the physical appearance, provided by the VLin leads to VL⁴⁵.

Second, our results also clarify the control over the TP. Indeed, while the input from ventrolateral TC provides the sensory evidence collected by VL, the timing of this input is set by VLin-OFC, which controls the VLin connections. This sets the time course of activity in the whole TP, as the direct input from outside leads to T does determine its latency but not its duration (Fig. 7). Thus, OFC, the only region outside TP hosting double selective leads in⁹, and directly connected to the TP by the uncinated fasciculus⁴⁶, has an extensive control over the collection and transmission of sensory evidence to the T. Importantly, most OFC leads are in the most rostral part of OFC, an evolutionarily novel part, the function of which has so far remained unknown⁴⁷. With respect to model suggestions⁸, it should be noted that the VLin-OFC controls the link between VLin leads and VL, i.e. the link spokes-HL1 in the model, rather than one of these layers, which were the optimal location for connections with the control regions according to the model⁸. In addition, the VL-T link (corresponding to the link HL1-HL2 in the model) is also under OFC control (quantified as a conditioning effect) and T itself receives input from PFC/ OFC (Fig. 6A), indicating that the link HL1-HL2 in the model is also under control of outside regions. An anatomical substrate for such outside control has recently been described for layer 6 of T⁴⁸. Thus, our results enrich control mechanisms by indicating the presence of control at multiple levels which targets both the links between and within TP subregions (i.e. layers in the model). This is not surprising as in the modelling study⁸ the demand for control was limited, serving only to select the relevant sensory feature, while here control sets the timing of TP.

In conclusion, our study of the responses of stereo-EEG leads and their functional connections reveals the directional flow within the right TP leading to a suggestion of how the gender categories are represented by an abstract label in area TG, the multimodal part of TP, and by category features in anterior TE, its visual part. By extending the study of directional flow to the outside connections of right TP it reveals (1) the use of physical appearance as input to the TP, underscoring the importance of the senses for concrete knowledge, and (2) the control of OFC (and ventrolateral PFC) over the different processing stages feeding this input into TG, extending to functional role of OFC beyond contributing to slow cognitive control⁴⁹. Finally our results also show that the fast processing in high level cognitive regions such as TP, is determined by the timing of the functional connections rather than that of the afferent areas, providing a mechanism for the human brain to operate on autonomous time scales. Further work is needed to extend the present results to more concrete semantic categories, to left TP, and to abstract and verbal categories.

Methods

Patients

Stereo Electroencephalography (stereoEEG) data were collected from 19 patients (8 female, age 21–49, mean 32 years, Tables S1 and S2) suffering from drug-resistant focal epilepsy. Fourteen of the 19 patients contributed to the previous study⁹. These patients were stereotactically implanted with intracerebral electrodes as part of their presurgical evaluation, at the Claudio Munari Centre of Epilepsy Surgery, to define the cerebral structures involved in the onset and propagation of seizure activity^50,51. The strategy of implantation was based on the presumptive location of the epileptogenic zone (EZ), derived from clinical history, examination of non-invasive long-term video-EEG monitoring, and neuroimaging. Patients were fully informed regarding the electrode implantation and stereoEEG recordings. The present study received the approval of the Ethics Committee of Niguarda hospital (ID 939–2.12.2013) and informed consent was obtained from all patients in the study. The experimental testing of the patients was performed during the pre-surgical evaluation in accordance with the Italian and European guidelines. The two subjects shown in Fig. 1A, B and S2-1 have given their informed consent for publication of identifying information/images in an online open-access publication.

Inclusion criteria

As described in⁹ patient selection was based on a series of stringent anatomical, neurophysiological, neurological and neuropsychological criteria, with the specific aim of minimizing the recording of any data from pathophysiological and functionally compromised sectors of the brain tissues.

· Anatomical criteria: only patients whose magnetic resonance imaging (MRI) showed no anatomical abnormalities, or only very restricted anomalies were included in the study.

· Neurophysiological criteria: this examination includes the inspection of the EEG tracks at rest, during wakefulness and sleep from both intracranial and scalp EEG. Pathological activity is characterized by the presence of epileptic discharge at the seizure onset, but epileptic spikes may be present in leads exploring the regions surrounding the EZ during the interictal periods. Since epileptic spikes could affect the quantification of task- and stimulus-related gamma reactivity, each trial presenting an interictal epileptic discharge (IED) at any latency during the stimulus presentation was removed. Besides inspecting the spontaneous EEG activity, the neurophysiological investigation also included an assessment of the normal reactivity of both intracranial and scalp EEG to a large set of peripheral stimulations (somatosensory, visual, vestibular and auditory stimulations) to verify normal conduction times and overall sensory reactivity.

Neurological criteria: no seizure, no alteration in the sleep/wake cycle, and no change in pharmacological treatment must have taken place within the last 24 h before the experimental recording of a patient included in the study. Neurological examination had to be unremarkable, with in particular no motor or visual deficit.

Neuropsychological criteria: a series of neuropsychological tests was administered by experienced neuropsychologists. The tests focused on the evaluation of the patient skills in language (production, comprehension, reading), verbal memory, visuospatial memory, visual exploration, executive and attentional functions, visual perception, abstract reasoning. Among them, we considered of particular relevance 5 items indexing skills which could impact the ability to carry out the required tasks:

Semantic fluency⁵²: the overall score is followed by the ranked index, where a value greater or equal to 2 indicates normal function, a value of 1 indicates a subclinical abnormality, a value of 0 indicates a pathological dysfunction.
Naming: this is a test item extracted from the Boston Naming Test⁵³; a score below 20 is considered pathological.
Visual exploration⁵⁴; a score below 30 is considered pathological.
Attentional matrices⁵² ; the overall score is followed by the ranked index, where a value greater or equal to 2 indicates normal function, a value of 1 indicates a subclinical abnormality, a value of 0 is considered pathological.
Face recognition: Benton Facial Recognition Test⁵⁵. A value outside the normed range (41–54) is considered pathological.

Their values, which were normal in the vast majority of patients and tests (70/78), are reported for 16/19 patients in Table S2; 3 patients were abnormal for semantic fluency, 3 also for face recognition.

Localization of recording sites with respect to lesions and epileptogenic zone

Only patients presenting either no anatomical alterations (n = 15) or only small abnormalities (n = 4), as evident on MRI, were included. Three of the 4 patients with positive MRI showed minimal periventricular nodular heterotopia, and one patient focal cortical dysplasia (FCD) in the frontal lobe.

The epileptogenic zone (EZ) involved generally parts of temporal or frontal cortex. Its extent was established in 18 out of the 19 patients and the overlap with the four TP parts was determined.

Electrode implantation

All implantations in the patients considered were made in the right hemisphere: most (16) had unilateral implantations, but 3 had bilateral implantations. A number of depth electrodes (range: 12–21) were implanted in different regions of the hemisphere using stereotactic coordinates. Each cylindrical electrode had a diameter of 0.8 mm and consisted of eight to eighteen 2-mm-long contacts (leads), spaced 1.5 mm apart (DIXI Medical, Besancon, France). Immediately after the implantation, cone-beam computed tomography was obtained with the O-arm scanner (Medtronic) and registered to preimplantation 3D T1-weighted MR images^56,57. Subsequently, multimodal views were constructed using the 3D Slicer software package⁵⁸, and the exact position in the brain of all leads implanted in a single patient was determined by using multiplanar reconstructions⁵⁹. Leads were identified, following clinical conventions, by a letter corresponding to the electrode shaft, followed by a number starting from the electrode tip. The number of leads localized in the grey matter of the right hemisphere ranged from 84 to 171 (Table S1).

Behavioral testing

Behavioral testing of the patients was exactly the same as described in⁹.

Setup

Patients were seated 70 cm from a liquid crystal display (Dell P2210, resolution 1680 × 1050 pixels, 60 Hz refresh rate) in a familiar environment. The visual stimuli were generated using a personal computer equipped with an open GL graphics card using the Psychophysics Toolbox extensions^60,61 for Matlab (The Math Works, Inc.).

Visual stimuli and tasks

The stimuli consisted of 1.167 s videos clips showing one of two actors (male or female), standing next to a table and dragging or grasping an object (a blue or red ball) on the table using the right hand. At the start of the video, the hand could be shaped either as a palm or a fist and its position could be either above or on the table. In half of the trials, we increased the size of the video (by 20% of the original) within the aperture. The aperture was created by multiplying videos with an elliptic mask causing video clips at the borders to gradually blur into the black background. Finally, the videos were shown as recorded (actor standing to the right of the table), or mirrored around the vertical axis (actor on the left side of the table). These manipulations resulted in 64 (2⁶) videos which were then presented either in the full length (Full trials) or truncated either at the time point corresponding to each individual’s 84% action discrimination threshold (ranging from 200 to 250 ms) or at one of 2 other points 100 ms earlier or later (Truncated trials). The remainder of the movie in the truncated trials was replaced by a dynamic noise which was produced by randomly scrambling every pixel in the display on subsequent frames.

All trials started with a baseline period (1 s), followed by a variable static phase, created by repeating a first video frame, identical for the two actions, for 283, 458, 583, 733 or 883 ms, and then followed by the video displaying either action. If patients could not respond during static and dynamic stimulus presentation, they were given another 2 s to reach a decision before the trial ended. As soon as patients pressed a button during any of the 3 trial phases (static frame, video, response epoch), the inter-trial period (1 s) started. In the analyses of the effects of the duration of static presentation, trials with 283 and 458 ms static presentation were considered short trials, those 733 ms and 883 ms with static presentations as long trials.

The trials were organized into 4 blocks of action (An) or gender (Ge) discrimination tasks⁹. The order of presentation was always An-Ge-An-Ge. Each block consisted of 2 sub-blocks of 32 trials such that one sub-block contained full trials and the other truncated ones, presented in pseudorandom order. At the beginning of each sub-block, the instruction saying either ‘action’ or ‘gender’ in Italian was displayed for 5 s. The patients had to follow this instruction by performing either the action or gender two-alternative forced-choice (2AFC) discrimination task by pressing, when ready, either a right or left button with the right hand to indicate their decision. In the first 2 blocks the original videos were displayed and in the last 2, we mirrored the videos moving the actor to the opposite visual field. During the trial, a fixation cross was presented near the manipulated object in the center of the screen. During the 1 s inter-trial interval, only the fixation cross was visible.

Patients were instructed to fixate the cross in the center of the screen. In all subjects the experimenter verified that subjects complied with the fixation instruction. Eye movements were recorded in 13 patients using a noninvasive monitor-mounted infrared video system (SMI iView X 2.8.26,) sampling the positions of both eyes at 500 Hz. The measurements confirmed that patients followed the instructions to fixate well and that fixation accuracy was similar in the action and gender tasks⁹.

Preliminary procedures

To familiarize patients with action and actor discrimination tasks, we presented them with 2 familiarization blocks of 30 full trials chosen pseudo-randomly such that they contained equal numbers (15) of the 2 tested actions and the 2 tested actors. Patients responded by pressing a button at the end of each trial and received an auditory feedback indicating either a correct (with low pitch tone) or an incorrect (with a high pitch tone) response. The procedure was repeated until patients made fewer than 2 errors per block. Patients first learned which button corresponded to which gender and then which button corresponded to which action. In addition, every test block was preceded by a short (10 trials) familiarization block reminding patients of the proper button-choices for an upcoming discrimination task.

After completing familiarization blocks, patients viewed a block of 30 pseudorandomly chosen trials in which videos were truncated at 3 different time points (150, 250 and 350 ms) from the motion onset with the end of the video replaced by dynamic noise. After collection of the action discrimination performance, the 84%-threshold was estimated for each patient and later used in the experiment to create trials in the truncated condition.

Masking

In four patients, the full trials of the gender task were repeated with either the face or the hand covered by a grey rectangle 8.2° × 4.1° in size for the hand and 5.5° × 6.1° for the face (Fig. S1-2). Two blocks of 64 trials each were collected: first block with the standard video and second block with the mirrored video. Each block contained two sub-blocks of 32 trials in which 16 non masked and 16 masked trials are interleaved: first sub-block hand masked, second face masked, third face masked, fourth hand masked. Thus 32 trials with hand or face masked could be compared to 64 trials without masking.

StereoEEG data recording

For each implanted patient, the initial recording procedure included the selection of an intracranial reference, chosen by clinicians using both anatomical and functional criteria. The reference was computed as the average signal of two adjacent leads both exploring white matter. These leads were selected patient-by-patient because they did not present any response to standard clinical stimulations, including somatosensory (median, tibial, and trigeminal nerves), visual (flash), and acoustical (click) stimulations, nor did electrical stimulation evoke any sensory and/or motor behavior²¹. The stereoEEG was recorded with a Neurofax EEG-1100 (Nihon Kohden System) at 1 kHz sampling rate.

Standard statistical analysis

The recordings from all leads in the gray matter were filtered (band-pass: 0.08–300 Hz; notch: 50 Hz) to avoid aliasing effects and were decomposed into time–frequency plots using complex Morlet’s wavelet decomposition⁹. Power in the gamma (50 to 150 Hz) frequency band was extracted within an interval, extending from 1000 ms before the start of the trial to 1000 ms after the latest response by the subject⁹, the timing of which differs between subject and task. This interval was subdivided into non-overlapping 25 ms bins. Following previous intracranial studies^9,62, gamma power was estimated for 10 adjacent non-overlapping 10-Hz frequency bands, and averaged. The quality of the data was visually inspected using plots of average gamma power in all trials collected for a given condition, to detect the possible presence of IEDs. All trials/leads in which any IED or other transient electrical artefacts appeared were removed.

The anatomical reconstruction procedures followed those of²¹ and included two basic steps: 1) identifying the recording leads located in the individual cortical surface using the multimodal reconstructions performed in each patient, and 2) importing these locations into a common template, using the warping of the individual cortical anatomy to the fs-LR average template.

Behavior data of the full trials were analysed as in⁹. This previous study established that the performance on the two tasks was extremely similar for the long trials. Accuracy (% correct) and reaction times (RTs) were computed for each of the 19 patients in the gender task (Table S2). Accuracies were high with a median of 98% (range 84–100%), and because of two patients with low score on face recognition, there is a correlation between accuracy and face recognition score (r = 0.72, p < 0.005). Reaction times were generally short (less than 1 s) with a median of 832 ms. Two patients had an RT exceeding 2 s, and those were removed from the analysis of the relationship between response latencies of leads and RTs. The remaining RTs ranged from 547 to 982 ms.

In this part of the present study only the full trials, 128 in total (64 trials for each task), were analyzed for responsiveness and specificity of the leads, in following of⁹. The analysis was performed on the average gamma band (50–150 Hz) power sampled with 25 ms bins, and z-scored against the 1 s baseline period⁹. Each trial was subdivided into three epochs: (1) the 1 s baseline epoch (BEp), ending with static stimulus onset; (2) the static epoch (SEp), defined as the 200 ms time window starting 75 ms after static onset (275 ms was duration of the shortest static phase); (3) the video epoch (VEp), which started at the end of the static presentation (variable across trials) and lasted until a patient gave a response. Responsive and specific leads were defined exactly as in⁹, using the distribution (mean and SD) of z-scores during the 1 s interval before stimulus onset as reference. Responsive leads had a z-score exceeding a threshold of 3SD during the static epoch in the gender task. Specific leads in addition did not exceed a z-score of 3SD in the static epoch of the action task (task specificity), nor in the video period of the gender task (epoch specificity). In the present study, responses were compared in different masking conditions. Since only 32 trials were sampled for each of the two masking conditions, the z-scored gamma band time courses of the responsive leads in T and VL were averaged in each patient: 6 leads in patient 1, 5 in patient 2, 3 in patient 3, and 5 in patient 7. Latency and duration of these average time courses were defined by the intersection with z-score of 2.

Granger causality analysis

Patient inclusion Granger analysis

We selected 7 patients (Patients 1–7 in Table S1) with most recent recordings or large numbers of responsive leads in TP, particularly in subregions VL and T, to investigate the directional flow of information within TP and between TP and outside cortex. Since Granger causality depends on the time course of individual trials it is much more prone to artefacts than the average power in the 50–150 Hz band used in the standard analysis. Therefore we developed a specialized artefact rejection procedure (see below) which resulted in a larger fraction of rejected trials compared to our standard analysis. Therefore we included in the analysis the full trials as well as the truncated trials for both tasks, doubling the number of trials available for analysis. This is acceptable because the truncating of the videos had little effect on the behavior in the gender tasks, in which patients relied on the static part of the stimulus⁹. Indeed for the 7 patients the median accuracy was 98% in truncated trials compared to 100% in full trials. The median RT was 696 ms for truncated compared to 832 ms for full trials. Also the TP responses to the static stimulus in the truncated trials of the gender task were similar to those in the full trials⁹.

Preprocessing for Granger analysis

The recordings were cut in segments aligned to 1 s before static onset and lasting at least until 1 s after static onset (depending on the onset of the following trial). Since the TP response in the gender task started on average 170 ms after static onset and lasted on average less than 200 ms, the segment included all of the TP responses. For each patient there were between 81 and 171 leads located in gray matter of the right hemisphere, for which 128 trials in the “gender” and “action” task were available (256 trials total). For the initial standard analysis, which was conducted to extract the onset and task-specificity of the response and focused on the high-gamma power, artifact rejection was based on outliers in power between 50 and 150 Hz (see above). For the further analysis that involved the interaction between leads evaluated through Granger causality this was inadequate. We were interested in effects that were induced by stimulus onset and designed criteria to remove other large-amplitude activity that was not locked to onset. We identified the following artifacts: (1) large amplitude low frequency and gamma band events; (2) large-scale activity correlated between leads; (3) lead saturation and glitches where there was a coherent jump in recorded values across leads. We therefore developed a multi-pronged procedure, implemented in Matlab (the mathworks, version R2017b, R2019b, R2021b), because each criterion by itself detected only part of the artifacts we had identified by eye. We applied each criterion to the data (see below) and accordingly set the traces for specific trials and leads (or channels) to NaNs (not a number) so as to automatically exclude them from further analysis in the matlab routines. For each patient there were on the order of ten thousand traces to be manually inspected per task condition, which was not feasible and would also not lead to reproducible results. We therefore developed a pipeline that is characterized by a few heuristic parameters that are set separately for each patient, by evaluating the trials around the decision boundary between those that are kept and those that are discarded. This ensures reproducibility because the same setting of parameters leads to the same selection of trials/leads. The used parameters are listed in Tables 2, S12 and S13. Note that initially, we considered the action and gender trials separately in the outlier analysis (Patients 1–3, 5–6), but found that it was more efficient to combine them when the pipeline matured (Patients 4 and 7). The pipeline included the following steps.

Table 2 Global performance of artefact rejection for Granger Causality computation.

Full size table

Feature-vector outliers

The data (each lead, each trial) were first detrended using the Matlab routine detrend in time by subtracting the best fitting linear function. The power across the entire trial epoch was determined using the routine pwelch, and summed across the high gamma band (50-150 Hz), this statistic was combined with the standard deviation across time into a two-component feature vector. Elements with arbitrary low values (< 1E-42) were removed before applying on the log-values the routine mcdcov from the LIBRA toolbox (https://wis.kuleuven.be/statdatascience/robust/LIBRA,⁶³) for each lead separately. We used log-values in order to obtain a distribution that was more similar to a multivariate Gaussian. The routine first calculates robust estimates for the mean and covariance via the minimum covariance determinant method, and then outputs the corresponding Mahalanobis distance for each sample (for a given lead)⁶³. For a bivariate Gaussian distribution of feature values, 97.5% of the samples lie below a Mahalanobis distance of 2.7162. For each patient, we set a percentile above which the trial-lead combinations would be rejected, this was implemented by finding samples that exceeded the corresponding Mahalanobis distance threshold. Across patients the percentile was a parameter that ranged from 90 to 99 (BivariateRejectPercentile in Table S12).

Wavelet calculation

The continuous wavelet transform (cwt) was applied using the matlab routine cwt with default settings on the time series at full resolution, i.e. 1 ms sampling interval. Specifically, we used the analytic Morse wavelet⁶⁴ with the symmetry parameter (gamma) equal to 3 and the time-bandwidth product equal to 60. For each octave ten frequencies were sampled. The resulting waveforms were smooth and were down sampled by a factor 10, hence time samples were spaced 10 ms apart (matlab function downsample). For each lead and trial, we extracted blobs in the frequency-scale x time sample image, representing transient oscillatory events with a limited duration in a limited frequency band. Only scales corresponding to frequencies higher than 10 Hz were considered. First, we took the absolute value (the wavelet data is complex), and then binarized the image using a threshold that corresponded to the 95^th percentile (matlab routine imbinarize), after which holes, that is, lower intensity pixels inside a blob, are filled through a call to imfill, with option ‘holes’. The so called connected components (the blobs) are then individually labeled through bwlabel, with the parameter connection number equal to 8 (i.e. we consider nearest and next nearest neighbors pixels to form the connected component). Via a call to regionprops, we retrieve for each connected component its Area, Centroid, BoundingBox, MeanIntensity, MaxIntensity. For each trial/lead we keep the blob with the largest area, which are then combined into one set across trials/leads. Out of this set we select those whose frequency centroid is above a preset value (parameter: LowerCutOffGlobalHighFreq, Table S12) and whose maximum intensity exceeds a preset percentile of the whole set (parameter PercentileRejectGlobalHighFreq, Table S12). This version of the procedure is referred to as Global, as opposed to Local where we would determine percentiles for each lead, which we had initially explored as well. The corresponding trial-lead combinations so selected are removed.

Putative interictal events

On some trials we detected coherent events that involved a large number of leads, but that were not locked to stimulus onset. We applied the following procedure to identify these trials. For each trial, we extract the lead vs time matrix, z-score each lead, and divide the trial interval up in time windows of length 400 samples (sampling rate 1 kHz, i.e. the window length = 400 ms), and shift these windows with an offset = 50 samples to cover the entire trial interval. For each window and each pair of leads we determine the cross correlation with lags ranging from -150 to 150 samples. We find the maximum of the absolute value and note the corresponding delay. We keep only the maximum value (and delay) across windows. From this data we reconstitute for each trial a lead x lead matrix filled with the value of the maximum correlation, we refer to these as the correlation matrix. We explored a variety of ways to extract lead groups & trials with excessive correlation, and settled on the following procedure. For each trial we construct a histogram with 20 bins of the off-diagonal elements in the correlation matrix. We determine a distance between trials using the optimal transport distance⁶⁵ between the corresponding histograms (c code at https://users.cs.duke.edu/~tomasi/software/emd.htm, ported to matlab via mex file https://nl.mathworks.com/matlabcentral/fileexchange/12936-emd-earth-movers-distance-mex-interface). Briefly, this is the cost to change one histogram into the other one by moving the mass of each density to the other across the shortest distance. The resulting distance matrix is clustered into two groups using fuzzy c-means⁶⁶ implemented in the matlab routine fcm. The smaller (in terms of leads) of the two clusters represented the outliers, which we confirmed by visual inspection. All the corresponding trials were putatively tagged for removal. The removal only took place if the trials also contained above a certain threshold of bivariate outliers (see below).

Trial amplitude outliers

We also applied an alternate procedure to identify trials that could be considered outliers. We took the maximum of the absolute value across all leads (for each trial and time sample), yielding for each trial a time series of maxima. We then again took the maximum across time, yielding a single value for each trial. Using the matlab isoutlier function with default settings we extracted the outlier-trials.

Glitches

We noticed that on some trials all leads in one time step changed coherently up or down, we caught this by calculating the mean across all leads of the absolute value of the difference between two consecutive time samples, for each time point. Subsequently, we extracted the local maxima across time, for each trial, and determined which of them were outliers using matlab isoutlier function with a specified parameter 'ThresholdFactor’ (our parameter GlitchThreshold in Table S13). This was only relevant for three subjects and removed between 3 and 12 trials.

Out of range measurements

We also noticed that some signals were cut-off and replaced by flat (constant) signal during extended periods in the trial. We detect this by labeling as 1 all occurrences where the difference between consecutive samples is less than 1E-10 and count the segments of length winlen, shifted by winstep to cover the entire trial period, in which all entries are one (parameters FlatnessWinstep and FlatnessWinlen in Table S13). All trial-lead combinations where the number of such segments exceed replen are removed (FlatnessRepLen in Table S13). Only one subject (P7) displayed this effect, and 34 trial-lead combinations were removed.

The outcome of the set of two trial-based procedures (Trial amplitude outliers, Putative interictal events) is a union of trials slated for removal. Since removing a trial would entail removing all leads within that trial, hence severely impact the amount of data available for analysis, we added an additional criterion: a trial should contain more than a certain number bivariate outliers in order to be removed (5 to 10, LeadThresholdForTrialBasedRejection in Table S12). All the trials identified by Glitches were removed and combined with the set that survived the preceding criterion.

The outcome of Wavelet calculation and Feature-vector outliers and Out of range measurements is a set of trial-lead combinations obtained from the wavelet blobs, the bivariate gamma-band/standard deviation outliers and out-of-range leads. We took the union of these. Note that some trial-lead combinations were present in more than one set, so the total number trial-lead combinations removed does not match to sum of combinations in the separate sets. Likewise, the completely removed trials also contain trial-lead combinations flagged for removal. The individual number of removed trials, trial-combinations is shown in Tables S12 and S13.

The resulting total number of trial-lead combinations removed is summarized in Table 2. For relatively clean data sets less than 9% of the trial-lead combinations are removed (P1 Gender/Action, Gender for P3 and P5, Table 2. This guarantees that at least 77 trials are available for each lead, for other patients more (up to 30%) trial-leads were removed, which meant that some leads had to be discarded from analyses because too few trials were available.

Lead-based analysis

We performed analyses on the leads by applying a wavelet transform (cwt, default parameters) and averaging leads into the 50–150 Hz band. Note that we recalculated this with enhanced artifact rejection procedure rather than reusing the results of the standard analysis (see above). The wavelet analysis had as downside that time across which responses were integrated depended on the frequency band, which was different from the outcome of GC analyses, making them hard to compare. Hence, we also explored the power spectral density calculated across the same windows across which the GC was determined.

Unconditioned Granger causality at two delays

We first selected pairs of interest, and for each determined the set of trials which were available in both leads, in a greedy fashion. This means that for each pair potentially a different set of trials was used. We also tried a setting in which we took the trials that were present in all leads of interest (across all pairs), but this reduced the number available trials too much yielding estimates of low statistical quality.

Each time interval was divided into time-windows (length: 100 samples at 1000 Hz) that were shifted with steps of 10 samples (‘offset’) across the entire interval. We used the mvgc tool box (version 1.0⁶⁷), which is a parametric method, but uses an efficient algorithm that in our hands was more robust than the frequency-based non-parametric method utilizing spectral decomposition⁶⁸. For each window the Granger causality was calculated, both in the time-domain (yielding a 2 × 2 matrix for each window, of which only the off-diagonal elements are relevant) and in the frequency domain (yielding one 2 × 2 matrix for each of the 101 frequencies between 0 and 125 Hz) with lags up to 4. Lags refers to how many time steps (samples) in the past are used in the parametric model to predict the current responses. We also refer to this maximum of samples in the past as delay, for this setting the delay is 4 times 1 ms, i.e. 4 ms. This parameter setting represented the standard set, labelled NS1. We also explored two additional settings where we doubled and quadrupled the distance between samples (i.e. sampling rate of 500 Hz and 250 Hz, respectively, indicated by NS2 and NS4, respectively), this allows to extract causal influences at a longer delay. Our model order (i.e. maximum lag, corresponding to the delay in ms), was chosen based on a trade-off between temporal resolution (window length) and the number of model parameters, for which the AIC criterion in the mvgc toolbox indicated lags of 4. We indeed do find effects with a lag of 4 with sampling rate of 1000 Hz, corresponding to a delay of 4 ms, which is rather short when considering the typical 10 ms that each cortical area adds to response latency, as assessed in experiments with non-human primates using visual stimuli^69,70. However, more recent mouse recordings using neuropixels do find spike to spike correlations, indicative of a direct synaptic connection between neurons, in different cortical regions with a median delay of 3.9 ms⁷¹. In the same study, the difference in response onset (first spike time) of between the visual area highest in the hierarchy and that lowest in the hierarchy is about 12 ms, indicating that our results at lag 4 with sampling rate 250 Hz, i.e. delay of 16 ms, could reflect direct interactions between further separated areas²², connected by long intracortical tracts, such as the uncinated tract, in addition to multi-step connections involving intermediate areas.

Granger causality and power in sending and receiving areas

An analytical calculation based on a first-order (lag 1) model for two leads indicates that the GC represents the logarithm of the strength of interaction and the ratio of activity in the sending over the receiving area. Hence, there is an expectation that there is correlation between GC score and power of either sending or receiving area or both. This calculation, however, does not take into account the effect of other areas on the pair of leads, nor the effects of z-scoring as well as higher model order (lag 4). We evaluated this potential correlation for both internal connections as well as external projections by determining the correlation between GC peak and power peak during the response period. In general (see Tables S4, S5) the uGC correlated weakly with the power of the source or of the target area, or the product of these powers, indicating that uGC captures the functional connection between areas without much effect of the power in the connected areas.

We performed three sets of runs, each for a different set of analyses. First, we took all of the selected trials and determined one time-varying Granger causality (GC) curve. Second, we selected 20 times a different subset of trials (with replacement, hence some trials could be missing and other trials could be used more than once), and determined the Granger causality for each. We refer to this as the bootstrap set. We then determined the mean and standard deviation across bootstraps for each time window. This gives an estimate of the statistical variability due to (limited) number of trials available, and allows us the compare Granger values for different pairs or in the same pair for different directions. Third, we computed the Granger causality between a lead with trials in one randomly permuted order, and the second lead of the pair with the trials in a different randomly permuted order. We repeated this calculation 20 times to obtain a mean and a standard deviation. This mean represents the GC value between leads that are not causally connected, hence it is an estimate of the bias in the GC and its variance. We use it to determine whether there is a significant causal influence of one lead on the other, by normalizing the corresponding GC score by subtracting the surrogate mean and dividing by the standard deviation. This is referred to as the score relative to surrogate data.

Connection strength and types of strong connections

We found that in most cases there was a significant GC during the entire trial period. Hence, to assess the task-dependent aspects we z-scored the GC scores using the mean value and standard deviation during the baseline period, which was defined as the period starting 900 ms before static stimulus onset and ending at 0 ms, at the static onset. This means that more variable GC scores that led to high standard deviation in the baseline, result in smaller z-scores. In addition, z-scores with negative sign are also of interest as they represent the case where the tasks reduces causal influences.

For each connection (both internal, between pairs of leads in T and VL as well as external between pairs with leads outside TP), we determined the maximum and minimum z-score during the response period, and combined these into a new score indicating the strength C of the response. If both minimum and maximum were positive, C was equal to the maximum, whereas if both were negative, C was equal to the minimum. When the maximum is positive and minimum is negative, and their absolute value differs less than 3, we take C as the mean of the two. If the absolute difference is larger than 3 we take as C the variable with the maximum absolute value, including its sign. We use this procedure to assign a useful signed single measure for the positive or negative strength of the response even when there is bimodal response profile in time.

These strength measures were combined across patients and calculated for two delays (sampling at 1 ms and 4 ms, NS1 and NS4, respectively) as well as the two task conditions (gender versus action). We determined the two-dimensional density of NS1 and NS4 using the matlab routine ksdensity with the bandwidth variable set equal to 0.8, and added level lines via matlab function contour for 8 levels between 0 and 0.006.

We assigned a type to external connections with the TP as well as to internal pairs (both leads within TP) based on the peak value of the GC during the response period for NS1 (sampling rate 1 kHz) and NS4 (sampling rate 250 Hz). We determined the bivariate distribution as described in the preceding section, and found that GC values exceeding a z-score of 5 constituted outliers. When the NS1 z-score exceeded 5, but the NS4 did not, we referred to those connections as disappear (D) connections, whereas if the NS4 also exceeded 5, they were ‘maintain’ (M). When the NS1 z-score was less than 5 (and not below -5), but the NS4 z-score exceeded 5, the connection was called ‘appear’ (A). When the NS4 score was below -5, and NS1 score was less than 5 (potentially also negative), we referred to the lead as ‘negative’ (N). We did not consider further the category opposite (when a z-score exceeding 5 or -5 flipped to the opposite sign when changing delay), since it was not present in our data to a sufficient level.

Conditioning of Granger causality

Granger causality between pairs of leads can be confounded by common input from other sources that are not included in the calculation. These influences can be exposed and accounted for by calculating the conditional Granger, which means fitting a parametric model that includes more than 2 leads, the more additional leads the better. There is however a trade-off between the window length, which needs to be short enough to follow the temporal dynamics of the tasks reflecting the engagement of causal influences between leads, and which needs to be large enough to have enough data relative to the number of parameters estimated. We found that a window length of 100 ms gave us just enough resolution to identify the brief responses. The number of parameters is the number of lags times the square of the number of leads included. It was not feasible to go beyond 3 leads, which corresponds to just one conditioning lead per pair. So as an alternative strategy we considered all remaining leads once as a conditioning lead for each pair of interest. The conditioning leads that reduced the GC z-score below 3 were considered the most influential leads because it meant that they provided the strongest common input to the pair.

Determination of time of response onset

We were interested in determining which leads were activated first by stimulus onset and which pairs increased the level of causal interactions the first. We therefore calculated the latency on a trial-by-trial basis. We determined the timing of power onset and peak GC scores by first finding the maximum in a region around the expected response (adjusted by patient), normalizing the traces by this maximum, determining the time sample at which 50% of peak height was first exceeded and then linearly interpolating to find where the curve would have crossed 50%. We applied this to gamma power estimates that were determined via matlab routine pwelch⁷² on the same set of windows that were used in the standard GC estimates (i.e. 100 samples of 1 ms) rather than the wavelet procedure introduced before so that the timing could be compared. The data were z-scored, and then bootstrapped to get 100 curves, for each of them the timing analysis was conducted and the mean determined. For GC scores we only selected those curves whose peak z-score exceeded 3. The error estimates are obtained across all available channels for power onset and across all available pairs of channels for GC (see Fig. 3I).

Statistical tests

We combined statistics on response strength as a continuous value as well as a binary value across patients and aimed to differentiate across different variables, such as brain region and task. Following the suggestion of a recent methodological paper⁷³, we used mixed effects models, where patients were a random variable (random effect) and region/task were predictors (fixed effect).

We use two types of models, models that predict continuous values such the strength of a connection and those that predict counts, either the responsiveness of a lead (binary: 0 or 1) or the count of the number of leads satisfying a particular property. In all cases, we start with a linear model that predicts a continuous value using a linear combination of the fixed effects and random effects. In some cases, where there were two distinct types of fixed effects, for instance region as well as direction of the GC score, the two effects were included as separate levels in the model. For these cases we in addition explored interaction effects, in which the product of two fixed effects was included in the formula for prediction, as long as there were enough data points relative to the number of parameters. When a continuous output is required, no additional terms are needed in the model and we fit it using the matlab routine fitlme, which uses maximum likelihood to find the optimal model parameters. When a binary value of count needs to be predicted we need to specify a response distribution: the binomial distribution for the binary variables and the Poisson for the counts. This distribution in addition specifies a nonlinear function, called the link function that is applied to the continuous predictor, which is the logit and log, respectively. For these models, we the matlab routine fitglme, in which the Laplace approximation is used to find the optimal model parameters.

Consider for example the first test in Supp. Text mixed effects.

We specify the formula as: ‘Proportion responsive = 1 + Region + (1|Patient)’.

This means that we predict whether particular leads in a particular region in a particular patient are responsive, for which the data is a sequence of zeros and ones, and corresponding region and patient labels. The model is comprised of an overall mean, indicated by the first ‘1’, and a fixed effect of the region, ‘Region’ in the equation. Note that if there are four regions, there are only three fixed effect levels, for which we use in the mathematical formulation indicator values (1 if the lead came from a particular region, zero otherwise). Three indicator values are then enough, because if they are all zero, the lead came from the fourth region. Hence, four indicator values would be dependent (their sum always equal to 1), and the algorithm would not converge. The patient is a random effect and it contributes a shift in mean, indicated by the last ‘1’.

We provide in addition the line: ‘(Binomial, 176, 4 fixed eff. coeff. = region, 19 random eff. coeff. = patient)’.

The first entry is the response distribution, Binomial, because we model binary data, followed by the number of observations, the number of fixed effects (4, for which we obtain only 3 values), and the number and identity of random effects.

The fixed effects are then quantified in the table in terms of the intercept (the coefficient of first ‘1’ in the model), and the names of the three fixed effect regions, for which we give the t value (with the effective number of degrees of freedom, i.e. the number observations minus fixed effect parameters) and the p value. We report a fixed effect as significant when the level value is significant relative to background (overall mean).

We also visualized the mixed effect models, for instance in Fig. 1E: we represented for each lead in one of the four TP anatomical regions (T, VL, Md, Ds) the responsiveness as a binary value. A mixed effect model was then determined as above with the patient considered as a random effect and the brain region as a fixed effect. The results are displayed using a violin plot (downloaded from https://github.com/bastibe/Violinplot-Matlab, written by Bastian Bechtold), based on the fraction responsive leads for each patient and each region (displayed as dots), with the horizontal lines representing the predicted mean responsiveness across patients for each region obtained from the model fit (using the matlab fitted function).

Visualization of lead location

We used a set of surface files (i.e. fsaverage.R.cartesian-std.164k_fs_LR.coord) downloaded from the caret legacy site (http://brainvis.wustl.edu/sumsdb/archive_index.html), exported to and rendered in matlab using user-written programs. We used both the flatmap as well as inflated surface representations⁷⁴.

Data availability

Data and code will be made available upon reasonable request to PT, subject to preserving patient confidentiality.

Abbreviations

2AFC:: Two alternative forced choice
A:: Appear connection
Ac:: Action
AIC:: Aikake information criterion, used in the mvgc toolbox to determine optimal lag of GC calculation
BEp:: Baseline epoch
C:: Signed strength of the TP response combining negative and positive peaks
Cen S:: Central sulcus
Cing S:: Cingulate sulcus
D:: Disappear connection
Ds:: Dorsal subregion of TP
EEG:: Electro encephalogram
EZ:: Epileptic zone
FCD:: Focal cortical dysplagia
FFA:: Fusiform face area
fMRI:: Functional Magnetic resonance imaging
GC:: Granger Causality
Ge:: Gender
HL1:: Hidden layer 1
HL2:: Hidden layer 2
Hz:: Hertz
IED:: Interictal epileptic discharge
IFG:: Inferior frontal gyrus
IFS:: Inferior frontal sulcus
IPS:: Intraparietal sulcus
IPL:: Inferior parietal lobule
ITS:: Inferior temporal sulcus
Lat:: Latency
LG:: Lingual gyrus
LG1:: Functional region on the LG
LR:: Left right
M:: Maintain connection
Md:: Medial subregion of TP
MR:: Magnetic resonance
MRI:: Magnetic resonance imaging
ms:: Millisecond
N:: Negative connection
NS1:: Number of samples 1, GC scores based on a sampling interval of 1 ms
NS4:: Number of samples 4, GC scores based on a sampling interval of 4 ms
OFC:: Orbito-frontal cortex
OTS:: Occipito-temporal sulcus
PFC:: Prefrontal cortex
RT:: Reaction time
SD:: Standard deviation
SEp:: Static epoch
STS:: Superior temporal sulcus
T:: Tip subregion of TP
TA:: Cyto-architectonic region of temporal lobe
TE:: Cyto-architectonic region of temporal lobe
TG:: Cyto-architectonic region of temporal lobe
Tipin:: Projections from cortex outside TP to Tip
Tipout:: Projections from T to cortex outside TP
TP:: Temporal pole
uGC:: Unconditioned Granger Causality
VEp:: Video epoch
VL:: Ventrolateral subregion of TP
VLin:: Projections from cortex outside TP to VL
VLout:: Projections from VL to cortex outside TP
VT:: Ventral temporal
Y:: Year

References

Ralph, M. A. L., Jefferies, E., Patterson, K. & Rogers, T. T. The neural and computational bases of semantic cognition. Nat. Rev. Neurosci. 18(1), 42–55. https://doi.org/10.1038/nrn.2016.150 (2017).
Article CAS PubMed Google Scholar
Clarke, A., Devereux, B. J., Randall, B. & Tyler, L. K. Predicting the time course of individual objects with MEG. Cereb. Cortex. 25(10), 3602–3612. https://doi.org/10.1093/cercor/bhu203 (2015).
Article PubMed Google Scholar
Binney, R. J., Parker, G. J. M. & Ralph, M. A. L. Convergent connectivity and graded specialization in the rostral human temporal lobe as revealed by diffusion-weighted imaging probabilistic tractography. J. Cognit. Neurosci. 24(10), 1998–2014. https://doi.org/10.1162/jocn_a_00263 (2012).
Article Google Scholar
Jackson, R. L., Hoffman, P., Pobric, G. & Ralph, M. A. L. The semantic network at work and rest: differential connectivity of anterior temporal lobe subregions. J. Neurosci. 36(5), 1490–1501. https://doi.org/10.1523/Jneurosci.2999-15.2016 (2016).
Article CAS PubMed PubMed Central Google Scholar
Ding, S. L., Van Hoesen, G. W., Cassell, M. D. & Poremba, A. Parcellation of human temporal polar cortex: A combined analysis of multiple cytoarchitectonic, chemoarchitectonic, and pathological markers. J. Comp. Neurol. 514(6), 595–623. https://doi.org/10.1002/cne.22053 (2009).
Article PubMed PubMed Central Google Scholar
Pascual, B. et al. Large-scale brain networks of the human left temporal pole: A functional connectivity MRI study. Cereb. Cortex. 25(3), 680–702. https://doi.org/10.1093/cercor/bht260 (2015).
Article PubMed Google Scholar
Hung, J. Y., Wang, X. Y., Wang, X. S. & Bi, Y. C. Functional subdivisions in the anterior temporal lobes: A large scale meta-analytic investigation. Neurosci. Biobehav. R. 115, 134–145. https://doi.org/10.1016/j.neubiorev.2020.05.008 (2020).
Article Google Scholar
Jackson, R. L., Rogers, T. T. & Ralph, M. A. L. Reverse-engineering the cortical architecture for controlled semantic cognition. Nat. Hum. Behav. 5(6), 774. https://doi.org/10.1038/s41562-020-01034-z (2021).
Article PubMed PubMed Central Google Scholar
Platonov, A. et al. Rapid and specific processing of person-related information in human anterior temporal lobe. Commun. Biol. 2(1), 5. https://doi.org/10.1038/s42003-018-0250-0 (2019).
Article PubMed PubMed Central Google Scholar
Smith, E. E., Shoben, E. J. & Rips, L. J. Structure and process in semantic memory - featural model for semantic decisions. Psychol. Rev. 81(3), 214–241. https://doi.org/10.1037/h0036351 (1974).
Article Google Scholar
Martin, A. & Chao, L. L. Semantic memory and the brain: Structure and processes. Curr. Opin. Neurobiol. 11(2), 194–201. https://doi.org/10.1016/S0959-4388(00)00196-3 (2001).
Article CAS PubMed Google Scholar
Granger, C. W. J. Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37(3), 424–438. https://doi.org/10.2307/1912791 (1969).
Article MATH Google Scholar
Bernasconi, C. & Konig, P. On the directionality of cortical interactions studied by structural analysis of electrophysiological recordings. Biol. Cybern. 81(3), 199–210. https://doi.org/10.1007/s004220050556 (1999).
Article CAS PubMed MATH Google Scholar
Tong, A. P. S., Vaz, A. P., Wittig, J. H., Inati, S. K. & Zaghloul, K. A. Ripples reflect a spectrum of synchronous spiking activity in human anterior temporal lobe. Elife 10, e68401. https://doi.org/10.7554/eLife.68401 (2021).
Article CAS PubMed PubMed Central Google Scholar
Tsao, D. Y., Schweers, N., Moeller, S. & Freiwald, W. A. Patches of face-selective cortex in the macaque frontal lobe. Nat Neurosci. 11(8), 877–879. https://doi.org/10.1038/nn.2158 (2008).
Article CAS PubMed PubMed Central Google Scholar
Allison, T., Puce, A., Spencer, D. D. & McCarthy, G. Electrophysiological studies of human face perception. I: Potentials generated in occipitotemporal cortex by face and non-face stimuli. Cereb Cortex. 9(5), 415–430. https://doi.org/10.1093/cercor/9.5.415 (1999).
Article CAS PubMed Google Scholar
Lachaux, J. P. et al. The many faces of the gamma band response to complex visual stimuli. Neuroimage 25(2), 491–501. https://doi.org/10.1016/j.neuroimage.2004.11.052 (2005).
Article PubMed Google Scholar
Fisch, L. et al. Neural “ignition”: Enhanced activation linked to perceptual awareness in human ventral stream visual cortex. Neuron 64(4), 562–574. https://doi.org/10.1016/j.neuron.2009.11.001 (2009).
Article CAS PubMed PubMed Central Google Scholar
Engell, A. D. & McCarthy, G. The relationship of gamma oscillations and face-specific ERPs recorded subdurally from occipitotemporal cortex. Cereb. Cortex. 21(5), 1213–1221. https://doi.org/10.1093/cercor/bhq206 (2011).
Article PubMed Google Scholar
Ghuman, A. S. et al. Dynamic encoding of face information in the human fusiform gyrus. Nat. Commun. 5, 5672. https://doi.org/10.1038/ncomms6672 (2014).
Article CAS PubMed ADS Google Scholar
Avanzini, P. et al. Four-dimensional maps of the human somatosensory system. P Natl. Acad. Sci. USA 113(13), E1936–E1943. https://doi.org/10.1073/pnas.1601889113 (2016).
Article CAS Google Scholar
Gregoriou, G. G., Gotts, S. J., Zhou, H. H. & Desimone, R. High-frequency, long-range coupling between prefrontal and visual cortex during attention. Science 324(5931), 1207–1210. https://doi.org/10.1126/science.1171402 (2009).
Article CAS PubMed PubMed Central ADS Google Scholar
Bosman, C. A. et al. Attentional stimulus selection through selective synchronization between monkey visual areas. Neuron 75(5), 875–888. https://doi.org/10.1016/j.neuron.2012.06.037 (2012).
Article CAS PubMed PubMed Central Google Scholar
Hirabayashi, T., Takeuchi, D., Tamura, K. & Miyashita, Y. Microcircuits for hierarchical elaboration of object coding across primate temporal areas. Science 341(6142), 191–195. https://doi.org/10.1126/science.1236927 (2013).
Article CAS PubMed ADS Google Scholar
Roberts, M. J. et al. Robust gamma coherence between macaque V1 and V2 by dynamic frequency matching. Neuron 78(3), 523–536. https://doi.org/10.1016/j.neuron.2013.03.003 (2013).
Article CAS PubMed Google Scholar
Chen, R. J., Wang, F., Liang, H. L. & Li, W. Synergistic processing of visual contours across cortical layers in V1 and V2. Neuron 96(6), 1388. https://doi.org/10.1016/j.neuron.2017.11.004 (2017).
Article CAS PubMed Google Scholar
Hirshorn, E. A. et al. Decoding and disrupting left midfusiform gyrus activity during word reading. P Natl Acad Sci USA 113(29), 8162–8167. https://doi.org/10.1073/pnas.1604126113 (2016).
Article CAS ADS Google Scholar
Woolnough, O. et al. Category selectivity for face and scene recognition in human medial parietal cortex. Curr Biol. 30(14), 2707. https://doi.org/10.1016/j.cub.2020.05.018 (2020).
Article CAS PubMed Google Scholar
Jia, X. X. et al. Multi-regional module-based signal transmission in mouse visual cortex. Neuron 110(9), 1585. https://doi.org/10.1016/j.neuron.2022.01.027 (2022).
Article CAS PubMed Google Scholar
Marinkovic, K. et al. Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron 38(3), 487–497. https://doi.org/10.1016/S0896-6273(03)00197-1 (2003).
Article CAS PubMed PubMed Central Google Scholar
Premereur, E., Taubert, J., Janssen, P., Vogels, R. & Vanduffel, W. Effective connectivity reveals largely independent parallel networks of face and body patches. Curr. Biol. 26(24), 3269–3279. https://doi.org/10.1016/j.cub.2016.09.059 (2016).
Article CAS PubMed Google Scholar
Vogels, R. More than the face: Representations of bodies in the inferior temporal cortex. Annu. Rev. Vis. Sci. 8, 383–405. https://doi.org/10.1146/annurev-vision-100720-113429 (2022).
Article PubMed Google Scholar
Orban, G. A., Zhu, Q. & Vanduffel, W. The transition in the ventral stream from feature to real-world entity representations. Front. Psychol. 5, 695. https://doi.org/10.3389/fpsyg.2014.00695 (2014).
Article PubMed PubMed Central Google Scholar
Landi, S. M., Viswanathan, P., Serene, S. & Freiwald, W. A. A fast link between face perception and memory in the temporal pole. Science 373(6554), 581. https://doi.org/10.1126/science.abi6671 (2021).
Article CAS PubMed PubMed Central ADS Google Scholar
Brewer, A. A., Liu, J. J., Wade, A. R. & Wandell, B. A. Visual field maps and stimulus selectivity in human ventral occipital cortex. Nat. Neurosci. 8(8), 1102–1109. https://doi.org/10.1038/nn1507 (2005).
Article CAS PubMed Google Scholar
Arcaro, M. J., McMains, S. A., Singer, B. D. & Kastner, S. Retinotopic organization of human ventral visual cortex. J. Neurosci. 29(34), 10638–10652. https://doi.org/10.1523/Jneurosci.2807-09.2009 (2009).
Article CAS PubMed PubMed Central Google Scholar
Cant, J. S. & Goodale, M. A. Attention to form or surface properties modulates different regions of human occipitotemporal cortex. Cereb. Cortex. 17(3), 713–731. https://doi.org/10.1093/cercor/bhk022 (2007).
Article PubMed Google Scholar
Peuskens, H. et al. Attention to 3-D shape, 3-D motion, and texture in 3-D structure from motion displays. J. Cognit. Neurosci. 16(4), 665–682. https://doi.org/10.1162/089892904323057371 (2004).
Article Google Scholar
Oldfield, R. C. & Wingfield, A. Response latencies in naming objects. Q J. Exp. Psychol. 17(4), 273–281. https://doi.org/10.1080/17470216508416445 (1965).
Article CAS PubMed Google Scholar
Chiou, R., Humphreys, G. F., Jung, J. & Ralph, M. A. L. Controlled semantic cognition relies upon dynamic and flexible interactions between the executive “semantic control” and hub-and-spoke “semantic representation” systems. Cortex 103, 100–116. https://doi.org/10.1016/j.cortex.2018.02.018 (2018).
Article PubMed PubMed Central Google Scholar
ter Wal, M. et al. Human stereoEEG recordings reveal network dynamics of decision-making in a rule-switching task. Nat. Commun. https://doi.org/10.1038/s41467-020-16854-w (2020).
Article PubMed PubMed Central Google Scholar
Quiroga, R. Q. Concept cells: The building blocks of declarative memory functions. Nat. Rev. Neurosci. 13(8), 587–597. https://doi.org/10.1038/nrn3251 (2012).
Article CAS PubMed Google Scholar
Fu, Z. Z. et al. The geometry of domain-general performance monitoring in the human medial frontal cortex. Science 376(6593), 595. https://doi.org/10.1126/science.abm9922 (2022).
Article CAS Google Scholar
Crepaldi, D. et al. Clustering the lexicon in the brain: A meta-analysis of the neurofunctional evidence on noun and verb processing. Front. Hum. Neurosci. 7, 303. https://doi.org/10.3389/fnhum.2013.00303 (2013).
Article PubMed PubMed Central Google Scholar
Bi, Y. C. Dual coding of knowledge in the human brain. Trends Cogn. Sci. 25(10), 883–895. https://doi.org/10.1016/j.tics.2021.07.006 (2021).
Article PubMed Google Scholar
Von Der Heide, R. J., Skipper, L. M., Klobusicky, E. & Olson, I. R. Dissecting the uncinate fasciculus: Disorders, controversies and a hypothesis. Brain 136, 1692–1707. https://doi.org/10.1093/brain/awt094 (2013).
Article PubMed Google Scholar
Fox, K. C. R. et al. Changes in subjective experience elicited by direct stimulation of the human orbitofrontal cortex. Neurology 91(16), E1519–E1527. https://doi.org/10.1212/Wnl.0000000000006358 (2018).
Article PubMed PubMed Central Google Scholar
Schmuhl-Giesen, S. et al. Sublamina-specific dynamics and ultrastructural heterogeneity of layer 6 excitatory synaptic boutons in the adult human temporal lobe neocortex. Cereb. Cortex. 32(9), 1840–1865. https://doi.org/10.1093/cercor/bhab315 (2022).
Article PubMed Google Scholar
Tang, H. L. et al. Cascade of neural processing orchestrates cognitive control in human frontal cortex. Elife 5, e12352. https://doi.org/10.7554/eLife.12352 (2016).
Article CAS PubMed PubMed Central Google Scholar
Munari, C. et al. Stereo-electroencephalography methodology - advantages and limits. Acta Neurol. Scand. 89, 56–67. https://doi.org/10.1111/j.1600-0404.1994.tb05188.x (1994).
Article Google Scholar
Cardinale, F. et al. Stereoelectroencephalography: Surgical methodology, safety, and stereotactic application accuracy in 500 procedures. Neurosurgery 72(3), 353–366. https://doi.org/10.1227/NEU.0b013e31827d1161 (2013).
Article PubMed Google Scholar
Spinnler, H. & Tognoni, G. Standardizzazione e taratura italiana di test neuropsicologici. Ital J Neurol Sci 8, 1–120 (1987).
Google Scholar
Kaplan, E., Goodglass, H., Weintraub, S., Segal, O. Boston Naming Test (Lea & Febiger, 1983).
Biancardi, A. & Stoppa, E. Il test delle Campanelle modificato: Una proposta per lo studio dell’attenzione in etá evolutiva [The Bells Test revised: A proposal for the study of attention in childhood]. Psichiatria dell’infanzia e dell’adolescenza. 64(1), 73–84 (1997).
Google Scholar
Benton, A. L. & Van Allen, M. W. Impairment in facial recognition in patients with cerebral disease. Trans Am Neurol Assoc. 93, 38–42 (1968).
CAS PubMed Google Scholar
Narizzano, M. et al. SEEG assistant: A 3DSlicer extension to support epilepsy surgery. BMC Bioinf. 18, 124. https://doi.org/10.1186/s12859-017-1545-8 (2017).
Article Google Scholar
Cardinale, F. et al. Stereoelectroencephalography: Retrospective analysis of 742 procedures in a single centre. Brain 142, 2688–2704. https://doi.org/10.1093/brain/awz196 (2019).
Article PubMed Google Scholar
Fedorov, A. et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging. 30(9), 1323–1341. https://doi.org/10.1016/j.mri.2012.05.001 (2012).
Article PubMed PubMed Central Google Scholar
Dale, A. M., Fischl, B. & Sereno, M. I. Cortical surface-based analysis – I. Segmentation and surface reconstruction. Neuroimage 9(2), 179–194. https://doi.org/10.1006/nimg.1998.0395 (1999).
Article CAS PubMed Google Scholar
Brainard, D. H. The psychophysics toolbox. Spat. Vis. 10(4), 433–436. https://doi.org/10.1163/156856897x00357 (1997).
Article CAS PubMed Google Scholar
Pelli, D. G. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis. 10(4), 437–442 (1997).
Article CAS PubMed Google Scholar
Vidal, J. R. et al. Category-specific visual responses: An intracranial study comparing gamma, beta, alpha, and ERP response selectivity. Front. Hum. Neurosci. 4, 195. https://doi.org/10.3389/fnhum.2010.00195 (2010).
Article PubMed PubMed Central Google Scholar
Rousseeuw, P. J. & Hubert, M. Anomaly detection by robust statistics. Wires Data Min. Knowl. 8(2), e1236. https://doi.org/10.1002/widm.1236 (2018).
Article Google Scholar
Lilly, J. M. & Olhede, S. C. Higher-order properties of analytic wavelets. IEEE T Signal Proces. 57(1), 146–160. https://doi.org/10.1109/Tsp.2008.2007607 (2009).
Article MathSciNet MATH ADS Google Scholar
Rubner, Y., Tomasi, C., Guibas, L.J., (eds.) A metric for distributions with applications to image databases. In IEEE International Conference on Computer Vision (Bombay, 1998).
Bezdek, J. C. Pattern Recognition with Fuzzy Objective Function Algorithms (Plenum, 1981).
Book MATH Google Scholar
Barnett, L. & Seth, A. K. The MVGC multivariate Granger causality toolbox: A new approach to Granger-causal inference. J. Neurosci. Meth. 223, 50–68. https://doi.org/10.1016/j.jneumeth.2013.10.018 (2014).
Article Google Scholar
Dhamala, M., Rangarajan, G. & Ding, M. Estimating granger causality from fourier and wavelet transforms of time series data. Phys Rev Lett. 100(1), 018701. https://doi.org/10.1103/PhysRevLett.100.018701 (2008).
Article CAS PubMed ADS Google Scholar
Vogels, R. & Orban, G. A. Activity of inferior temporal neurons during orientation discrimination with successively presented gratings. J. Neurophysiol. 71(4), 1428–1451. https://doi.org/10.1152/jn.1994.71.4.1428 (1994).
Article CAS PubMed Google Scholar
Schmolesky, M. T. et al. Signal timing across the macaque visual system. J. Neurophysiol. 79(6), 3272–3278. https://doi.org/10.1152/jn.1998.79.6.3272 (1998).
Article CAS PubMed Google Scholar
Siegle, J. H. et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature 592(7852), 86. https://doi.org/10.1038/s41586-020-03171-x (2021).
Article CAS PubMed ADS Google Scholar
Press WH. Numerical Recipes: The Art of Scientific Computing 3rd edn. xxi, 1235 (Cambridge University Press, Cambridge; New York, 2007).
Yu, Z. X. et al. Beyond t test and ANOVA: Applications of mixed-effects models for more rigorous statistical analysis in neuroscience research. Neuron 110(1), 21–35. https://doi.org/10.1016/j.neuron.2021.10.030 (2022).
Article CAS PubMed Google Scholar
Van Essen, D. C., Glasser, M. F., Dierker, D. L., Harwell, J. & Coalson, T. Parcellations and hemispheric asymmetries of human cerebral cortex analyzed on surface-based atlases. Cereb. Cortex. 22(10), 2241–2262. https://doi.org/10.1093/cercor/bhr291 (2012).
Article PubMed Google Scholar

Download references

Acknowledgements

The authors are indebted to P. Avanzini for help with the anatomical reconstruction of the recording electrodes and storage of the data. They also thank R. Vogels, I. Fried, T. Womelsdorf and M. ter Wal for comments on an earlier version of the manuscript. Supported in part by GO: ERC Advanced Grant nr. 323606 (Parietalaction), GO/PT: the European Union’s Seventh Framework Programme (FP7/2007-2013) Grant nr. 600925 (NeuroSeeker), PT: NWO Grant 680-91-318 (NeuronsReunited), National Institute of Mental Health of the National Institutes of Health; Award Number R01MH123687.

Author information

Authors and Affiliations

Neuroinformatics Department, Faculty of Science, Radboud University, Heyendaalseweg 135, 6525AJ, Nijmegen, The Netherlands
P. Tiesinga
Department of Medicine and Surgery, University of Parma, Via Volturno 39/E, 43125, Parma, Italy
A. Platonov & G. A. Orban
Claudio Munari Center for Epilepsy Surgery, Ospedale Niguarda-Ca’ Granda, 20162, Milan, Italy
V. Pelliccia, G. LoRusso & I. Sartori

Authors

P. Tiesinga
View author publications
You can also search for this author in PubMed Google Scholar
A. Platonov
View author publications
You can also search for this author in PubMed Google Scholar
V. Pelliccia
View author publications
You can also search for this author in PubMed Google Scholar
G. LoRusso
View author publications
You can also search for this author in PubMed Google Scholar
I. Sartori
View author publications
You can also search for this author in PubMed Google Scholar
G. A. Orban
View author publications
You can also search for this author in PubMed Google Scholar

Contributions

G.O., P.T. conceived and designed the analysis; G.O., P.T., A.P. analyzed the data; G.O. and P.T. wrote and revised the paper; A.P., V.P., G.L., and I.S. collected the data; G.O. acquired funding. All authors read and commented on the paper as well as the revision.

Corresponding authors

Correspondence to P. Tiesinga or G. A. Orban.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information 1.

Supplementary Information 2.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Cite this article

Tiesinga, P., Platonov, A., Pelliccia, V. et al. Uncovering the fast, directional signal flow through the human temporal pole during semantic processing. Sci Rep 13, 6831 (2023). https://doi.org/10.1038/s41598-023-33318-5

Download citation

Received: 14 December 2022
Accepted: 11 April 2023
Published: 26 April 2023
DOI: https://doi.org/10.1038/s41598-023-33318-5

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.