Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Quantitative models reveal the organization of diverse cognitive functions in the brain

## Abstract

Our daily life is realized by the complex orchestrations of diverse brain functions, including perception, decision-making, and action. The essential goal of cognitive neuroscience is to reveal the complete representations underlying these functions. Recent studies have characterised perceptual experiences using encoding models. However, few attempts have been made to build a quantitative model describing the cortical organization of multiple active, cognitive processes. Here, we measure brain activity using fMRI, while subjects perform 103 cognitive tasks, and examine cortical representations with two voxel-wise encoding models. A sparse task-type model reveals a hierarchical organization of cognitive tasks, together with their representation in cognitive space and cortical mapping. A cognitive factor model utilizing continuous, metadata-based intermediate features predicts brain activity and decodes tasks, even under novel conditions. Collectively, our results show the usability of quantitative models of cognitive processes, thus providing a framework for the comprehensive cortical organization of human cognition.

## Introduction

The cortical basis of daily cognitive processes has been studied using voxel-wise encoding and decoding modeling approaches1. These approaches employ multivariate regression analysis to determine how the brain activity in each voxel is modeled by target factors, including visual features2,3, object or scene categories4,5,6, sound features7,8,9, and linguistic information10,11,12. Moreover, other studies have further described the cortical (e.g., semantic) representational space that elucidates important categorical dimensions in the brain (e.g., mobile vs. nonmobile, animate vs. inanimate) and how such representations are mapped onto the cortex4,10. However, the previous studies have relied on brain activity recorded during passive listening or viewing tasks, and no study has yet clarified the comprehensive cortical representations underlying active cognitive processes

## Results

### Hierarchical organization of cognitive tasks

To examine how the cortical representations of over 100 tasks were related, we modeled task-evoked brain activity using a task-type model (Fig. 1) and calculated a representational similarity matrix (RSM) using the estimated weights, which were concatenated across six subjects (Fig. 2a). The training dataset consisted of 3336 samples (6672s) and the test dataset consisted of 412 samples (824s, repeated four times). The representational relationship of over 100 tasks was further visualized by a dendrogram obtained using hierarchical clustering analysis (HCA). The HCA results suggested that the tasks formed six clusters based on their associated representational patterns in the cerebral cortex. The largest clusters contained tasks based on sensory modalities, such as visual (“AnimalPhoto,” “MapSymbol”), auditory (“RateNoisy,” “EmotionVoice”), and motor (“PressLeft,” “EyeBlink”) tasks. In addition, some clusters contained higher cognitive components, such as language (“WordMeaning,” “RatePoem”), introspection (“ImagineFuture,” “RecallPast”), and memory (“MemoryLetter,” “RecallTaskEasy”). Although six clusters were determined by visual inspection for a descriptive purpose, the same analyses can be performed on any subclusters in the dendrogram. We also obtained a similar hierarchy pattern using an RSM calculated directly from brain activity (see Supplementary Fig. 1 and Supplementary Note 1).

The tasks were further represented in the subclusters of the specific cognitive properties (Fig. 2b–d). For example, in the visual cluster, tasks with food pictures (“RateDeliciousPic”, “DecideFood”) were closely located (Fig. 2b), whereas tasks with negative pictures (“RateDisgustPic”, “RatePainfulPic”) formed a separate cluster. Memory tasks involving calculations (“CalcEasy”, “CalcHard”) were closely located (Fig. 2c), whereas those involving simple digit matching (“MemoryDigit”, “MatchDigit”) formed a separate cluster. For the introspection cluster (Fig. 2d), tasks involving the imagination of the future and recalling of past events (“ImagineFuture,” “RecallPast”) were more closely located than those involving the imagination of places or faces (“ImaginePlace”, “RecallFace”).

To further explore whether hierarchical information is useful in capturing cortical representation, we constructed an additional hierarchical model that was based on the task clusters subordinated by each non-terminal node in the dendrogram (see Methods section). By comparing the prediction accuracy of brain activity using the task-type model (Fig. 1c) and the hierarchical model, we found that the hierarchical model outperformed the task-type model (hierarchical model, mean ± SD, 0.313 ± 0.046; task-type model, 0.293 ± 0.053; one-sided Wilcoxon signed-rank tests, p < 0.001 for all subjects). These results indicated that the hierarchically organized brain representations of cognitive tasks can be captured using this modeling procedure.

### Visualization of cognitive space and its cortical mapping

To interpret each PC, we first determined those that were meaningful based on the explained variance (Fig. 3b). The top four PCs that explained more than 5% of the variance were regarded as meaningful, and the dominant cognitive components were further analyzed based on these PCs. We then quantified the relative contribution of each task using PCA loadings; tasks with higher PCA loading values were contributing more to the target PC (Supplementary Table 2 and Supplementary Note 4). Thus, each PC was labeled based on these cognitive tasks. To obtain an objective interpretation of the PC labeling, we also performed a metadata-based inference of the PC-related cognitive factors (Supplementary Table 3).

To further examine the relationship between HCA and PCA results, we next calculated the relative contribution of the top four PCs to the six largest task clusters (Fig. 3c). By averaging the PCA loadings of the tasks included in each of the target clusters, we found that the top PCs corresponded well to the related clusters, with mean PCA loadings significantly different from zero (two-sided sign test, p < 0.05, FDR-corrected). PC1 (the auditory component) contributed to the auditory cluster; PC2 (the audiovisual component) contributed to the auditory and visual clusters; PC3 (the language component) contributed to the language cluster; and PC4 (the introspection component) contributed to the motor and introspection clusters. These results revealed the representational correspondence between the HCA and PCA results.

To further visualize the cortical distributions of the cognitive space representations, voxel-wise PCs were projected to the cortical sheet of each subject (Fig. 4a, Supplementary Figs. 4, 5 and Supplementary Note 5) using the same RGB color scheme as shown in Fig. 3a. For example, the occipital areas were presented mostly in green, showing that voxels in these areas signify movie-related and image-related tasks (Fig. 3a). The frontal areas showed intricate patterns, including language-related representations (blue) in the left lateral regions. This topographical organization was consistent across subjects (Supplementary Fig. 5), indicating that our analyses provided a broad representation of the cognitive space in the human cerebral cortex.

### Prediction of brain activity during novel tasks

Although the task-type model revealed the distinctive relationships among the tasks, it is too sparse to encompass the latent and continuous features and is not generalizable to novel tasks. To address these issues, we transformed over 100 tasks into the 715-dimensional latent feature space and constructed a voxel-wise cognitive factor model (Fig. 1d). The latent feature space was obtained based on the 715 terms and their reverse-inference maps from the Neurosynth database13. To produce the cognitive transform function (CTF) for each subject, we calculated the correlation coefficients between the weight map for each task in the task-type model and the reverse-inference map. We then obtained the feature matrix of the cognitive factor model by multiplying the CTF by the feature matrix of the task-type model.

To examine the generalizability of this model under novel task conditions (i.e., on a task that was not used to train the model), we trained the cognitive factor model with 80% of the tasks (approximately 82 or 83 tasks) and predicted the brain activity for the remaining 20% (Fig. 1e). In doing so, the CTF was estimated using data excluding those of the target subject, assuring the generalizability to novel tasks for each subject. Our results indicated that the model achieved significant prediction accuracy throughout the entire cortex (Fig. 5 and Supplementary Figs. 6, 7; mean ± SD, 0.322 ± 0.042; 86.2 ± 5.1% of voxels were significant; p < 0.05, false discovery rate (FDR)-corrected).

To confirm that these results could not be merely explained by simple sensorimotor effects, we performed an additional encoding model analysis that regressed out the relevant visual, auditory, and motor components (see Methods section). This additional analysis revealed significant prediction accuracy across the cerebral cortex (mean ± SD, 0.285 ± 0.035; 82.4 ± 4.9% of voxels were significant; p < 0.05, FDR-corrected; Supplementary Fig. 8), indicating that the generalizability of the cognitive factor model was due to higher-order (i.e., not sensory) cognitive components.

## Discussion

In the majority of the subjects, task clusters and top PCs contained auditory and visual clusters and components. Thus, it may be argued that the cortical representations of over 100 tasks can be solely explained by the difference of stimulus modality (i.e., sensory input). However, we found that there were task clusters in more abstract cognitive domains, such as memory and language. We also showed that the tasks in the same stimulus domain could be distinguished by decoding analysis. Furthermore, additional analyses with sensorimotor regressors showed significant prediction accuracy covering the whole cerebral cortex. These findings confirmed that these sensory regressors covered the relevant low-level features. Although stimulus modality is an important element in our cognitive space, detailed task representations also included abstract cognitive factors.

Previous studies have identified several clusters and components found in the present study using multiple tasks15,16,17,18,19,20,21. Compared with these previous works, the current study provides a framework for elucidating more general and quantitative aspects of human cognition in the following points. Using our broad task sampling paradigm, we revealed a gradual shift in the cognitive space and corresponding cortical organization, from the perceptual to more complex cognitive tasks. HCA and PCA further provided a cortical representational basis for the modeling analyses by demonstrating that the over 100 tasks cover the activity of the whole cerebral cortex and that they were differently organized in each cortical voxel. This is particularly effective in association cortices, such as the IPL, which has been associated with various cognitive domains22. By building the relevant encoding and decoding models using data from single subjects, we were able to successfully quantify the generalizability of our model for arbitrary novel task conditions. Such subject-wise modeling may also form the quantitative basis for elucidating the personal traits associated with cognitive functions23.

Although the tasks used here do not cover the entire domain of human perception and cognition (e.g., they did not cover odor perception, speech, social interaction), our method is applicable to any arbitrary task that can be performed in a scanner. Taken together, our framework provides a powerful step toward the complete modeling of the representations underlying human cognition.

## Methods

### Subjects

Six healthy subjects (aged 22–33 years, two females; referred to as ID01–ID06) with normal vision and hearing participated in the current experiment. Subjects were all right-handed (laterality quotient = 70–100), as assessed using the Edinburgh inventory24. Written informed consent was obtained from all subjects prior to their participation in the study. This experiment was approved by the ethics and safety committee of the National Institute of Information and Communications Technology in Osaka, Japan.

### Stimuli and procedure

We prepared 103 naturalistic tasks that could be performed without any preexperimental training (see Supplementary Methods for the detailed description of each task; see Supplementary Fig. 10 and Supplementary Note 6 for the behavioral results). Tasks were selected to include as many cognitive domains as possible. Each task had 12 instances; 8 instances were used in the training runs, whereas 4 were used in the test runs. The stimuli were presented on a projector screen inside the scanner (21.0 × 15.8° of visual angle at 30 Hz). The root-mean square of the auditory stimuli was normalized. During scanning, subjects wore MR-compatible ear tips. The experiment was performed for 3 days, with six runs performed each day. Presentation software (Neurobehavioral Systems, Albany, CA, USA) was used to control the stimulus presentation and the collection of behavioral data. To measure button responses, optic response pads with two buttons in each of the left and right hands were used (HHSC-2 × 2, Current Designs, Philadelphia, PA, USA).

### MRI data acquisition

The experiment was conducted using a 3.0 T scanner (TIM Trio; Siemens, Erlangen, Germany) with a 32-channel head coil. We scanned 72 interleaved axial slices that were 2.0-mm thick without a gap, parallel to the anterior and posterior commissure line, using a T2*-weighted gradient-echo multiband echo-planar imaging (MB-EPI) sequence25 [repetition time (TR) = 2000 ms, echo time (TE) = 30 ms, flip angle (FA) = 62°, field of view (FOV) = 192 × 192 mm2, resolution = 2 × 2 mm2, MB factor = 3]. We obtained 275 volumes for each run, with each following three dummy images. For anatomical reference, high-resolution T1-weighted images of the whole brain were also acquired from all subjects with a magnetization-prepared rapid acquisition gradient echo sequence (MPRAGE, TR = 2530 ms, TE = 3.26 ms, FA = 9°, FOV = 256 × 256 mm2, voxel size = 1 × 1 × 1 mm3).

### fMRI data preprocessing

Motion correction in each run was performed using the statistical parametric mapping toolbox (SPM8; Wellcome Trust Center for Neuroimaging, London, UK; http://www.fil.ion.ucl.ac.uk/spm/). All volumes were aligned to the first EPI image for each subject. Low-frequency drift was removed using a median filter with a 240-s window. The response for each voxel was then normalized by subtracting the mean response and scaling it to the unit variance. We used FreeSurfer26,27 to identify the cortical surfaces from the anatomical data and to register them to the voxels of the functional data. For each subject, the voxels identified in the cerebral cortex were used in the analysis (53,345–66,695 voxels per subject).

The task-type model was composed of one-hot vectors, which were assigned 1 or 0 for each time bin, indicating whether one of the 103 tasks was performed in that period. The total number of task-type model features was thus 103.

### Encoding model fitting

In the encoding model, cortical activity in each voxel was fitted with a finite impulse response model that captured the slow hemodynamic response and its coupling with neural activity3,28. The feature matrix FE [T × 3 N] was modeled by concatenating sets of [T × N] feature matrices with three temporal delays of 2, 4, and 6 s (T = # of samples; N = # of features). The cortical response RE [T × V] was then modeled by multiplying the feature matrix FE with the weight matrix WE [3 N × V] (V = # of voxels):

$${\hat{\mathbf{R}}}_{\mathbf{E}} = {\mathbf{F}}_{\mathbf{E}}{\mathbf{W}}_{\mathbf{E}}$$

We used an L2-regularized linear regression using the training dataset to obtain the weight matrix WE. The training dataset consisted of 3336 samples (6672s). The optimal regularization parameter was assessed using 10-fold cross-validation, with the 18 different regularization parameters ranging from 100 to 100 × 217.

The test dataset consisted of 412 samples (824 s, repeated four times). To reshape the data spanning over six test runs into the four times-repeated dataset, we discarded 6 s of the no-task period at the end of each run, as well as the 2-s feedback periods at the end of the third and sixth test runs. Four repetitions of the test dataset were averaged to increase the signal-to-noise ratio. Prediction accuracy was calculated using Pearson’s correlation coefficient between the predicted signal and the measured signal in the test dataset. Statistical significance (one-sided) was computed by comparing estimated correlations to the null distribution of correlations between two independent Gaussian random vectors with the same length as the test dataset3,10. The statistical threshold was set at p < 0.05 and corrected for multiple comparisons using the FDR procedure29.

### Evaluation of optimal regularization parameters

To keep the scale of the weight values consistent across subjects, we performed a resampling procedure to assess the optimal regularization parameter used for group HCA and PCA10. To this end, for each subject, we randomly divided the training dataset into training samples (80%) and validation samples (20%) and performed model fitting using an L2-regularized linear regression. This procedure was repeated 50 times, with the 18 different regularization parameters ranging from 100 to 100 × 217. The resultant prediction accuracies were averaged across the six subjects for each parameter. We selected the optimal regularization parameter that provided the highest mean prediction accuracy across subjects. This regularization parameter was used for model fitting for group HCA and PCA.

### Hierarchical cluster analysis

To examine hierarchical relations across tasks, we conducted an HCA. First, we concatenated the weight matrix of predictive voxels of the task-type model across six subjects. Concatenation of the estimated weights was performed to obtain a group-level representation that provides a common basis that is comparable across subjects4,10. To choose predictive voxels, for each subject, we selected the voxels that exhibited a significant prediction accuracy, with p < 0.05 (with FDR correction, 39,485–56,634 voxels per subject), and averaged three time delays for each task. We then obtained the RSM by calculating the Pearson’s correlation coefficients between the averaged weights of all task pairs. A dendrogram of 103 tasks was then described using the task dissimilarity (1—correlation coefficient) as a distance metric, using the minimum distance as a linkage criterion. Each cluster was labeled based on the included cognitive tasks.

To investigate task clusters in a model-independent way, we also conducted HCA using the brain activity of the whole cerebral cortex (Supplementary Fig. 1 and Supplementary Note 1) and visualized the 103 tasks on the two-dimensional space using non-metric multi-dimensional scaling (Supplementary Fig. 3 and Supplementary Note 3).

### Interpretation of cognitive factors related to task clusters

To interpret the plausible cognitive factors related to the target subclusters obtained in the HCA, we used Neurosynth (http://neurosynth.org; accessed 26 January 2018) as a metadata reference of the past neuroimaging literature13. From the approximately 3000 terms in the database, we manually selected 715 terms that covered the comprehensive cognitive factors while also avoiding redundancy. In particular, we removed several plural terms that also had their singular counterpart (e.g., “concept” and “concepts”) and past tense verbs that also had their present counterpart (e.g., “judge” and “judged”) from the dataset. We also excluded those terms that indicated anatomical regions (e.g., “parietal”). We used the reverse-inference map of the Neurosynth database for each of the 715 selected terms. The reverse-inference map indicated the likelihood of a given term being used in a study if the activity was observed at a particular voxel. Each reverse-inference map in the MNI152 space was then registered to the subjects’ reference EPI data using FreeSurfer26,27. For each of the cortical maps of the task cluster weight matrix (Supplementary Fig. 2 and Supplementary Note 2), we calculated Pearson’s correlation coefficients with the 715 registered reverse-inference maps, which resulted in a cognitive factor vector with 715 elements. Terms with higher correlation coefficient values were regarded as contributing more to the target cluster.

### Principal component analysis of task-type weights

For each subject, we performed PCA on the weight matrix of the task-type model concatenated across six subjects. We selected the voxels that showed significant prediction accuracy with p < 0.05 (with FDR correction, 39,485–56,634 voxels per subject) and averaged three time delays for each task. To show the structure of the cognitive space, 103 tasks were mapped onto the two-dimensional space using the loadings of PC1 (1st PC) and PC2 as the x-axis and y-axis, respectively. The tasks were further colored in red, green, and blue, based on the relative PCA loadings in PC1, PC2, and PC3, respectively. To obtain an objective interpretation of the PCs, we also performed metadata-based inference of the cognitive factors related to each PC (Supplementary Table 3 and Supplementary Note 4).

To represent the cortical organization of the cognitive space for each subject, we extracted and normalized the PCA scores from each subject’s voxels. The resultant cortical map indicated the relative contribution of each cortical voxel to the target PC (denoted as ‘PCA score map’, Supplementary Fig. 4 and Supplementary Note 5). By combining the PCA score maps from the top three PCs for each subject, we were able to visualize how each cortical voxel is represented by the associated cognitive clusters. Each cortical voxel was colored based on the relative PCA scores of PC1, PC2, and PC3, corresponding to the color of the tasks in the two-dimensional space.

### Hierarchical model

To further quantify the importance of the estimated hierarchy, we constructed a hierarchical model based on the result of HCA. We first conducted HCA using data from five subjects (excluding the target subject’s data). This dendrogram (cluster tree) includes 102 non-terminal nodes (the terminal nodes correspond to the 103 tasks). The feature matrix of the hierarchical model consisted of 102-element binary vectors, which were assigned either 1 or 0 for each time bin, indicating whether any tasks subordinated by the target non-terminal node were performed in that period.

### Cognitive factor model

To obtain task representations using continuous features in the human cognitive space, we transformed sparse task-type features into the latent cognitive factor feature space (Fig. 1d). We used the reverse-inference map of the Neurosynth database13 for each of the 715 terms selected. Each reverse-inference map in the Neurosynth database in MNI152 space was registered to the subjects’ reference EPI data using FreeSurfer26,27.

We then calculated the correlation coefficients between the weight map for each task in the task-type model and the registered reverse-inference maps. This resulted in the [103 × 715] coefficient matrix. We next obtained the CTF for each subject by averaging the coefficient matrices of the other five subjects. The CTF served to transform the feature values of the 103 tasks into the 715-dimensional latent feature space. The feature matrix of the cognitive factor model was then obtained by multiplying the CTF with the feature matrix of the task-type model. Note that the CTF (and the resultant feature matrix) of each target subject was independent of their own data. The total number of cognitive factor model features was 715.

### Encoding model fitting with sensorimotor regressors

To evaluate the possible effect of low-level sensorimotor features on model predictions, we performed an additional encoding model fitting while regressing out the sensorimotor components. To this end, we concatenated motion energy (ME) model features (visual), modulation transfer function (MTF) model features (auditory), and button response (BR) model features (motor) with the original feature matrix during model training (see the following subsections for details). ME model features were obtained by applying three-dimensional spatiotemporal Gabor wavelet filters to the visual stimuli3. MTF model features were obtained by applying spectro-temporal modulation-selective filters to the cochleogram of the auditory stimuli30. BR model features were obtained based on the number of button responses made by each subject. Model testing excluded the sensorimotor regressors from the concatenated feature matrix and the corresponding weight matrix. This analysis revealed that the model prediction accuracy was independent of low-level sensorimotor features.

### Motion energy model

We used the ME model that has been used in previous studies3,31,32 and provided in a public repository (https://github.com/gallantlab/motion_energy_matlab). First, movie frames and pictures were spatially downsampled to 96 × 96 pixels. The RGB pixel values were then converted into the Commission International de l’Eclairage (CIE) LAB color space, and the color information was subsequently discarded. The luminance (L*) pattern was passed through a bank of three-dimensional spatiotemporal Gabor wavelet filters. The outputs of the two filters with orthogonal phases (quadrature pairs) were squared and summed to yield local ME. ME was compressed with a log-transformation and temporally downsampled to 0.5 Hz. Filters were tuned to six spatial frequencies (0, 1.5, 3.0, 6.0, 12.0, 24.0 cycles per image) and three temporal frequencies (0, 4.0, 8.0 Hz), without directional parameters. Filters were positioned on a square grid that covered the screen. The adjacent filters were separated by 3.5 standard deviations of their spatial Gaussian envelopes. The total number of ME model features was 1395.

### Modulation transfer function model

A sound cochleogram was generated using a bank of 128 overlapping bandpass filters ranging from 20 to 10,000 Hz33. The window size was set to 25 ms and the hop size to 10 ms. The filter output was averaged across 2 s (TR). We further extracted the features from the MTF model30 which we provided in a public repository (https://osf.io/ea2jc/). For each cochleogram, a convolution with modulation-selective filters was then calculated. The outputs of the two filters with orthogonal phases (quadrature pairs) were squared and summed to yield the local modulation energy3. Modulation energy was then log-transformed, averaged across 2 s, and further averaged within each of the 10 nonoverlapping frequency ranges logarithmically spaced along the frequency axis. The filter outputs of the upward and downward sweep directions were used. Modulation-selective filters were tuned to five spectral modulation scales (0.50, 1.0, 2.0, 4.0, 8.0 cycles per octave) and five temporal modulation rates (4.0, 8.0, 16.0, 32.0, 64.0 Hz). The total number of MTF model features was 1000.

### Button response model

The BR model was constructed based on the number of button responses within 1 s for each of the four buttons, with the right two buttons pressed by the right thumb and the left two buttons pressed by the left thumb. The total number of BR model features was four.

### Decoding model fitting

In the decoding model, the cortical response matrix RD [T × 3 V] was modeled using concatenating sets of [T × V] matrices with temporal delays of 2, 4, and 6 s. The feature matrix FD [T × N] was modeled by multiplying the cortical response matrix RD with the weight matrix WD [3 V × N]:

$${\hat{\mathbf{F}}}_{\mathbf{D}} = {\mathbf{R}}_{\mathbf{D}}{\mathbf{W}}_{\mathbf{D}}$$

The weight matrix WD was estimated using an L2-regularized linear regression with the training dataset, following the same procedure for the encoding model fitting.

To test decoding performance in a model-independent way, we also decoded over 100 tasks directly from brain activity using a support vector machine (Supplementary Fig. 11 and Supplementary Note 7).

### Encoding and decoding with novel tasks

In a further analysis, we used a random shuffling procedure to obtain a null distribution of mean prediction accuracy (Supplementary Fig. 12). Specifically, all elements of the [715 × 412] feature matrix of the cognitive factor encoding model in the test dataset were randomly shuffled in an element-wise manner, and predicted responses were calculated by multiplying the shuffled feature matrix with the original weight matrix. The mean prediction accuracy across all voxels was calculated. This procedure was repeated 1000 times.

Additionally, we used a random shuffling procedure to obtain a null distribution of mean decoding accuracy (Supplementary Fig. 13). Specifically, all elements of the [103 × 715] CTF matrix were randomly shuffled in an element-wise manner, and the task likelihood was measured using Pearson’s correlation coefficient between the shuffled CTF of each task and each decoded cognitive factor vector. The mean decoding accuracy across all tasks was calculated. This procedure was repeated 1000 times.

All model fitting and analyses were conducted using custom software written on MATLAB. For data visualization on the cortical maps, pycortex was used34.

### Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

## Data availability

The source data underlying all Figures, Supplementary Figures, and Tables except Fig. 1 are provided as a Source Data file at the Open Science Framework (OSF, https://osf.io/ea2jc/). The raw MRI data are available at the OpenNeuro.org (https://openneuro.org/datasets/ds002306).

## Code availability

The MATLAB code used in the current study and the datasets generated and/or analyzed during the current study are available at the OSF repository (https://osf.io/ea2jc/).

## References

1. 1.

Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI. Neuroimage 56, 400–410 (2011).

2. 2.

Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–355 (2008).

3. 3.

Nishimoto, S. et al. Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1641–1646 (2011).

4. 4.

Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012).

5. 5.

Stansbury, D. E., Naselaris, T. & Gallant, J. L. Natural scene statistics account for the representation of scene categories in human visual cortex. Neuron 79, 1025–1034 (2013).

6. 6.

Çukur, T., Nishimoto, S., Huth, A. G. & Gallant, J. L. Attention during natural vision warps semantic representation across the human brain. Nat. Neurosci. 16, 763–770 (2013).

7. 7.

Hoefle, S. et al. Identifying musical pieces from fMRI data using encoding and decoding models. Sci. Rep. 8, 2266 (2018).

8. 8.

Norman-Haignere, S., Kanwisher, N. G. & McDermott, J. H. Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Neuron 88, 1281–1296 (2015).

9. 9.

Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V. & McDermott, J. H. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98, 630–644 (2018).

10. 10.

Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).

11. 11.

Nishida, S. & Nishimoto, S. Decoding naturalistic experiences from human brain activity via distributed representations of words. Neuroimage 180, 232–242 (2018).

12. 12.

de Heer, W. A., Huth, A. G., Griffiths, T. L., Gallant, J. L. & Theunissen, F. E. The hierarchical cortical organization of human speech processing. J. Neurosci. 37, 6539–6557 (2017).

13. 13.

Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C. & Wager, T. D. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8, 665–670 (2011).

14. 14.

Mitchell, T. M. et al. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195 (2008).

15. 15.

Cole, M. W. et al. Multi-task connectivity reveals flexible hubs for adaptive task control. Nat. Neurosci. 16, 1348–1355 (2013).

16. 16.

Cole, M. W., Bassett, D. S., Power, J. D., Braver, T. S. & Petersen, S. E. Intrinsic and task-evoked network architectures of the human brain. Neuron 83, 238–251 (2014).

17. 17.

Shine, J. M. et al. Human cognition involves the dynamic integration of neural activity and neuromodulatory systems. Nat. Neurosci. 22, 289–296 (2019).

18. 18.

Varoquaux, G. et al. Atlases of cognition with large-scale human brain mapping. PLOS Comput. Biol. 14, e1006565 (2018).

19. 19.

Dosenbach, N. U. F. et al. A core system for the implementation of task sets. Neuron 50, 799–812 (2006).

20. 20.

Barch, D. M. et al. Function in the human connectome: task-fMRI and individual differences in behavior. Neuroimage 80, 169–189 (2013).

21. 21.

Yeo, B. T. T. et al. Functional specialization and flexibility in human association cortex. Cereb. Cortex 25, 3654–3672 (2015).

22. 22.

Humphreys, G. F. & Lambon Ralph, M. A. Fusion and fission of cognitive functions in the human parietal cortex. Cereb. Cortex 25, 3547–3560 (2015).

23. 23.

Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D. & Kriegeskorte, N. Unique semantic space in the brain of each beholder predicts perceived similarity. Proc. Natl Acad. Sci. 111, 14565–14570 (2014).

24. 24.

Oldfield, R. C. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97–113 (1971).

25. 25.

Moeller, S. et al. Multiband multislice GE-EPI at 7 tesla, with 16-fold acceleration using partial parallel imaging with application to high spatial and temporal whole-brain FMRI. Magn. Reson. Med. 63, 1144–1153 (2010).

26. 26.

Dale, A. M., Fischl, B. & Sereno, M. I. Cortical surface-based analysis: I. Segmentation and surface reconstruction. Neuroimage 9, 179–194 (1999).

27. 27.

Fischl, B., Sereno, M. I. & Dale, A. M. Cortical surface-based analysis II: inflation, flattening, and a surface-based coordinate system. Neuroimage 9, 195–207 (1999).

28. 28.

Kay, K. N., David, S. V., Prenger, R. J., Hansen, K. A. & Gallant, J. L. Modeling low-frequency fluctuation and hemodynamic response timecourse in event-related fMRI. Hum. Brain Mapp. 29, 142–156 (2008).

29. 29.

Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).

30. 30.

Chi, T., Ru, P. & Shamma, S. A. Multiresolution spectrotemporal analysis of complex sounds. J. Acoust. Soc. Am. 118, 887–906 (2005).

31. 31.

Nishimoto, S. & Gallant, J. L. A three-dimensional spatiotemporal receptive field model explains responses of area MT neurons to naturalistic movies. J. Neurosci. 31, 14551–14564 (2011).

32. 32.

Ikezoe, K., Amano, M., Nishimoto, S. & Fujita, I. Mapping stimulus feature selectivity in macaque V1 by two-photon Ca2+ imaging: encoding-model analysis of fluorescence responses to natural movies. Neuroimage 180, 312–323 (2018).

33. 33.

Ellis, D. P. W. Gammatone-like spectrograms. web resource. http://www.ee.columbia.edu/~dpwe/resources/matlab/ (2009).

34. 34.

Gao, J. S., Huth, A. G., Lescroart, M. D. & Gallant, J. L. Pycortex: an interactive surface visualizer for fMRI. Front. Neuroinform. 9, 23 (2015).

## Acknowledgements

We thank MEXT/JSPS KAKENHI (grant numbers JP17K13083 and JP18H05091 in #4903 (Evolinguistics) for T.N., JP15H05311 and JP18H05522 for S.N.) as well as JST CREST JPMJCR18A5 and ERATO JPMJER1801 (for S.N.) for the partial financial support of this study. The funders had no role in the study design, data collection, and analysis, decision to publish, or preparation of the manuscript.

## Author information

Authors

### Contributions

T.N. and S.N. designed the study; T.N. collected and analyzed the data; T.N. and S.N. wrote the manuscript.

### Corresponding author

Correspondence to Shinji Nishimoto.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

Peer review information Nature Communications thanks Nikolaus Kriegeskorte and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Nakai, T., Nishimoto, S. Quantitative models reveal the organization of diverse cognitive functions in the brain. Nat Commun 11, 1142 (2020). https://doi.org/10.1038/s41467-020-14913-w

• Accepted:

• Published:

• ### Neural Coding of Cognitive Control: The Representational Similarity Analysis Approach

• Michael C. Freund
• , Joset A. Etzel
•  & Todd S. Braver

Trends in Cognitive Sciences (2021)

• ### Extensive sampling for complete models of individual brains

• Thomas Naselaris
• , Emily Allen
•  & Kendrick Kay

Current Opinion in Behavioral Sciences (2021)

• ### Distinct dimensions of emotion in the human brain and their representation on the cortical surface

• Naoko Koide-Majima
• , Tomoya Nakai
•  & Shinji Nishimoto

NeuroImage (2020)