Super-resolution generative adversarial networks with static T2*WI-based subject-specific learning to improve spatial difference sensitivity in fMRI activation

The spatial resolution of fMRI is relatively poor and improvements are needed to indicate more specific locations for functional activities. Here, we propose a novel scheme, called Static T2*WI-based Subject-Specific Super Resolution fMRI (STSS-SRfMRI), to enhance the functional resolution, or ability to discriminate spatially adjacent but functionally different responses, of fMRI. The scheme is based on super-resolution generative adversarial networks (SRGAN) that utilize a T2*-weighted image (T2*WI) dataset as a training reference. The efficacy of the scheme was evaluated through comparison with the activation maps obtained from the raw unpreprocessed functional data (raw fMRI). MRI images were acquired from 30 healthy volunteers using a 3 Tesla scanner. The modified SRGAN reconstructs a high-resolution image series from the original low-resolution fMRI data. For quantitative comparison, several metrics were calculated for both the STSS-SRfMRI and the raw fMRI activation maps. The ability to distinguish between two different finger-tapping tasks was significantly higher [p = 0.00466] for the reconstructed STSS-SRfMRI images than for the raw fMRI images. The results indicate that the functional resolution of the STSS-SRfMRI scheme is superior, which suggests that the scheme is a potential solution to realizing higher functional resolution in fMRI images obtained using 3T MRI.

www.nature.com/scientificreports/ to medical imaging 8 . Recently, these schemes have been improved by combining the SR technique with generative adversarial networks (GANs) 9 to form SRGANs 10 . A SRGAN facilitates the generation of more realistic images than simple convolutional neural network-based (CNN-based) SR techniques [11][12][13] . To generate a high spatial resolution fMRI series from low-resolution data, a source of high spatial resolution information is required. Static T2*-weighted images (T2*WI) and gradient-echo EPI fMRI data exhibit similar contrast because fMRI relies on T2* relaxation 14 . Since T2*WI can be acquired at high spatial resolution, we focus here on static T2*WI as the images needed to train an SRGAN for fMRI.
In this study we have developed a new GAN-based SR scheme for fMRI, called Static T2*WI-based Subject-Specific Super Resolution fMRI (STSS-SRfMRI), to enhance the functional resolution of fMRI. The key element of the proposed method is the utilization of static T2*-WI obtained from each subject in order to train a subjectspecific model. This study aims to assess the enhancement of functional resolution using the STSS-SRfMRI scheme in comparison to the results obtained from the raw unprocessed fMRI images (raw fMRI).

Materials and methods
Subjects. Adhering to the Declaration of Helsinki, informed consent was obtained in writing from all participants prior to participation. The experimental protocols, which were approved by the Institutional Review Board at the National Institutes for Quantum and Radiological Science and Technology, conformed to the safety guidelines for MRI research.
A total of 35 healthy female volunteers (mean age 26.9 ± 6.7 years) with no history of neurological disease were selected as candidates for this study. The data from five subjects were excluded for the following reasons: the image data were damaged due to a technical error (1 subject), the candidate was visually impaired and unable to perform the task appropriately (1 subject), there were severe motion artifacts (1 subject), and the candidate failed to perform the task satisfactorily for indeterminate reasons (2 subjects).
Finger-tapping procedure. A finger-tapping task was performed during fMRI scanning. Supplementary   Figure 1 outlines the task protocol, which included phases of tapping either the thumb or little finger of one hand and resting phases between each task. Prior to beginning the experiment, participants were given sufficient time to familiarize themselves with the tasks and select which hand they would use for tapping. The instructions on which finger to tap or rest were provided on a screen behind the participant's head, and were viewed through a mirror mounted on the head coil. The projection was presented using E-prime 1.0 (Psychology Software Tools, PA, USA). Each subject was instructed to tap the cued finger, but not the adjacent fingers, at their own pace. Functional analysis. Before functional analysis, the first 60 scans were excluded from the analysis to ensure that the magnetization reached equilibrium 15 . After coregistration of the T1WI structured data to the automated anatomical labeling (AAL) atlas 16 , the functional data was coregistered to the T1WI data. The transformations were then combined to identify the motor area in the functional data sets. In addition, linear trends in the time series were removed, and the noise level was reduced by applying a low-pass filter to each pixel. Spatial filtering was also applied using a Gaussian filter with σ = 1.5.
After this preprocessing, functional activation maps were obtained from the image time series by correlating the signal intensity time-course of each pixel with an on/off task design convolved with a canonical hemodynamic response function. SPM12 (revision 7219) 17 was used for the analysis. The cross-correlation (CC) coefficient was calculated for each pixel using www.nature.com/scientificreports/ where − → R x is the reference task design and − → R y is the signal intensity time-course of the pixel 15 . All image preprocessing and functional analysis was performed in MATLAB R2018b (Mathworks, Natick, MA, USA).
Deep learning-based super-resolution. Figure 1 depicts an overview of the proposed method. The STSS-SRfMRI scheme includes two unique ideas: first, it uses high spatial resolution static T2*WI as the training data; second, it applies subject-specific learning. As described in the introduction, the static T2*WI were used to introduce high spatial resolution information into the training process. Also, as functional signal changes are usually quite small, subject-specific learning was used to eliminate any anatomical variation that might be artificially introduced by including T2*WI data from other subjects.
Before training, the pixel intensity of the T2*WI training data was adjusted and scaled to match the intensity of the fMRI data. All 30 slices of the T2*WI data from each subject were used for training and validation to build a subject-specific model. The trained model was then applied to the fMRI data from the same subject.
The SRGAN used in this work was customized in several ways. Rather than using an up-sampling block in the generator G, the low resolution images were upscaled to a 128 × 128 matrix size using lanczos 3 interpolation 18,19 before being input. All the batch normalization layers were also removed 20 . A discriminator (D) was applied with the number of convolutional layers set to 10 to accommodate the size of the input. We implemented the modified SRGAN network using an adaptive moment estimation (Adam) optimizer with an initial decay rate of 0.9, a scaling factor of 2, patch size of 64, batch size of 2, an initial learning rate of 0.0001, and 100,000 iterations. The training images were the 30 slices of the corresponding T2*WI data. The experiments were implemented in PyTorch 1.1.0 on Ubuntu 16.04 LTS.
Identifying the neural activation-related region. The activation maps generated from the low-resolution fMRI data (the raw map) and from the processed output of the STSS-SRfMRI scheme (STSS-SR fMRI map), were compared based on how effectively they localized the activation region. For this purpose, the regions corresponding to the thumb and little finger activation tasks were separately identified for the raw fMRI and STSS-SRfMRI maps of each subject. First, a CC map was calculated for each input image series (i.e., the raw or STSS-SR data) for each subject and each activated finger. Second, the activation-related region in each CC map was defined as the region consisting of pixels having values equal to or above a threshold value, see Fig. 2. The threshold value was defined as Figure 1. Overview of the Static T2*WI-based Subject-Specific Super Resolution fMRI (STSS-SRfMRI) scheme proposed in this study. The upper and lower parts correspond to the training and testing phases, respectively. In the training phase, the generator (G) was optimized to form a relationship between the low-resolution and high-resolution T2*WI. The discriminator (D) made a decision whether the input was "real" (i.e., the reference high-res T2*WI) or "fake" (i.e., the generated high-res T2*WI). G learned to generate more realistic output via feedback from D. In the testing phase, a high-resolution functional MRI (fMRI) time series was reconstructed from the low-resolution fMRI data using the optimized generator, and subsequently a high-resolution functional map was calculated based on the high-resolution fMRI. www.nature.com/scientificreports/ The number of pixels included in the activation-related region of the raw fMRI map was compared to that of the STSS-SR fMRI map for each finger of each subject. As the STSS-SR fMRI maps had pixels that were four times smaller than those of the raw fMRI maps for the same sized area, the number of pixels in the STSS-SR fMRI maps was divided by 4 before comparison.
Independence of the extracted activated regions for the different tasks. The raw fMRI and STSS-SR fMRI maps obtained in the previous sub section were compared to determine which of them has a higher functional resolution for the thumb and little finger tasks. For this purpose, a Dice coefficient 21,22 was calculated for the extracted activation-related regions of the thumb and little finger for each subject (Fig. 3). This

Figure 2.
Overview of how the activation-related region was defined for each tapping task. First the activation maps were obtained from the raw and the Static T2*WI-based Subject-Specific Super Resolution fMRI (STSS-SRfMRI) image series (top row). Second, the top 25% between the max and minimum CC values was set as the threshold (middle row). Finally, the region consisting of pixels having values equal to or higher than the threshold value was defined as the activation-related region (bottom). Statistical analysis. The number of pixels included in each activation-related region, and the Dice coefficient calculated from the raw fMRI and STSS-SR fMRI maps were statistically compared using the Wilcoxon signed-rank test (p < 0.05 was considered significant). The EZR graphical interface to R version 3.5.2 25 , was used to make these statistical comparisons.

Results
Identifying the neural activation-related region. Figure 4 presents representative examples of the CC maps obtained via analysis of the raw unpreprocessed and STSS-SRfMRI processed data. The STSS-SRfMRI method appears to enhance the functional resolution. Figure 5 compares the number of pixels in the activationrelated regions of the motor areas corresponding to thumb-tapping and little finger-tapping. The activationrelated regions extracted from the STSS-SRfMRI maps had significantly fewer pixels than those extracted from the raw fMRI maps for both the thumb (p < 0.001) and little finger (p < 0.001) tasks. Figure 6 illustrates the activated regions corresponding to the finger-tapping tasks. The activated regions obtained using the STSS-SRfMRI scheme had less overlap compared to those obtained using the raw unpreprocessed data. Figure 7 shows the Dice coefficients for the extracted thumb-and little finger-tapping related regions. The Dice coefficients were significantly smaller for the STSS-SRfMRI scheme (p = 0.00466).

Discussion
In this study, we proposed a novel method based on a SRGAN that uses static T2*WI and subject-specific learning to improve functional resolution for fMRI. On visual assessment, the contrast of the activation map produced by the STSS-SRfMRI scheme was enhanced (Fig. 1). Quantitatively speaking, significantly fewer pixels were contained in the activation-related region derived from the STSS-SRfMRI processed data in comparison to the number obtained from the raw unpreprocessed data (Fig. 5). In addition, the Dice coefficients calculated for the activated regions corresponding to the two finger-tapping tasks were significantly lower for the STSS-SRfMRI processed data (Fig. 7). These results suggest that the STSS-SRfMRI method can improve functional resolution. The thumb and little-finger related activation areas were narrower and more distinct in STSS-SRfMRI produced maps (Figs. 5, 6). This was quantitatively supported by the Dice coefficient analysis, where the values were significantly lower for the STSS-SRfMRI scheme in comparison to those obtained for the raw fMRI results (Fig. 7). These results suggest that the STSS-SRfMRI scheme may help to distinguish thumb and little-finger related activations more distinctly compared to the raw fMRI results. Previous studies have investigated finger somatotopy at both 3T 26,27 and 7T 28 . While there was no gold-standard reference to verify the results at either www.nature.com/scientificreports/ field, it is likely that the 7T results will be more accurate because it is possible to image at higher resolution, which decreases the partial volume effect. Processing 3T fMRI data with the STSS-SRfMRI scheme might enable discrimination of activated areas that is comparable to that obtained using a 7T MRI scanner. As noted above, the Dice coefficient tended to be lower for the STSS-SRfMRI results. However, there were seven individual cases where the Dice coefficient was found to be larger for the STSS-SRfMRI result. Closer examination of these cases found that the Dice coefficient was larger for the following reasons: (i) For two subjects, there was some misregistration of the motor cortex with the reference image, leading to some high CC pixels in the motor cortex being incorrectly discarded. It was not clear why the misregistration occurred, but after expanding the motor area using a region growing method 29,30 the Dice coefficients were recalculated and found to be lower than the corresponding raw fMRI results. (ii) For one subject, although there were pixels within the brain with CCs over 0.5, the maximum CC in the motor area was less than 0.5. It is likely that in this case the subject did not adequately perform the tapping task. (iii) The activation area for one other subject was very broad, which suggests that accessory physical motion beyond the required task occurred. (iv) For three subjects, there were some artifacts in the T2*WI training images, which suggests that the corresponding SRGAN trained with those images was affected, and hence the generated STSS-SRfMRI images were defective.  www.nature.com/scientificreports/ Several studies have shown that using a SRGAN can improve the quality of medical imaging, and in particular MRI [11][12][13] . However, despite the improved appearance, few studies have suggested that MRI images reconstructed using a GAN are clinically or neuroscientifically significant 31 . An important feature of the present study is that the modified SRGAN not only generated acceptable higher resolution images, but maintained the embedded functional information.
Even though spatial filtering is widely used as a preprocessing step in the analysis of fMRI data, it could be argued that the super resolution networks in the STSS-SRfMRI scheme are just removing the smoothing effect of the filtering. To test this possibility the STSS-SRfMRI scheme was also applied to the unsmoothed data of all 30 subjects included in the final analysis (see Supplementary Fig. 2). It was found that the Dice scores without smoothing were lower for the STSS-SRfMRI processed data than for the raw fMRI data (0.417 (0.320-0.575) and 0.355 (0.238-0.457); data presented as median (interquartile range)). Although the median Dice scores for both schemes were lower than when filtering was used, a similar trend was found with the results of STSS-SRfMRI being significantly smaller than the raw fMRI results (p = 0.00000276).
One idea that could make the procedures proposed in this work more robust is to test the SRGAN trained for each subject on additional high-resolution T2*WI obtained from the same individual. Applying the STSS-SRfMRI scheme to the extra data would provide a first assessment of the accuracy of the results. Unfortunately, this idea was not applied in the present study because only one T2*WI data set was available for each subject.
One limitation of the present study is that there was no gold standard reference to verify the high-resolution functional maps generated using the proposed STSS-SRfMRI scheme. In the example shown in Supplementary  Fig. 3, after analysis of the STSS-SRfMRI data the CC map for the thumb-tapping task appears to consist of several clusters of highly correlated pixels, whereas this feature was not observed for the raw fMRI maps. A previous study has determined that the activation regions in the primary motor cortex overlap for distinct movements of the fingers, wrist, and elbow 32 . Hence, it is possible that the clusters in Supplementary Fig. 3 reflect accessory movement during the thumb-tapping task. The absence of a gold standard reference prevented us from assessing whether this hypothesis was true or if it was simply an error due to the STSS-SRfMRI scheme generating incorrect EPI images.
Another possible limitation was that the T2*WI images obtained for subject-specific training were in 2D, which meant that a 2D GAN had to be used instead of a 3D GAN. As neural activity in the brain occurs in some 3D volume of tissue, a similar study using 3D images could increase the performance of STSS-SRfMRI in the future. Finally, as only healthy volunteers participated here, it was uncertain whether the proposed method is applicable for patients with neurological disorders. Clinical cases need to be studied in the future.

Conclusions
In conclusion, we proposed a novel application of SR for fMRI using static T2*WI for training and applying subject-specific learning. The results suggest that the STSS-SRfMRI scheme has the potential to enhance the functional resolution of 3T fMRI by adequately increasing the spatial resolution of the original fMRI images.

Data availability
The data supporting the findings of the current study are available from the corresponding author on reasonable request.