Feasibility study of immersive virtual prism adaptation therapy with depth-sensing camera using functional near-infrared spectroscopy in healthy adults

Prism Adaptation (PA) is used to alleviate spatial neglect. We combined immersive virtual reality with a depth-sensing camera to develop virtual prism adaptation therapy (VPAT), which block external visual cues and easily quantify and monitor errors than conventional PA. We conducted a feasibility study to investigate whether VPAT can induce behavioral adaptations by measuring after-effect and identifying which cortical areas were most significantly activated during VPAT using functional near-infrared spectroscopy (fNIRS). Fourteen healthy subjects participated in this study. The experiment consisted of four sequential phases (pre-VPAT, VPAT-10°, VPAT-20°, and post-VPAT). To compare the most significantly activated cortical areas during pointing in different phases against pointing during the pre-VPAT phase, we analyzed changes in oxyhemoglobin concentration using fNIRS during pointing. The pointing errors of the virtual hand deviated to the right-side during early pointing blocks in the VPAT-10° and VPAT-20° phases. There was a left-side deviation of the real hand to the target in the post-VPAT phase, demonstrating after-effect. The most significantly activated channels during pointing tasks were located in the right hemisphere, and possible corresponding cortical areas included the dorsolateral prefrontal cortex and frontal eye field. In conclusion, VPAT may induce behavioral adaptation with modulation of the dorsal attentional network.


Methods
Participants. Inclusion criteria for this study were healthy individuals aged between 18 and 50 years, right handed-assessed by the Edinburgh handedness inventory 33 , feasibility to wear the Oculus Rift DK2 (Oculus VR, LLC, CA, USA), and the ability to detect objects in immersive VR. Subjects with any history of disease involving the central nervous system such as stroke, traumatic brain injury, or Parkinson's disease were excluded. All subjects provided written informed consent for participation. Although this was a pilot study, we calculated sample size. Thirteen subjects were needed to detect a 1° pointing bias with standard deviation of 0.9° after VPAT (with β = 0.20 and α = 0.05) based on a previously published study in healthy subjects with PA 34 . Fourteen healthy subjects (seven male, seven female), aged between 22 and 39 (27.8 ± 4.1) years, participated in this study. This study was carried out according to the Declaration of Helsinki and Good Clinical Practice Guidelines, and the protocol was approved by the Seoul National University Bundang Hospital Institutional Review Board. This study protocol and preliminary behavioral results, without fNIRS data in four study subjects, were published previously 35  www.nature.com/scientificreports/ Concept of the VPAT system. VR systems with head mounted displays provide immersive visual feedback. Hand tracking with a head mounted display eliminates the gap between the virtual hand location and the location of the virtual object in virtual space. This clearly differs from systems using a 2D or 3D display, stylus, and mouse. Synchronization of the coordinate system of VR contents and hand tracking coordination can induce more realistic interactions with virtual objects. It is also possible to intentionally misalign the virtual hand and virtual object during the pointing action. This may be similar to a visual field shifting in the PA paradigm using prism goggles. However, while prism goggles shift all visual fields, our VPAT system only shifts the virtual hand trajectory in the immersive VR. The virtual hand shift using our VPAT system induces initial pointing errors. In the condition without VPAT mode (no virtual hand shift), the subject directly points the virtual target (Fig. 1a). During the early VPAT period, the subject's virtual hand is shifted to the right from the VR target (Fig. 1b), which might make the subject perceive the real target location to the left of the initial pointing location. During the late VPAT period, the subject adapts to VPAT mode and the pointing errors of the virtual hand to the target diminish and finally disappear (Fig. 1c). After the VPAT mode is switched off, the subject's real hand will point to the left of the target (Fig. 1d), if the adaptation effect of VPAT is similar to conventional PA therapy.
Hardware and software design of the VPAT system. The (Fig. 2a). Before projecting the user's hands into the virtual space in real time, the hand skeleton dummies detected by the Leap Motion sensor are rotated based on the longitudinal axis for the visual hand shift.
The VR environment was created with Unity3D version 5.4.3 (Unity Technologies, CA, USA, https:// unity 3d. com). The environment is based on a general living environment and composed of an office with a desk (Fig. 2b). The objects used in the environment were constructed to avoid interfering with the view or interaction with the target. The red cross marked on the bookshelf is used to calibrate the body and center of vision. A white ball is used as the target for pointing. The remaining trial time is indicated by a health bar, and to the right is an icon showing the current task.
The device was calibrated to determine where the target was located and visible. Figure 2c shows feedback that can only be seen when the index finger is within the visible area. The virtual hand is only visible beyond the invisible area shown in Fig. 3b. Even if the index finger does not exactly touch the target, the virtual hand becomes visible when the index finger reaches above the target's z distance. The target is a 6 cm diameter ball hanging from a rod. The virtual hand is not displayed until it is within a visible area, set to a depth similar to the target. When pointing at a target in a trial, the hand is prevented from being controlled accurately via real-time visual feedback.
The location of a target object on a virtual table is defined as angle θ from the center of view. In the initial calibration phase, the distance of the object is specified as a comfortably reachable distance allowing the user to touch them (Fig. 3a). First, the coordinate systems between the head and hand should be co-aligned. The coordinate system of the visual hand is transformed by the angle ϕ in relation to the head center (Fig. 3b). If a  www.nature.com/scientificreports/ point from the HMD coordinate system is p x|hmd and the point in the hand sensor coordinate system is p x|hand , the relationship is given by the following equation: where the matrix consisting of R and d is transformation matrix, representing the physical transformation between hand tracking sensor and HMD. The hand tracking sensor is attached to the front of the HMD without rotation. R is a 3 × 3 unit matrix where there is no rotation of sensor axes, and d is the relative position of the hand tracking sensor from the HMD. Second, the experimental condition is given by R shift and d shift , which indicate rotation matrix and translation vector for hand shift, respectively. The hand shift was applied using the following equation: In our experiment, the hand only rotated horizontally. So, R shift is and d shift = 0.
Link between the VPAT system and fNIRS. The VPAT system was connected to the fNIRS system, NIRScout (NIRx Medical Technologies, LLC, MN, USA). The VPAT signal acted as a trigger for recording in the fNIRS system. The VPAT system was implemented with Unity 3D. The remote keyboard control software using TCP/IP communication was also implemented to synchronize the starting event with the fNIRS system. The fNIRS system is controlled by Superlab 5.0 (Cedrus Corporation, San Pedro, USA, https:// cedrus. com/ super lab/) as an event trigger method. The remote command key in a computer where Superlab 5.0 is installed was used to initiate the fNIRS recording.
Experimental design. Our protocol of video-recorded experiments was extensively documented previously 35 . Briefly, subjects sat comfortably and wore the VPAT system. They pointed the target in the VPAT system without the prism mode for simple familiarization where they experience target shifts in the prism mode.
Then the fNIRS cap with optodes in the montage set for this study was added. The experimental design is shown in Fig. 4. The experiments started from Phase 1, no VPAT mode. Four pointing and four clicking blocks appeared alternatively for 4 min. Phase 2 and Phase 3 were conducted with VPAT mode (Phase 2: 10° deviation, Phase 3: 20° deviation); each phase was composed of five pointing blocks and five resting blocks, which appeared alternatively for 5 min. Phase 4 was conducted without VPAT mode and represents the post-prismatic adaptation period. Phase 4 consists of five pointing and clicking blocks alternatively appearing for 5 min. In the clicking block, subjects were instructed to click the button as soon as possible when the visual target appeared at 3 s intervals, using their right index finger. Each clicking block consisted of 10 clicks for 30 s. The pointing task block was composed of 10 pointing for 30 s in total. Subjects were instructed to point the visual targets with the right index finger as fast as possible. Clicking represents a task requiring attention but minimal motion, and pointing represents a task with motion and same level of attention as the clicking. Therefore, the cortical activation during pointing in contrast to clicking may reveal more activation on the left primary motor cortex. The visual target was presented on a position deviated 10° -right or 10°-left from the midline at 3-s intervals and in random orders. When the subject correctly pointed the target, the color changed to red. fNIRS measurement. The continuous-wave type fNIRS system, NIRScout was used to measure changes in oxygenated hemoglobin [HbO] caused by cortical activations during the experiment. The manufacturer's NIRStar 14.1 program was operated to control the NIRScout. The fNIRS system uses two wavelengths of nearinfrared light, 760 and 850 nm, for measurement and records raw optical density data with a sampling rate of 4.17 Hz. A total of 39 measurement channels were formed by positioning 15 LED source and 13 detector optodes (optical probes). The optode positions were designed to mainly cover the dorsal frontoparietal network and the www.nature.com/scientificreports/ motor cortices of both hemispheres (Fig. 5), based on the international 10-20 system 36 . The separation between each source and detector was approximately 3 cm, enabling the near-infrared light to reach cortical areas. For optode fixation, textile EEG caps (EASYCAP, Herrsching, Germany) in three different sizes (54, 56, and 58 cm circumference) were prepared and selected depending on subject's head size. To ensure the correct positioning of the applied optodes during the experiment, three researchers attended and checked the positions of four anatomical markers (Nasion, Inion, Left and right pre-auricular points (PAL and PAR respectively)) before and after the experiment for each subject.
Pointing error analysis. All hand trajectory data was recorded with a sampling rate of 60 Hz during the experiment. The pointing error was calculated from the initial calibration position of HMD on the xz plane to the angle between the virtual hand pointing endpoint and the target position. The first pointing errors during each block in the four phases were used for analysis. They are presented using a box and whisker plot. Multiple Wilcoxon signed-rank tests were used to test the differences between six time points (three transitions between phases and three time points between the first and last block in phases 2 to 4). A 2-tailed Bonferroni correction, with a p < 0.008 considered statistically significant, was performed to adjust the type I error due to multiple comparisons. Statistical analyses were performed using the PASW statistical package (SPSS version 18.0, SPSS, https:// www. ibm. com/ analy tics/ spss-stati stics-softw are). [HbO] levels were used for analysis because it is more sensitive to the cortical blood flow changes according to the task and deoxyhemoglobin has shown considerable individual differences in task-related changes 40 . General linear model (GLM)-based statistical data analysis for the task "pointing" was performed. In Level-1 analysis (within-subject), the GLM coefficients of [HbO] in each channel were estimated with the canonical hemodynamic response function (HRF) after temporal filtering by discrete cosine transform function and, in turn, precoloring by HRF was implemented. Then, the three t-contrasts were specified for the comparisons of the pointing blocks in each of the three phases (VPAT-10°, VPAT-20°, Post-VPAT) against the pointing blocks in the default phase, pre-VPAT. Subsequently, for each subject, SPM t-maps were computed based on those t-contrasts with p < 0.05 (uncorrected). In Level-2 analysis (group), group t-maps for the three comparisons were generated with t-tests to identify the channels most significantly activated in both positive and negative directions by pointing, in contrast to pre-VPAT pointing (VPAT-10° vs. Pre-VPAT, VPAT-20° vs. Pre-VPAT, Post-VPAT vs. Pre-VPAT). The most positive channel indicates the area most significantly activated by a specific phase against Pre-VPAT and the most negative channel indicates the most significant deactivation. All procedures were performed with Statistical Parametric Mapping (SPM, version 8, https:// www. fil. ion. ucl. ac. uk/ spm/ softw are/ spm8/) 41 , integrated within the nirsLAB program.
Ethics approval and consent to participate. Written, informed consent was obtained from each subject. The Seoul National University Bundang Hospital Institutional Review Board (IRB) approved this study protocol (IRB number: B-1605/345-006).

Results
Pointing errors during the experiment. Pointing errors in each block were presented using a box and whisker plot (Fig. 6). Pointing errors were around 0° in the pre-VPAT phase. In the VPAT-10° phase, the errors initially deviated towards the right with a median value of 6.32° and interquartile range (IQR) of 3.00° and gradually decreased to 2.41° (IQR: 2.33°). At the start of the VPAT-20° phase, pointing errors moved rightward 8.10° (IQR: 3.00°) and decreased to 4.43° (IQR: 4.23°). In the post-VPAT phase, there was a substantial leftward pointing error (8.48°, IQR: 5.69°) in the first block of pointing, decreasing gradually afterward. The differences in pointing errors between the three phase transitions were statistically significant (p < 0.008), as were the changes between the first and last block in the three phases (VPAT-10°, 20° and post-VPAT phase) (p < 0.008) (Fig. 6). The average number of outliers was 0.83 (Standard deviation-1.40) in each block, and these were excluded   (Table 1 and Fig. 7). The most significantly activated channels were all located in the right hemisphere, and possible corresponding cortical areas include the dorsolateral prefrontal cortex and the frontal eye field (Brodmann areas 8, 9 and 46) 42 . The most deactivated channels were located around the left intraparietal sulcus. The cortical activation maps during pointing in each phase are presented in the supplementary Fig. S1.

Discussion
The aim of this study was to examine the feasibility of VPAT by assessing after-effect and cortical activation measured with fNIRS. After-effect, as measured with pointing errors, exhibited a similar response to conventional PA as reported in past papers. The most significantly activated channels determined by fNIRS during VPAT were right dorsolateral prefrontal cortex and the frontal eye field (Brodmann areas 8, 9 and 46), while the most deactivated channels were located around the left intraparietal sulcus. The mechanisms used in our VPAT system differ from real prism therapy, in that in conventional PA, the whole visual field is shifted to the right. However, in our VPAT system, only the virtual hand trajectory is deviated to the right, while the actual target remains in its real position (Fig. 3) 43 . This difference in mechanism occurs due to the nature of immersive-VR and was found in the previous studies on VPAT using HMD 27,29,30 . According to Redding and Wallace 14 realignment during PA occurs in an additive manner regarding visual and proprioceptive realignment, which is mainly responsible for after-effect from PA 13 . Newport et al. 44 separated visual and proprioceptive realignment using real prism therapy and found that proprioceptive realignment alone was able to induce after-effect. On this basis, we hypothesized that VPAT, which induces proprioceptive realignment, can generate after-effect.  The open-loop pointing errors in post-VPAT adaptation at 20° deviation was 8.48° in our study, which is similar to a previous study performed in healthy individuals 27 . Whether the size of after-effect induced by VPAT is comparable to the conventional PA remains controversial. The after-effect of conventional PA reported by Newport et al. 44 and Michel et al. 45 were larger, while Ramos et al. 28 compared conventional PA with two types of VPAT and demonstrated that VPAT can generate a larger after-effect than conventional PA. Moreover, after-effect measured with open-loop pointing error is considered to be strongly related to transfer 15 ; however, findings on the relationship between after-effect and transfer using VPAT in healthy individuals 27,28 and stroke patients with USN 30 were not consistent with findings in conventional PA. Gammeri et al. demonstrated a dose-response effect of VPAT as only participants adapted to 30° showed transfer 27 , while Bourgeois et al. have shown no transfer in the patients with USN although after-effect was observed 30 . Since our study did not investigate transfer, we could not conclude whether after-effect induced by VPAT can lead to transfer. These findings raise the question of whether the quality of after-effect and underlying neural mechanism of VPAT is comparable to that of conventional PA, and further study is required to elucidate the relationship between after-effect and transfer in VPAT.
Investigation of the relevant cortical area related to VPAT was another objective in this study. Since the induction mechanism of behavioral adaptation in our VPAT may be different from conventional therapy using a real prism, comparison of the activated cortical areas with previous studies on cortical activations with a real prism should be scrutinized. Although no other studies have used fNIRS to assess cortical activation during conventional PA, studies using fMRI 34,46,47 have shown that both parietal cortex and cerebellum are activated during the recalibration and realignment process in PA, while the cognitive effect of PA by bottom-up activation occurs in temporo-occipital and prefrontal areas 34,48 . In our study, the most activated areas during and after VPAT included the right frontal eye field and the dorsolateral prefrontal cortex (Table 1 and Fig. 7) 42 , although without statistical significance in the VPAT-10° and post-VPAT periods.
The frontal eye field is an area involved in the dorsal attentional network, which mediates top-down stimulus-response selection 49 , producing inattention to an object in the contralesional far-space when that area is damaged 50 . A recent functional MRI study found that rightward PA increased resting functional connectivity in healthy adults between the right frontal eye field and right anterior cingulate cortex after PA, which may mean more attention to the left visual field 51 .
In our study, the activated area was the right dorsolateral prefrontal cortex, not the left. A previous functional MRI study in patients with right hemispheric damage showed that rightward PA increased activation in the left prefrontal cortex 48 . It was postulated that PA induced bottom-up activation of the prefrontal cortex, which then led to the enhanced function of the left hemisphere to perceive the left visual field 16 . However, the role of the left prefrontal cortex on PA's effect has not been well elucidated. Bottom-up stimulus driven control is known to be related to the ventral attentional network usually including the ventral prefrontal cortex, not the dorsal one 52 . In addition, despite an ongoing debates, the dorsolateral prefrontal cortex is considered to be included in the dorsal attentional network 52,53 . The shift from using the left ventral attentional network by PA, secondarily activates the right dorsal attentional network to perceive the left visual field, thus in our study, VPAT may induce the activation of the dorsal attentional network including both the right dorsolateral prefrontal cortex and frontal eye field. The various subject characteristics (right prefrontal cortex damaged patients vs. healthy adults) could also result in the different activation side in the prefrontal cortex between our study and that by Croattaz-Herbette et al. 48 . Due to these uncertainties in the role of the prefrontal cortex on PA and the difficulties in directly measuring the hemodynamic changes of the deeply located ventral prefrontal cortex and cerebellum by fNIRS, further research is warranted.
Application of immersive VR to PA has several advantages compared to conventional PA: (1) gradual adjustment of deviations according to the patient's adaptation 27 , (2) easier adjustment of deviation angles 28 , (3) possibility of combination with other therapeutic devices 29 , (4) blockage of external visual cues, and (5) quantification and monitoring of pointing errors during therapy. The VPAT system can help patients and therapists set up treatment environments in the patient's home, with monitoring or self-guided algorithms (e.g., automatically adjusting the deviations according to the patient's behavioral adaptation). Despite these possible advantages of VPAT, further study on VPAT is still required as the neural mechanism of adaptation in VPAT has not yet been elucidated. Moreover, evaluation of the transfer effect of VPAT on visuo-motor cognitive domain and alleviation of USN needs to be performed to establish VPAT as an alternative therapeutic tool to conventional PA in the treatment of USN in stroke patients.
This study has several limitations. First, the study did not assess the transfer of adaption effect. Considering that the purpose of PA is to ameliorate USN, transfer to various visuo-motor cognitive domains needs to be examined in future studies. Second, the exact localization of involved cortical areas using fNIRS was difficult without co-registration of each channel with an individual magnetic resonance image (MRI) or mapping to a normalized brain based on the digitization of anatomical markers. We attempted to alleviate this limitation by following the 10-20 system, using an appropriately sized cap according to the subject's head size and identifying channels able to measure areas of interest by applying the fNIRS optodes' location decider program 42 . In addition, deeply located structures possibly related with the effect of PA such as the cerebellum 34,46 and anterior cingulate cortex 54 could not be directly monitored by fNIRS, which hindered us from investigating the cortical network underlying VPAT effect, such as the cerebello-parietal network 16 . Third, the contamination influence from extracerebral layers, such as Mayer's wave, was not eliminated. This complicated the interpretation of the results. With the current fNIRS system with a source-detector separation of 3 cm, global regression was a recommended option. However, this could not be applied because the unequal weight distribution and the fixation band of the VPAT device might violate the assumption that the extracerebral contamination was applied equally across all channels. Fourth, the single statistically activated channel was only found in the VPAT-20° condition, although most significantly activated channels were consistently located around the right dorsolateral prefrontal cortex and frontal eye field. This can be explained by VPAT's dose-response effect as previously shown by www.nature.com/scientificreports/ Gammeri et al. 27 In addition, a total of 50 pointing with 30 s resting time after each 10 pointing were used in our experiment, which was lower than the number of consecutive pointing (one hundred) in previous VPAT studies 27,30 . Finally, in situations where the hand was not detected properly due to occlusions or self-occlusions, there were often cases where the hand did not appear, even though it actually touched the virtual object during VPAT. Using VR controllers would reduce this error. However, fatigue might occur due to the holding pose and the controller weight. Hand tracking technology using multiple cameras on the front of the HMD could reduce errors caused by occlusion without fatigue 55 .

Conclusions
The VPAT system, which deviated the hand trajectory rightward induced after-effect. The most activated cortical areas during and immediately after VPAT were the dorsolateral prefrontal cortex and the frontal eye field, which are associated with the dorsal attentional network. Our findings demonstrated the feasibility of VPAT in producing after-effect and identified possible neural networks activated during the adaptation process, However, whether the underlying neural mechanism of VPAT and its therapeutic effect are comparable to that of conventional PA still requires further scrutiny. Future clinical trials using VPAT with a high degree of rightward deviation with multiple sessions over longer periods, assessment of transfer to visuo-motor cognitive domain, and comparison with conventional PA is required in stroke patients with USN.

Data availability
The datasets generated and analyzed during the current study are available on the supplementary data.