A Novel Approach to Sensorimotor Skill Acquisition Utilizing Sensory Substitution: A Driving Simulation Study

The aim of this study is to demonstrate the potential of sensory substitution/augmentation (SS/A) techniques for driver assistance systems in a simulated driving environment. Using a group-comparison design, we examined lane-keeping skill acquisition in a driving simulator that can provide information regarding vehicle lateral position by changing the binaural balance of auditory white noise delivered to the driver. Consequently, lane-keeping accuracy was significantly degraded when the lower visual scene (proximal part of the road) was occluded, suggesting it conveyed critical visual information necessary for lane keeping. After 40 minutes of training with auditory cueing of vehicle lateral position, lane-keeping accuracy returned to the baseline (normal driving) level. This indicates that auditory cueing can compensate for the loss of visual information. Taken together, our data suggest that auditory cueing of vehicle lateral position is sufficient for lane-keeping skill acquisition and that SS/A techniques can potentially be used for the development of driver assistance systems, particularly for situations where immediate time-sensitive actions are required in response to rapidly changing sensory information. Although this study is the first to apply SS/A techniques to driver assistance, further studies are however required to establish the generalizability of the findings to real-world settings.

augmentation in healthy individuals, which allows for the acquisition of novel or even non-innate sensory modalities and perceptual skills. For example, Nagel et al. 9 proposed a waist belt-type vibration device that constantly indicates the magnetic north using a built-in digital compass. They demonstrated that individuals after six weeks of training could successfully learn to use the novel sense of magnetic orientation for navigation in outdoor environments. In another example, Konttinen et al. 10 developed an auditory feedback system to augment the somatosensory information received during rifle shooting. The system provides real-time auditory feedback whose frequency corresponds to the aiming error measured during shooting. After four weeks of training using the system, participants showed better shooting performance when compared with participants who received the same amount of training without the auditory feedback.
Here, we propose a novel application of a sensory substitution/augmentation (SS/A) technique designed to assist drivers. Modern cars are loaded with various kinds of sensors that measure several features of the surrounding traffic environment, which are used to provide safety warning signals to drivers. To date, however, no assistant systems have been designed to augment the sensory information processing capacity of drivers. The use of SS/A techniques constitutes a promising avenue to successfully develop such systems. The most significant challenge for applying SS/A techniques to driver assistance, however, may be related to time constraints. Driving a car is an everyday sensorimotor skill that demands immediate actions in response to ever-changing contextual traffic situations, which require the real-time integration of multisensory (though mostly visual) information. Even in the simplest situation where one is driving a car on a road without any other cars or obstacles, the driver must keep the car in the correct lane by constantly using visual information from the lower visual scene (proximal part of the road) and must negotiate slight curves using visual information from the upper visual scene (distant part of the road) [11][12][13] (see also the recent review by Lappi and Mole 14 for a more in-depth discussion). To our knowledge, the online use of SS/A techniques under such high time pressure constraints has not been validated in previous applications.
In the current study, we thus aimed to demonstrate the potential application of SS/A techniques in the development of driver assistance systems. Specifically, using a group-comparison design, we examined lane-keeping skill acquisition in a driving simulator that can provide additional information regarding vehicle lateral position within a lane via the binaural balance of an auditory stimulus. If auditory cueing of vehicle lateral position can be used to compensate for occluded visual information that is necessary for lane-keeping (i.e., proximal part of the road), it is predicted that participants with SS/A assistance will show lane-keeping performance comparable to that of participants without SS/A assistance but that have access to the visual information necessary for lane keeping. It should be noted here that these experimental configurations were designed to test the applicability of SS/A techniques to driver assistance using a well-studied occlusion paradigm 12,13 , rather than to show whether the auditory cueing of vehicle lateral position is actually useful for lane-keeping in real traffic situations. Driving simulator. Our driving simulator consisted of a fixed-base cockpit (GTD-SPECi, Rossomodello Co., Ltd., Tomioka, Japan), a force-feedback steering device (T500RS, Guillemot Corp., Carentoir, France), and a 60-inch LCD monitor (LC60XL10, SHARP Corp., Sakai, Japan) located in front of the cockpit (Fig. 1A). In this configuration, the monitor was viewed from a distance of approximately 115 cm. The custom-made software computed vehicle position and direction based on vehicle dynamics combined with steering inputs, and then displayed traffic scenes from a driver's point of view on the monitor. The software could also occlude an arbitrary part of the visual scene, in addition to providing binaural auditory white noise (sampling frequency = 44.1 kHz) via headphones (HD 630VB, Sennheiser electronic GmbH & Co. KG, Wedemark, Germany). The simulation time step was 1/60 s. procedure. The experiments were conducted in a quiet, dimly lit room. All participants were given a pretest session to familiarize themselves with the driving simulator. Specifically, they performed a lane-keeping task in which they were asked to keep the vehicle in the centre of a winding road course by using the steering wheel. In this task, the winding road course was defined by left and right lane marks that produced a lane width of 3.5 m (Fig. 1B) and was composed of alternating left and right bends with a random curvature between 1/180 to 1/80 m −1 , interleaved with straight sections having a random length between 50 to 150 m. The vehicle automatically travelled with a constant speed of 80 km/h, which was empirically shown to be not too difficult for participants when negotiating curves in the present experimental set-up, and, therefore, no pedal operations were needed. In addition, no other road users were present and white noise with a constant level (−50 dB attenuated from the maximum level that was determined to be tolerable in a preliminary test to ensure participant comfort) was presented binaurally via headphones (Fig. 1B). The pretest session was composed of five one-minute trials with short breaks in between. After the pretest session, participants were randomly assigned to one of three experimental groups (15 participants per group): normal driving (ND) training condition, visual occlusion (VO) training condition, and sensory substitution (SS) training condition. The ND training group performed the lane-keeping task in an identical manner to that in the pretest session (Fig. 1B). For the VO training group, the lower visual scene (proximal part of the road) was occluded ( Fig. 1C) to restrict the available visual cues necessary for the estimation of vehicle lateral position. For the SS training group, while the lower visual scene was also occluded, information regarding vehicle lateral position was provided via binaural auditory stimuli (Fig. 1D). More specifically, a leftward (rightward) deviation from the lane centre was signalled by increasing the sound level of the white noise in the left (right) ear and by decreasing it in the right (left) ear with a constant gain of 25 dB/m; in the lane centre, the sound level was equal in both ears (−50 dB). For all groups, the training session comprised 40 one-minute trials, with short breaks provided in between trials. Following the training session, participants completed a posttest session that was identical to the pretest session (i.e., five trials of the ND training condition). The entire experiment, including set-up and rest breaks, took approximately 60 min to complete.

Data analysis.
For each trial, driving performance was evaluated using two metrics 13 : accuracy, which was defined as the standard deviation of lateral position (SDLP) of the vehicle, and stability, which was defined as the mean maximum steering wheel velocity (SWV) during curve negotiation, averaged across all curves. For both performance metrics, lower values represented better performance. According to previous studies 12, 13 , the accuracy metric reflects compensatory steering control that makes use of the visual information provided by the proximal part of the road, whereas the stability metric reflects anticipatory steering control that makes use of the information provided by the more distant part of the road. These findings demonstrate that the occlusion of the lower visual scene (proximal part of the road) compromises the accuracy but not the stability metric 12,13 .
The two performance metrics were separately submitted to two-way repeated measures analysis of variance (ANOVA) tests with the trial (first and last) as a repeated measure and with training condition (ND, VO, and SS) as a between-subjects factor. In addition, to examine whether the use of SS/A assistance for lane-keeping adversely affected ordinary driving skills acquired without assistance, the performance metrics were also compared between the last trial in the pretest session and the first trial in the posttest session for each training condition, using a two-way repeated measures ANOVA test with session (pretest and posttest) as a repeated measure and training condition (ND, VO, and SS) as a between-subjects factor. For all ANOVA tests, degrees of freedom were adjusted for sphericity using the Greenhouse-Geisser correction 16 . When a significant interaction was found, further simple main effect analysis and, if necessary, subsequent multiple comparisons using Shaffer's modified sequentially rejective Bonferroni procedure 17 were performed. A significance threshold was set at P < 0.05 for all tests.
ethics approval and consent to participate. All experiments were conducted in accordance with a protocol approved by the RIKEN Research Ethics Committee (Wako3 28-17 (4)). Figure 2 shows driving data obtained from a representative participant for each training condition. In the ND training condition ( Fig. 2A), vehicle lateral position appeared to be relatively constant from the last trial of the pretest session to the last trial of the training session. In the VO training condition (Fig. 2B), as a result of the visual occlusion of the proximal part of the road, vehicle lateral position was found to be unstable throughout the entire training session compared with the last trial of the pretest session. In the SS training condition (Fig. 2C), increased variation in vehicle lateral position due to visual occlusion was also observed in the first trial of the training session (Fig. 2C). However, in the last trial of the training session, the participant again became able to control the vehicle more accurately, suggesting that auditory cueing was sufficient to enable proper vehicle control. In contrast, there seemed to be no remarkable changes in steering velocity throughout the trials. Figure 3 shows the average temporal trajectory of both driving performance metrics for each training condition. In the pretest session, SDLP rapidly decreased for all conditions as trials were repeated (Fig. 3A). No systematic changes were observed for SWV (Fig. 3B). At the beginning of the training session, SDLP in the VO and SS training conditions increased due to, as expected, the loss of visual cues used for the estimation of vehicle www.nature.com/scientificreports www.nature.com/scientificreports/ lateral position (Fig. 3C). However, SDLP was found to decrease in the SS, but not in the VO training condition, as the training progressed and eventually reached a comparable level to that observed for the ND training condition by the end of the session (Fig. 3C). In contrast, SWV in the VO training condition tended to be lower than that observed in the ND and SS training conditions during the later trials of the training session (Fig. 3D). In the posttest session (Fig. 3E, F) (Fig. 4B).

Results
Moreover, an ANOVA on SDLP pretest-posttest differences showed a significant interaction between session and training condition [F(2, 42) = 9.90, P < 0.001, η p 2 = 0.32] (Fig. 5A). Subsequent post-hoc tests revealed no significant simple main effect of training condition for the pretest session [F(2, 42) = 0.33, P = 0.72, η p 2 = 0.02], but did however reveal a significant simple main effect of training condition for the posttest session [F(2,  (Fig. 5B). Importantly, no obvious adverse interference effects of SS training were found for either the SDLP or SWV metrics when compared to those obtained during normal (vision-based) driving.
To summarize, as hypothesized, lane-keeping accuracy was significantly degraded once the lower portion of the visual scene was occluded in the VO and SS training conditions. However, in the SS but not the VO training condition, vehicle lateral position stability was improved as the training progressed and the participants learned to use the auditory cueing. In addition, training with auditory cueing did not affect lane-keeping skills previously acquired based on intact visual information. www.nature.com/scientificreports www.nature.com/scientificreports/

Discussion
There is increasing evidence that sensory substitution devices can be used not only for sensory prostheses designed for sensory-impaired individuals but also for sensory augmentation purposes in healthy individuals. However, no study to date has examined whether such SS/A techniques can be used to develop driver assistance systems.  www.nature.com/scientificreports www.nature.com/scientificreports/ Successful validation of SS/A-based driver assistance may extend the applicability of SS/A techniques to situations where immediate time-sensitive actions are required in response to rapidly changing sensory information.
In the present study, to demonstrate the potential applicability of SS/A techniques for the development of driver assistance systems, we examined the impact caused by auditory cueing of vehicle lateral position on lane-keeping skill acquisition in a simulated driving environment. We found that lane-keeping accuracy was initially degraded when the lower visual scene (proximal part of the road) was occluded. However, performance was found to recover when auditory cueing of vehicle lateral position was provided. This indicates, on the one hand, that the proximal part of the road is informative of vehicle lateral position and is critical for proper lane keeping, which is a robust replication of previous findings by Land and Horwood 12 and Frissen and Mars 13 . On the other hand, and perhaps more importantly, these findings indicate that auditory cueing of vehicle lateral position provides sufficient information for proper lane keeping in the absence of the necessary visual information. Taken together, these results suggest that auditory cueing of vehicle lateral position is sufficient for lane-keeping skill acquisition and that SS/A techniques can be incorporated into driver assistance systems to support situation awareness, particularly in situations where rapid motor responses are required, such as requests to intervene in self-driving cars. This constitutes an important and novel finding, as the majority of prior SS/A studies examined performance in more static situations not requiring rapid responses 18-21 . Contrary to accuracy, steering stability was not found to vary as a function of the different training conditions used in the present study. These findings are in agreement with a theoretical framework proposed by Donges 11 , which proposes that steering accuracy and stability are determined by two independent control processes. The first is a compensatory process that enables accurate steering control by using the visual information from the proximal part of the road (i.e., vehicle lateral position), and the second is an anticipatory process that enables stable steering control by using the visual information from the distant part of the road (i.e., curvature). In the present study, although the visualization of the proximal part of the road varied between training conditions, the distant part of the road was always visible for all training conditions. Therefore, the present results are consistent with the two-level theoretical framework of steering control, and also further suggest that auditory cueing of vehicle lateral position is important for the compensatory but not the anticipatory steering control process.
One concern regarding the application of SS/A techniques to healthy individuals was related to the possibility that the acquisition of newly acquired sensorimotor skills might interfere with an existing (intact) sensorimotor skill. In fact, it is well established that the consolidation of a motor skill is disrupted by a subsequently acquired motor skill, particularly when the latter is learned immediately after the former [22][23][24] . Nonetheless, the present study shows that training with a SS/A driver assistance system for lane-keeping did not adversely affect lane-keeping skills previously acquired. This indicates that sensory-motor skill may be robust against changing the source of sensory information insofar as it has sufficient accuracy.
Although our data indicate that SS/A techniques can be applied to the development of driver assistance technologies, its generalizability to real-world settings is still rather limited. One limiting factor in particular relates to the use of a driving simulator. Indeed, we tested the driver assistance feedback under optimal driving conditions where there were no pedestrians and oncoming cars. In real-world traffic environments, however, drivers are required to pay attention to various kinds of related and/or unrelated events on the road [25][26][27] . Such situations where attention must be carefully deployed might interfere with the use of substituted/augmented sensory information, or, conversely, the use of SS/A assistant system might interfere with the appropriate deployment of attention required for safe driving. In addition, we conducted all experiments under a constant vehicle speed. Thus, the generalizability of the current findings to different or varying speed conditions will need to be further investigated. Another limiting factor relates to the SS conversion rule used. In the present study, we examined auditory cueing of vehicle lateral position using a natural conversion rule, where approaching the left (right) lane mark is represented by an increase in the auditory stimulus intensity in the left (right) ear. We cannot conclude from the current findings whether SS/A driver assistance systems would also work when adopting a more artificial conversion rule that, for example, could translate vehicle lateral position to an auditory tone using a spatial-to-spectral conversion rule. Moreover, the usefulness of auditory cueing in real traffic environments may be somewhat limited because of potential acoustic interference from the radio or individuals conversing. Thus, the use of another sensory modality, e.g., haptic feedback 28,29 , could potentially prove to be useful for driver assistant systems using SS/A techniques.
Developing a better understanding of the neural substrate that underlies the processing of auditory cueing of vehicle lateral position is a relevant goal for future research. In the traditional view of sensory processing, sensory signals originating from a specific sensory organ are propagated toward a dedicated sensory cortex and, consequently, give rise to experiences in a unique sensory modality. This unimodal view of sensory processing suggests that auditory cueing of vehicle lateral position is primarily processed in the auditory cortex. However, recent findings have demonstrated that the prolonged use of SS/A can cause a reorganization of sensory cortices (for review, see Cecchetti et al. 30 ). For example, it has been shown that after training with a visual-to-auditory substitution device, trained auditory stimuli activate cortical areas that are, in general, considered critical for the processing of visual features [31][32][33][34] . These findings highlight the multimodal nature of sensory cortices and suggest that, in fact, auditory cueing of vehicle lateral position might be processed in visual cortical areas that normally play a role in the extraction of vehicle lateral position from the proximal part of the road during normal driving.
In addition to exploring the underlying neural mechanisms, we envision several additional avenues of future research. For instance, a promising research direction is to expand the applicability of visual-to-auditory substitution for enhanced vehicle control. While in the present study we successfully conveyed the proximal part of the road with auditory cueing, future research could investigate the feasibility of additionally replacing the visual information from the distant part of the road with auditory cueing to determine if participants can accurately maintain lane-keeping with no visual information at all. Another direction of future research is to investigate the impact of auditory substitution on the driver's eye-hand coordination strategy. It is well established that eye movement and steering behaviour are tightly related during curve negotiation; indeed, it has been shown that the conclusions We demonstrated that auditory cueing of vehicle lateral position enables accurate steering control, even in a situation where visual information normally necessary for lane-keeping is unavailable. This suggests, more generally, that SS/A techniques can be used in situations that demand immediate actions in response to the substituted sensory information. Further studies are required to establish the generalizability of the findings to more realistic real-world settings.