For teleoperation tasks requiring high control accuracy, it is essential to provide teleoperators with information on the interaction between the end effector and the remote environment. Real-time imaging devices have been widely adopted, but it delivers limited information, especially when the end effectors approach the target following the line-of-sight. In such situations, teleoperators rely on the perspective at the screen and can apply high force unintentionally at the initial contact. This research proposes to deliver the distance information at teleoperation to the fingertips of teleoperators, i.e., proximity sensation. Transcutaneous electrical stimulation was applied onto the fingertips of teleoperators, with the pulsing frequency inversely proportional to the distance. The efficacy of the proximity sensation was evaluated by the initial contact force during telerobotic pinch in three sensory conditions: vision only, vision + visual assistance (distance on the screen), and vision + proximity sensation. The experiments were repeated at two viewing angles: 30–60° and line-of-sight, for eleven healthy human subjects. For both cases, the initial contact force could be significantly reduced by either visual assistance (20–30%) or the proximity sensation (60–70%), without additional processing time. The proximity sensation is two-to-three times more effective than visual assistance regarding the amount of force reduction.
In dexterous telerobotic operations, the structure of robotic end effectors has evolved from a simple rod into a complex gripper, which resembles human fingers to provide enhanced intuitiveness and control accuracy. The grippers generally follow the motion input from the handle grasped by human fingers at the master interface, as introduced in the typical master-slave surgical robotic systems1,2,3. The interface has been designed to provide a kinematic transparency between the operator’s fingers and the robotic gripper tips, so that the teleoperator can feel the gripper tips as one’s own, extended fingers.
Human finger motion for physically interactive tasks can be categorized into three phases: approach, contact, and interact in the order of time. For each phase, a different set of peripheral sensory feedback is provided to the human nervous system and helps human operators secure a certain level of control accuracy. Although visual feedback plays a major role in delivering sensory information during human motor control with its incomparable information transfer capability, peripheral sensory feedback effectively compensates for limitations of visual feedback. For example, tactile feedback from the fingertips provides texture and pressure during the contact and interact phases, which can be only roughly detected by visual feedback. Proprioception at the muscles proximal to the fingertip is commonly employed throughout all the phases and provides useful spatial information, especially during the approach phase as the fingertip delivers very limited information before contact occurs, such as temperature or air flow4. These roles of peripheral sensory feedback can be even more important for telerobotic operations as shown in Fig. 1, because of the limited visual feedback delivered through the televideo system.
The proprioception, however, is not reliable for the fine fingertip control because of the considerable distance between the joints along the arm and fingertips5,6. A small estimation error in using the proprioceptive feedback or the intrinsic kinematic parameters of the upper-limb saved in the brain can cause severe error at the fingertip. Therefore, a teleoperator usually relies heavily on visual feedback, especially during the approach phase, leading to the integration of advanced vision algorithms with high-definition sensors. Despite that, visual feedback cannot be an ultimate solution because it delivers a limited amount of information when the gripper tips and target are aligned in the line of sight. Also, in surgical teleoperation, an endoscopic camera is always located behind two or more grippers7,8 to form triangulation. When the vision is blocked or unclear due to anatomical structures, blood, or instruments, which is often the case, the camera cannot capture the approaching motion properly. Further, understanding visual feedback for use in fingertip manipulations needs significant cognitive involvement and processing time to interpret the real-time video, which can be a high burden to the teleoperators who need to handle multiple tasks demanding continuous decision making during the teleoperation. The intrinsic visual-proprioceptive mapping error is another challenge, because even the high-resolution visual feedback may not be effective with the visual-proprioceptive mapping error9,10.
The lack of reliable sensory feedback during the approach phase can result in higher force than the desired at the moment of contact between the end effector under teleoperation and the target objects, potentially disturbing subsequent manipulations or permanently damaging sensitive objects11,12. A high level of concentration is required to address this issue13,14, but it is still challenging to accurately estimate the necessary information and to complete the task on a timely basis. As an alternative, visual assistance displaying texts or numbers on the screen and virtual reality have shown their efficacy for fine force control during robotic surgery15,16,17,18,19. The efficacy of visual assistance in improving control accuracy can be further enhanced by intuitive display (e.g., graphical display). Graphics, however, still occupy visual feedback, which can negatively affect the teleoperator’s concentration on the target site. A similar problem exists when using auditory feedback as an assistance, as auditory feedback is often occupied by verbal communications. Also, its efficacy as an assistance is limited by the coding capacity of the human auditory system and its nature optimized to process either natural sounds or speech20,21.
Peripheral sensory augmentation usually requires less cognitive load than visual or auditory does to deliver sensory information21,22. Thus, it could be another way to compensate for limited sensory feedback during the approach phase of the teleoperation, which has high accuracy needs. Also, it is not as heavily occupied in daily life as visual or auditory feedback. Haptic feedback is one of the most promising candidates for such peripheral sensory augmentation in telerobotic applications23,24,25 because it uses the immense potential of the periphery for the reception of sensory information. However, so far in most cases, haptics has not been provided during the approach but instead during the contact and interact phases. For example, haptic interface devices were developed to let the user feel the stiffness of the tissue at dissection, via the information from a multi-axis force and torque sensor attached to the cutting instrument26. Most commercial haptic interfaces showed their efficacy in distinguishing between different tissues during the contact and interact phases27,28,29. Although a few examples of haptic feedback have been integrated during the approach phase16,17,30,31, all of them were designed to deliver the predefined boundaries of a safe workspace32,33,34, rather than providing real-time sensory information during the approach.
In this paper, we propose to provide a proximity sensation directly on the fingertips during the approach phase, by employing an electrically-evoked pulsing sensation that can be implemented in a small form factor, suitable for human fingertips. The proposed proximity sensation will allow teleoperators to enjoy the advantage of high resolution and rich cortical engagement of the human fingertip, which is one of the body parts with the highest level of sensory resolution as demonstrated in several studies and represented by the large corresponding somatosensory cortex area35,36,37. The effectiveness of the sensation was evaluated experimentally for a telerobotic pinching operation, where the distance between the robotic gripper tip and a target object was fed back to the operator in real-time, i.e., proximity sensation, as a continuous electrical pulse at the frequency inversely proportional to the distance.
Goal and Hypotheses
The overall goal of the current research was to determine the effect of the proposed fingertip proximity sensation on telerobotic grasping and manipulation of a remote object with finger-like slave grippers, especially when the visual feedback provides limited information. The main hypothesis was that the proximity sensation between the fingertip and the target object would help the operator to reduce the initial contact force of telerobotic pinching if necessary. Since a lower contact force initiates the contact phase in a more stable way, task performance potentially can be improved via a safer and smoother transition toward subsequent manipulation. We expected that the effect of the proximity sensation would be more prominent when the object was in the line of sight, because limited visual information makes it hard to control the initial contact force. The second hypothesis was that the proximity sensation would be more effective than visual assistance provided by numbers, on minimizing the initial contact force in telerobotic pinching. The proximity sensation is direct and intuitive, as it is applied to the fingertip corresponding to the end effector that is interacting with the target object, while visual assistance needs an additional visual-proprioceptive mapping between the numbers on the screen (i.e., distance) and the proprioceptive feedback in charge of following the fingertip movement. Also, visual assistance needs to share visual resource that is heavily occupied by the observation of the end effector.
Implementation of the Proximity Sensation
Implementation of the proximity and tactile sensations together on the fingertip
To evoke proximity sensation on the fingertip, transcutaneous electrical stimulation onto the palmar digital nerve was selected with a stimulation frequency inversely proportional to the distance between the robotic tool and the targeted object. Electrical stimulation was selected due to its smaller form factor compared to that of mechanical stimuli, which need several moving parts to interact with the fingertip38,39. The inversely-proportional relationship between the frequency and the distance was selected to follow the practice of existing applications that use frequency-modulated sensory interventions in delivering distance information40,41. The location of stimulation, the distal part of the palmar digital nerve, is composed mostly of cutaneous axons and located at the bony area with minimal fats and muscles near to the skin. Therefore, transcutaneous electrical stimulation of the skin over this nerve easily elicits the electrotactile feedback on the fingertip42,43. Since each subject has a different perceptual sensitivity to the stimulation frequency, the perceptible range of each subject was tested and used to find the optimal frequency range. First, the minimum frequency was set to 10 Hz for the information transfer rate to be reasonable (i.e., deliver pulsing sensation for every 100 ms), while maximizing the usable frequency range. Second, the maximum frequency for the proximity sensation was set to 5 Hz lower than the highest frequency (fmax) that evokes pulsing sensation. A 5 Hz margin was added to make sure that subjects felt the pulsing sensation over the entire frequency range used to evoke proximity sensation.
It is also important to integrate the tactile sensation together with the proximity sensation on the fingertip, considering that the pinching operation evokes tactile feedback after the contact is made. By integrating both the proximity and tactile sensations, the sensory feedback on the fingertip can be continuous over the entire phase of the pinching operation. When the fine motor control at the fingertip is crucial at contact, the continuity of sensory feedback at contact would be important to minimize the unwanted overshoot or undershoot of motor output at contact. To deliver both proximity and tactile sensations during a telerobotic pinch, the electrotactile feedback should be changed discretely at contact, for the nervous system not to become confused44 between the two sensations. When the stimulating frequency is above certain threshold, our nervous system can no longer feel the pulsing and will instead feel buzzing. As this buzzing-type electrotactile feedback is clearly distinguishable from the pulsing-type electrotactile feedback, we selected pulsing and buzzing sensations, respectively for proximity and tactile sensations, as shown in Fig. 2. To evoke buzzing sensation on the fingertip after the contact was made at teleoperation, we programmed the stimulation frequency to be increased discretely by 10 Hz, from fmax − 5 Hz to fmax + 5 Hz, when the contact is made. With this discrete change of frequency and the following discrete change of electrotactile feedback, we could provide sensory discrepancy between the approach and contact phases, with pulsing during the approach phase and buzzing during the contact phase. During the interact phase after contact, the stimulation frequency stayed the same, so as not to deliver further information and purely evaluate the efficacy of the proximity sensation at the initial contact.
Implementation of the proximity sensation onto the telerobotic pinch via the robotic gripper
The telerobotic pinching operation was selected to test the effectiveness of the proximity sensation in telerobotic object manipulation because it is the most basic mode of fine finger control45,46,47,48 and has been widely used for telerobotic surgeries49,50. Also, tips of the robotic gripper move along one direction, making the data analysis and comparison simple and clear. The sensorimotor interface used in this study allowed bi-directional communication between the teleoperator’s finger and the robotic gripper. First, as a motor pathway from the finger to the gripper (Fig. 3), the servo motor of the gripper was controlled by the output of an optical distance sensor installed on the index finger via a silicone fingertip guard, which provided the information of distance between the thumb and the index finger. The distance information between the two fingers was subsequently transferred to the servo motor that controls the gripper tips, via a motor control signal with a pulse width modulation. As the distance between the subject’s thumb and index finger becomes smaller, the gripper tips approach each other to a corresponding distance, and vice versa. Note that the movement of the gripper tip is discrete based on the specification of the servo motor, and the resolution is 60 µm. Second, as a sensory pathway from the gripper to the finger (Fig. 3), the distance information between each of the gripper tips and the object was measured by the optical distance sensor and delivered to the fingertips of the thumb and the index finger in the form of a pulsing sensation (i.e., proximity sensation). The frequency of electric stimulus was inversely proportional to the distance from each of the gripper’s fingertips to the object. The relationship between the sensor value (i.e., distance) and the stimulation frequency (fstim) is described in the equation below.
where fmax is the maximum frequency of each subject for pulsing sensation, dopen and dclose are the optical distance sensor readings corresponding to the distance between gripper tips when the gripper is fully open and fully closed, respectively (i.e., dopen = 15 mm, dclosed = 0 mm), while dcurrent is the current distance between gripper tips and the target object. According to the equation, the subject is expected to feel a slow pulsing sensation (fstim = 10 Hz) at the maximum distance between the gripper tips and the target object and a fast pulsing sensation (fstim = (fmax − 5) Hz) right before the contact is made.
Comparison of the initial contact force among Vision only, Vision + Visual Assistance, and Vision + Proximity Sensation, in two viewing angles
As seen in Fig. 4a, with the proximity sensation, subjects could lower the initial contact force by 68.4% at 30–60° viewing angle and 65.6% at a line-of-sight viewing angle (p < 0.001 for both viewing angles), compared to the initial contact force with only visual feedback. The significant reduction of the initial contact force with the proximity sensation supports the main hypothesis that the proximity sensation between the fingertip and the target object would help the operator to reduce the initial contact force of telerobotic pinching when necessary.
With visual assistance, subjects could lower the initial contact force by 23.8% and 30.5% at 30–60° and line-of-sight viewing angles, respectively. However, the reduction at the 30–60° viewing angle is not statistically significant (p = 0.047) while the reduction at the line-of-sight viewing angle is statistically significant (p < 0.001). For both viewing angles, the amount of force reduction with the proximity sensation was 2 × −3x times larger than the force reduction with the visual assistance, both compared to the force applied in the vision-only condition. These results support our second hypothesis that the proximity sensation would be more effective than visual assistance provided by numbers, in minimizing the initial contact force for telerobotic pinching.
Comparison of the approach time among Vision only, Vision + Visual Assistance, and Vision + Proximity Sensation, in two viewing angles
In both the 30–60° and line-of-sight viewing angles, no statistically-meaningful time difference was observed between any two conditions of sensory feedback, as shown in Fig. 4b. In other words, the approach time did not change by the condition of sensory feedback. The approach time did not change by the viewing angle either, based on the comparison under the same sensory condition. At the vision-only condition, although it was not statistically significant, subjects tend to spend less time with the line-of-sight viewing angle than with the 30–60° viewing angle (p = 0.024).
Amplitude/Frequency range for each subject
Fig. 5a summarizes the usable voltage ranges to deliver electrotactile feedback for each of eleven subjects. Note that, we recorded the minimum voltage level when subjects reported any sensation (i.e., perception threshold), and recorded the maximum voltage level when subjects reported any discomfort from the stimulation (i.e., comfort threshold). When setting the frequency of the stimulus at 100 Hz, the average perception threshold was 15.3 V with a standard error (STE) of 1.03 V and the average comfort threshold was 18.2 V with an STE of 0.829 V. Fig. 5b summarizes the usable frequency range to deliver pulsing sensation for each of eleven subjects. When setting the amplitude of the stimulus at the mean value of the perception and comfort thresholds collected for each person, all subjects could feel a pulsing sensation from 10 Hz and the average maximum frequency for subjects to feel a pulsing sensation was 51.4 Hz with an STE of 2.95 Hz. To make sure the frequency range for pulsing sensation was consistent during the experiment, we confirmed that the frequency range did not change after the whole experiment for each subject.
Results of the questionnaire to evaluate the perception of electrotactile feedback according to the stimulation frequency
Fig. 6 shows the results of the questionnaire, with the average and standard error among the data from eleven subjects. Results of question 1 suggest that subjects felt very clear pulsing sensation at 10 Hz. Results of question 2 suggest that, at (fmax − 5) Hz, subjects felt both pulsing and buzzing sensation, somewhat closer to pulsing. Results of question 3 suggests that subjects mostly felt a constant buzzing sensation instead of pulsing sensation at (fmax + 5) Hz. Results of question 4 suggest that subjects clearly felt the difference in pulsing frequency between the pulsing at (fmax − 5) Hz and the pulsing at 10 Hz. Results of question 5 suggest that subjects felt a higher intensity of pulsing sensation at (fmax − 5) Hz than at 10 Hz but with a small difference.
Tactile feedback on the fingertip can be an efficient information modality to deliver the distance information during the approach phase of telerobotic pinch
Tactile feedback on the fingertip is not normally engaged in the approach phase of telerobotic pinch, as no contact has been made between the fingertip and the object. The most important message that this paper delivers to the readers is the immense potential of tactile feedback on the fingertip to be used as a proximity sensation to improve the control accuracy of the initial contact force. The addition of proximity sensation could decrease the initial contact force by 60–70%. The comparison between the effects of proximity sensation and visual assistance also demonstrates that the tactile feedback on the fingertip (as a proximity sensation) is very “effective” to deliver the varying distance information during the approach phase of a telerobotic pinch, considering that visual feedback has been considered as the most effective modality to deliver information.
The efficacy of visual assistance is variable among subjects, perhaps because of the different visual-proprioceptive coordination capability, while the effect of the proximity sensation is consistent over all subjects, perhaps because of the consistent electrotactile-proprioceptive coordination capability
By examining the data in more detail, we found that some of the subjects (4 of 11 at the 30–60° viewing angle and 2 of 11 at the line-of-sight viewing angle) applied even higher initial contact force on average with visual assistance, compared to the force applied to the visual-only condition. On the other hand, all subjects applied lower initial contact force with the proximity sensation, compared to the force applied with the visual-only condition. We expect that those 2–4 subjects, for whom visual assistance was not at all effective for telerobotic pinching, have poor visual-proprioceptive coordination. It is perhaps because the efficacy of visual assistance would depend heavily on the visual-proprioceptive mapping capability of each subject (i.e., the mapping between visual feedback and proprioceptive feedback on muscles associated with pinching)9,10,51. Another possible interpretation is that visual assistance could not be used effectively in real-time telerobotic pinching, because the visual feedback was heavily engaged in the observation of the robotic gripper. On the other hand, all 11 subjects could use the proximity sensation effectively to reduce the initial contact force, which suggests that electrotactile-proprioceptive mapping (i.e., mapping between the electrically-evoked proximity sensation on the fingertip and proprioceptive feedback on the muscles associated with pinching) is consistent over subjects.
Neither the proximity sensation nor visual assistance increases approach time
As shown in Fig. 4b, it was observed that the average time that subjects spent to approach the object (i.e., approach time) did not change when adding either the proximity sensation or visual assistance. This result suggests that both the proximity sensation and visual assistance provide distance information to the teleoperator in an intuitive way that requires minimal processing time, although further study is needed to test the cognitive involvement. Despite no statistical significance, the approach time tends to somewhat decrease with either the proximity sensation or visual assistance, at the 30–60° viewing angle. We also cannot exclude the possibility that the approach time can be further reduced after the teleoperator learns how to use the additional sensory information.
Electrically-evoked pulsing sensation on the fingertip conveys analog information intuitively to human operator, if the stimulation parameters are well customized for each subject
The results of the questionnaire indicate that subjects could feel the difference in the frequency of stimulation, via both frequency and intensity of pulsation, if the frequency was within the range for pulsing sensation (see Fig. 6). Therefore, we expect that the frequency modulation, within the frequency range for pulsing sensation, can be used to deliver analog information. It agrees with previous studies regarding the immense potential of electrotactile feedback on the fingertip in sending multi-bit digital information via frequency modulation52,53,54,55. Based on the questionnaire results, subjects could distinguish the pulsing sensation from the buzzing sensation and the pulsing sensation can be discretely changed to the buzzing sensation by a leap of 10 Hz from (fmax − 5) Hz to (fmax + 5) Hz. This result suggests that pulsing and buzzing sensations can be used together in a single system to deliver different information. For example, the pulsing sensation with frequency modulation can deliver the distance at proximity and the buzzing sensation can deliver the discrete on/off state for contact.
Additionally, it was found that both the perception threshold of stimulation amplitude and the range of pulsing frequency are variable among subjects, as seen in Fig. 5. This result suggests that the stimulation parameters should be customized for each teleoperator based on the perception test before the stimulation is used to evoke electrotactile feedback. However, it is still possible to define a common frequency range for pulsing, generally applicable to all subjects56,57,58,59,60, as all subjects reported a pulsing sensation at a frequency range between 10 and 35 Hz.
Inversely-proportional relationship between frequency and distance works well, but further investigation is needed to find the best modulation method to deliver the distance information
Humans feel an object is approaching when the frequency of the sensory feedback is increasing40. This perhaps is related to the Doppler effect that explains the frequency increase of sound, light, or other waves as the source and observer move toward each other. Such a perception has been applied to the parking assistant system of cars, which generates a warning sound that beeps with higher frequency as the exterior of the car approaches a wall41. Similarly, the sensory feedback with higher frequency could be mapped intuitively to the closer distance between the fingertip and the object. We found that this relationship worked well in the proposed telerobotic pinch system, considering that the proximity sensation reduced the initial contact force, as intended.
However, it has not been tested whether the inversely-proportional relationship between frequency and distance is the best option to deliver the distance information. Although the frequency or amplitude modulations have been developed to work efficiently with the electrical system, the best option for the neural communication might be different. We expect that another effort is necessary to find the neural-friendly code to deliver the distance information effectively and intuitively. Also note that we excluded the option of amplitude modulation in delivering distance information for two reasons: 1) the range of usable amplitude was very narrow for some subjects, as they felt discomfort right above their perception threshold (see Fig. 5), and 2) there is an adaptation issue when a stimulation is applied for a long time period (i.e., the amplitude of stimulation needs to be increased as the nervous system adapts to the stimulation).
Further investigation is needed to show the viability of this approach if the object is not located exactly in between two gripper tips or any obstacle presents between the gripper tip and the target
In the experiments of this study, the gripper tips were designed to be located symmetrically to the target during the telerobotic pinching operation. Therefore, subjects felt symmetrical proximity sensation on the two fingers (i.e., the thumb and the index finger) during the telerobotic pinch. However, in actual situations of telerobotic surgery, two gripper tips are often located asymmetrically to the target. Future studies can test how well the teleoperators do the voluntary adjustment using the asymmetric proximity sensation, to gain the symmetry between the two distances and to avoid unintended excessive force on one side. Further, any unexpected obstacle in between the gripper tip and the target object can cause another challenge to minimizing the initial contact force using the proximity sensation. Future studies also can test how well the teleoperators address the problem of unexpected obstacles, perhaps by the coordination between visual feedback and the proximity sensation.
In this paper, we proposed to add the proximity sensation to the fingertip of the teleoperator, to improve the control accuracy of the telerobotic pinching operation. Particularly, the sensation was evoked electrically on the fingertip in action, with the pulsing frequency inversely proportional to the distance between the end effector and the target object. Since available sensory feedback (i.e., vision and proprioception) is not reliable for fine fingertip control during the approach phase of teleoperation, the additional sensory feedback on the site of actuation (i.e., fingertip in action) will be highly beneficial to enhance control accuracy. Such additional sensation can help teleoperators perform delicate telerobotic operations more accurately and safely, allowing them to locate effectors precisely to the objects and to significantly reduce the initial contact force and minimize the potential disturbance to subsequent manipulations. The proposed proximity sensation also occupies a minimal cognitive load, as suggested by the absence of additional time to approach and contact objects, allowing timely completion of telerobotic operations without unnecessary fatigue. Such a sensation at the periphery could be a significant addition to the telerobotic neurosurgery, which requires high control accuracy and safety. The application of the proximity sensation, however, is not limited to surgery but applicable to other telerobotic applications that need high control accuracy and delicacy at the approach, such as rescue missions or remote maintenance of structures in harsh environments such as space and underwater.
Human subject recruitment
The study was performed in accordance with relevant guidelines/regulations, according to the procedure described in the protocol approved by the Institutional Review Board of Texas A&M University (IRB2018-0893D). Eleven healthy human subjects participated in the study. The number of subjects was chosen based on previous studies on modulating fingertip cutaneous feedback to determine its effect on finger-grip force control61,62. Subjects consisted of three females and eight males. All subjects were over the age 18 with the average of 29.6. All subjects gave informed consent. All subjects were right-handed and used their right hand to perform experiments.
Exp I: Customization of stimulation parameters for each subject and questionnaire to evaluate the perception
First, we customized stimulation amplitude and frequency for each subject based on verbal responses. Regarding amplitude, we set the minimum voltage level at where subjects reported any sensation (i.e., perception threshold), and set the maximum voltage level at where subjects reported any discomfort from the stimulation (i.e., comfort threshold), at the frequency of 100 Hz. To set the perception threshold, we started to apply the stimulation with very low intensity (5 V) and gradually increased it by 0.5 V, until subjects reported any sensation. We then kept increasing it by 0.5 V until subjects reported any discomfort, to set the comfort threshold. Regarding frequency, we searched the frequency range to evoke the pulsing sensation, while setting the amplitude at the mean value of the perception and comfort thresholds for each subject. The stimulation frequency was increased from 10 Hz by 5 Hz step until each subject reported a constant buzzing sensation and no more pulsing sensation. The frequency right before each subject started to feel the constant buzzing sensation was defined as fmax. Therefore, subjects still felt the pulsing sensation at fmax.
We then evaluated how each subject perceived the electrotactile feedback according to the frequency, via a questionnaire. Each subject was asked to fill out a questionnaire that consisted of five questions with a 5-point Likert scale. The first and second questions were to confirm if subjects felt a pulsing sensation at 10 Hz and (fmax − 5)Hz, respectively. The third question was to confirm if subjects still felt a constant buzzing sensation at (fmax + 5)Hz. The fourth question is to evaluate how differently subjects felt the frequency of pulsing between 10 Hz and (fmax − 5)Hz. The fifth question was to compare the intensity of sensation between 10 Hz and (fmax − 5)Hz.
Exp II: Test the efficacy of interventions (visual assistance and proximity sensation) on improving the control accuracy of telerobotic pinching
With the amplitude and frequency range of stimulation to evoke the pulsing sensation, customized for each subject, we tested the efficacy of the proximity sensation on minimizing the initial contact force and compared it with the efficacy of visual assistance. By receiving the data transmitted from two optical distance sensors attached onto the gripper tips, visual assistance displayed the distance between each gripper tip and the hardwood block on the monitor via numbers in the unit of millimeter, as described in Fig. 7. To compare the efficacy of two different interventions, the contact force was measured at each side of the hardwood block under three different conditions of sensory feedback: (1) vision only, (2) vision + visual assistance, and (3) vision + proximity sensation. In the first condition, subjects were provided with only visual feedback of the gripper tips. In the second condition, subjects were allowed to look at both the gripper tips and the monitor showing the numbers for distance. In the third condition, the proximity sensation was provided to the fingertips of the thumb and index finger with visual feedback on the gripper tips. All three tests were repeated with two different viewing angles: 30–60° and line-of-sight. Note that, the viewing angle is defined as the angle between the line connecting the top center of the hardwood block and the center of subjects’ eyes and the horizontal line on the desk, as seen in Fig. 7.
For all experimental conditions, the gripper tips were initially positioned symmetrical to the target, placing both tips at 15-mm distance from the target. Wearing the instrumented silicone finger cap and having bipolar gel electrodes on the fingertips, subjects were asked to apply minimal force at the contact. Additionally, subjects were asked to stop the pinching when they believed that contact was made and maintain the initial pinching force for 5 seconds. For each trial, the operator gave subjects a verbal sign to start pinching consistently when the operator clicked the start button for time ticking, to measure the time spent for the pinching. Each subject completed ten trials for each sensory condition with the order randomized to exclude any learning effect. This procedure of experiments was repeated for both viewing angles. Therefore, total number of trials for each subject was 60 (n = 30 for each viewing angle).
Implementation of electronics and electrodes
Silicone fingertip guards were made to be worn on the index finger to measure the distance between the fingers, with the optical reflective sensor (HSDL9100-021, Broadcom) integrated onto the surface corresponding to the fingertip.
Bipolar fingertip gel electrodes were attached to the palmar digital nerve of each subject’s thumb and index finger, as seen in Fig. 3. The electrodes were custom made in the lab. Each electrode was composed of 1-cm2 area of a hydrogel, which adhered to a congruent sheet of conductive carbon fiber. This carbon fiber layer maintained robust electrical conduction to a multi-threaded 36-AWG wire via silver conductive epoxy.
Robotic gripper (ROT3U Robotic Clamp Claw, Aideepen) was selected as the slave arm, whose tips move similarly to human fingers for pinching operation. The servo motor (MG996R, Towerpro), used for gripper operation, was responsive enough to support the real-time telerobotic operation. A hard object was made from a hardwood block (10″×10″×1/4″) and firmly fixated onto the experiment table to keep the position and orientation of the target object between subjects. As shown in Fig. 3, two optical reflective sensors (HSDL9100-021, Broadcom) were installed onto each tip of the gripper, and covered by a transparent silicone elastomer (Sylgard 184, Dow Corning). The measurable range of the sensor was 0 to 60 mm, which covered the entire range of motion for the gripper tip.
Microcontroller (SAM3X8E, Microchip) received the distance information measured by optical distance sensors, and sent it to the operator’s thumb and index finger as an electrical stimulus delivering the proximity sensation. Microcontroller, along with the built-in stimulator, also generated a bi-phasic electrical stimulus with a pulse width of 1 ms for each polarity63.
Two force sensors (S15-4.5 N, SingleTact) were installed on each side of the hardwood block to detect the force applied by each tip of the gripper at initial contact. The full-scale force of the sensor was selected as 4.5 N, as a soft tapping force from the fingertip is reported as 0.5 to 1.47N64. The measurement resolution is 9mN, which could divide the reported soft-pinching force range into more than 100 steps.
To evaluate the efficacy of each intervention, a two-tailed Welch’s t-test was performed at the 95% confidence level, as part of the datasets under comparison had unequal variances. To verify that the data satisfies the prerequisites for a t-test, we tested normality of data distribution using the Kolmogorov-Smirnov test of normality. All datasets satisfied the condition of p > 0.05 and normality could be assumed. We also applied Bonferroni correction considering nine comparisons were conducted with the same dataset, and then we used p < 0.0056 as the condition for statistical significance.
Comparison of initial contact force
We first identified the plat part of the force output after the initial change of the force output from the baseline (i.e., first plateau). Then, the range of the first plateau was identified by including the force output within ±5% of the mean value of the stable range of the force output in the middle of the plateau. The value after the first plateau was ignored to exclude the re-adjusted contact force after the contact was made. The force output during the plateau was averaged for each condition. For example, in the trial shown in Fig. 8, the plateau was identified between 5.25 s and 6.50 s, and the initial contact force was calculated as the mean value of the identified plateau range. The average and standard error of the initial contact force from eleven subjects were calculated for each condition, and the graphical representation along with the result of the statistical test are shown in Fig. 4a.
Comparison of approach time
We also identified the approach time required to reach the first plateau for each condition. For any single trial, the approach time was defined as the interval from the moment the operator gave the verbal signal of start to the subjects until the moment the waveform reached the first plateau (see Fig. 8). The average and standard error of the approach time from the eleven subjects were calculated for each condition, and the graphical representation along with the result of the statistical test are shown in Fig. 4b.
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Samadi, D. B. History and the future of Robotic Surgery, https://www.roboticoncology.com/history-of-robotic-surgery/.
Lanfranco, A. R., Castellanos, A. E., Desai, J. P. & Meyers, W. C. Robotic surgery: a current perspective. Ann Surg 239, 14–21, https://doi.org/10.1097/01.sla.0000103020.19595.7d (2004).
Sung, G. T. & Gill, I. S. Robotic laparoscopic surgery: a comparison of the Vinci and Zeus systems. Urology 58, 893–898, https://doi.org/10.1016/S0090-4295(01)01423-6 (2001).
Scheidt, R. A., Conditt, M. A., Secco, E. L. & Mussa-Ivaldi, F. A. Interaction of Visual and Proprioceptive Feedback During Adaptation of Human Reaching Movements. Journal of Neurophysiology 93, 3200–3213, https://doi.org/10.1152/jn.00947.2004 (2005).
Wann, J. P. & Ibrahim, S. F. J. E. B. R. Does limb proprioception drift? Experimental Brain Research 91, 162–166 (1992).
van Beers, R. J., Sittig, A. C. & Van Der Gon, J. J. D. The precision of proprioceptive position sense. Experimental Brain Research 122, 367–377 (1998).
Hubens, G., Coveliers, H., Balliu, L., Ruppert, M. & Vaneerdeweg, W. A performance study comparing manual and robotically assisted laparoscopic surgery using the da Vinci system. Surgical Endoscopy and other interventional techniques 17, 1595–1599, https://doi.org/10.1007/s00464-002-9248-1 (2003).
Byrn, J. C. et al. Three-dimensional imaging improves surgical performance for both novice and experienced operators using the da Vinci Robot System. The American Journal of Surgery 193, 519–522, https://doi.org/10.1016/j.amjsurg.2006.06.042 (2007).
Kuling, I. A., de Brouwer, A. J., Smeets, J. B. J. & Flanagan, J. R. Correcting for natural visuo-proprioceptive matching errors based on reward as opposed to error feedback does not lead to higher retention. Experimental Brain Research 237, 735–741, https://doi.org/10.1007/s00221-018-5456-3 (2019).
Burns, E. et al. The hand is more easily fooled than the eye: Users are more sensitive to visual interpenetration than to visual-proprioceptive discrepancy. Presence: teleoperators & virtual environments 15, 1–15 (2006).
Kraft, B., Jäger, C., Kraft, K., Leibl, B. & Bittner, R. The AESOP robot system in laparoscopic surgery: Increased risk or advantage for surgeon and patient? Surgical Endoscopy Other Interventional Techniques 18, 1216–1223 (2004).
Mills, J. T. et al. Positioning Injuries Associated with Robotic Assisted Urological Surgery. The Journal of Urology 190, 580–584, https://doi.org/10.1016/j.juro.2013.02.3185 (2013).
Bourassa, G. L. & Guion, R. M. A factorial study of dexterity tests. Journal of Applied Psychology 43, 199–204, https://doi.org/10.1037/h0045568 (1959).
Camarillo, D. B., Krummel, T. M. & Salisbury, J. K. Jr. Robotic technology in surgery: Past, present, and future. The American Journal of Surgery 188, 2–15, https://doi.org/10.1016/j.amjsurg.2004.08.025 (2004).
Reiley, C. E. et al. Effects of visual force feedback on robot-assisted surgical task performance. The Journal of Thoracic and Cardiovascular Surgery 135, 196–202, https://doi.org/10.1016/j.jtcvs.2007.08.043 (2008).
Horeman, T., Rodrigues, S. P., van den Dobbelsteen, J. J., Jansen, F.-W. & Dankelman, J. Visual force feedback in laparoscopic training. Surgical endoscopy 26, 242–248, https://doi.org/10.1007/s00464-011-1861-4 (2012).
Horeman, T. et al. Visual Force Feedback Improves Knot-Tying Security. Journal of Surgical Education 71, 133–141, https://doi.org/10.1016/j.jsurg.2013.06.021 (2014).
Trejos, A. L., Patel, R. V. & Naish, M. D. Force sensing and its application in minimally invasive surgery and therapy: A survey. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science 224, 1435–1454, https://doi.org/10.1243/09544062jmes1917 (2010).
Meccariello, G. et al. An experimental study about haptic feedback in robotic surgery: may visual feedback substitute tactile feedback? Journal of Robotic Surgery 10, 57–61, https://doi.org/10.1007/s11701-015-0541-0 (2016).
Smith, E. C. & Lewicki, M. S. Efficient auditory coding. Nature 439, 978–982, https://doi.org/10.1038/nature04485 (2006).
Sigrist, R., Rauter, G., Riener, R. & Wolf, P. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic bulletin & review 20, 21–53, https://doi.org/10.3758/s13423-012-0333-8 (2013).
Roemmich, R. T., Long, A. W. & Bastian, A. J. Seeing the Errors You Feel Enhances Locomotor Performance but Not Learning. Curr Biol 26, 2707–2716, https://doi.org/10.1016/j.cub.2016.08.012 (2016).
Okamura, A. M. Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol 19, 102–107, https://doi.org/10.1097/MOU.0b013e32831a478c (2009).
Okamura, A. M. Methods for haptic feedback in teleoperated robot‐assisted surgery. Industrial Robot: An International Journal 31, 499–508, https://doi.org/10.1108/01439910410566362 (2004).
Bethea, B. T. et al. Application of Haptic Feedback to Robotic Surgery. Journal of Laparoendoscopic & Advanced Surgical Techniques 14, 191–195, https://doi.org/10.1089/1092642041255441 (2004).
Tholey, G., Chanthasopeephan, T., Hu, T., Desai, J. P. & Lau, A. Measuring grasping and cutting forces for reality-based haptic modeling. International Congress Series 1256, 794–800, https://doi.org/10.1016/S0531-5131(03)00492-8 (2003).
Hu, T., Castellanos, A. E., Tholey, G. & Desai, J. P. Real-time haptic feedback in laparoscopic tools for use in gastro-intestinal surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention, 66–74 (2002).
Wottawa, C. R. et al. Evaluating tactile feedback in robotic surgery for potential clinical application using an animal model. Surgical endoscopy 30, 3198–3209, https://doi.org/10.1007/s00464-015-4602-2 (2016).
Abiri, A. et al. Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic Surgery. Scientific Reports 9, 5016, https://doi.org/10.1038/s41598-019-40821-1 (2019).
Wang, F., Poston, T., Teo, C. L., Lim, K. M. & Burdet, E. Multisensory learning cues using analytical collision detection between a needle and a tube. 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS ‘04. Proceedings., 339–346, https://doi.org/10.1109/HAPTIC.2004.1287218 (2004).
Osu, R., Burdet, E., Franklin, D. W., Milner, T. E. & Kawato, M. Different Mechanisms Involved in Adaptation to Stable and Unstable Dynamics. Journal of Neurophysiology 90, 3255–3269, https://doi.org/10.1152/jn.00073.2003 (2003).
Davies, B. L., Fan, K. L., Hibberd, R. D., Jakopec, M. & Harris, S. J. ACROBOT - using robots and surgeons synergistically in knee surgery. 1997 8th International Conference on Advanced Robotics. Proceedings. ICAR'97, 173–178, https://doi.org/10.1109/ICAR.1997.620179 (1997).
Davies, B. A review of robotics in surgery. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine 214, 129–140, https://doi.org/10.1243/0954411001535309 (2000).
Cobb, J. et al. Hands-on robotic unicompartmental knee replacement: a prospective, randomised controlled study of the acrobot system. The Journal of bone and joint surgery. British volume 88, 188–197, https://doi.org/10.1302/0301-620X.88B2.17220 (2006).
Mancini, F. et al. A fovea for pain at the fingertips. Curr Biol 23, 496–500, https://doi.org/10.1016/j.cub.2013.02.008 (2013).
Mancini, F., Haggard, P., Iannetti, G. D., Longo, M. R. & Sereno, M. I. Fine-grained nociceptive maps in primary somatosensory cortex. J Neurosci 32, 17155–17162, https://doi.org/10.1523/JNEUROSCI.3059-12.2012 (2012).
Pfannmöller, J. P., Schweizer, R. & Lotze, M. Automated analysis protocol for high resolution BOLD-fMRI mapping of the fingertip somatotopy in brodmann area 3b. Journal of Magnetic Resonance Imaging 43, 479–486, https://doi.org/10.1002/jmri.24980 (2016).
Choi, I., Culbertson, H., Miller, M. R., Olwal, A. & Follmer, S. Grabity: A Wearable Haptic Interface for Simulating Weight and Grasping in Virtual Reality. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 119–130, https://doi.org/10.1145/3126594.3126599 (2017).
Frediani, G., Mazzei, D., De Rossi, D. E. & Carpi, F. Wearable wireless tactile display for virtual interactions with soft bodies. Front Bioeng Biotechnol 2, 31–31, https://doi.org/10.3389/fbioe.2014.00031 (2014).
MacLean, K. & Enriquez, M. Perceptual design of haptic icons. Proc. of EuroHaptics, 351–363 (2003).
Borenstein, J. & Ulrich, I. The guidecane-a computerized travel aid for the active guidance of blind pedestrians. Proceedings of International Conference on Robotics and Automation 2, 1283–1288 (1997).
Yoshimoto, S., Kuroda, Y., Uranishi, Y., Imura, M. & Oshiro, O. Roughness modulation of real materials using electrotactile augmentation. International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, 10–17 (2014).
Manoharan, S. & Park, H. Supernumerary Body Schema Extension to Non-Corporeal Object by Adding Artificial Tactile Feedback using Electrical Stimulation. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), 989–992, https://doi.org/10.1109/NER.2019.8717127 (2019).
Johansson, R. S. & Cole, K. J. Sensory-motor coordination during grasping and manipulative actions. Current opinion in neurobiology 2, 815–823 (1992).
Li, K., Nataraj, R., Marquardt, T. L. & Li, Z.-M. Directional Coordination of Thumb and Finger Forces during Precision Pinch. PLOS ONE 8, e79400, https://doi.org/10.1371/journal.pone.0079400 (2013).
Yokogawa, R. & Hara, K. Manipulabilities of the Index Finger and Thumb in Three Tip-Pinch Postures. Journal of Biomechanical Engineering 126, 212–219, https://doi.org/10.1115/1.1691444 (2004).
Kapandji, I. The physiology of the joints, volume I, upper limb. American Journal of Physical Medicine & Rehabilitation 50, 96 (1971).
Cutkosky, M. R. On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEEE Transactions on Robotics and Automation 5, 269–279, https://doi.org/10.1109/70.34763 (1989).
Smith, F. M., Backman, D. K. & Jacobsen, S. C. Telerobotic manipulator for hazardous environments. Journal of Robotic Systems 9, 251–260, https://doi.org/10.1002/rob.4620090209 (1992).
Brown, J. D., Ibrahim, M., Chase, E. D. Z., Pacchierotti, C. & Kuchenbecker, K. J. Data-driven comparison of four cutaneous displays for pinching palpation in robotic surgery. 2016 IEEE Haptics Symposium (HAPTICS), 147–154, https://doi.org/10.1109/HAPTICS.2016.7463169 (2016).
Prachyabrued, M. & Borst, C. W. Effects and optimization of visual-proprioceptive discrepancy reduction for virtual grasping. 2013 IEEE Symposium on 3D User Interfaces (3DUI), 11–14, https://doi.org/10.1109/3DUI.2013.6550190 (2013).
Tan, H. Z., Reed, C. M. & Durlach, N. I. Optimum Information Transfer Rates for Communication through Haptic and Other Sensory Modalities. IEEE Transactions on Haptics 3, 98–108, https://doi.org/10.1109/TOH.2009.46 (2010).
Summers, I. R. et al. Tactile information transfer: A comparison of two stimulation sites. The Journal of the Acoustical Society of America 118, 2527–2534 (2005).
Summers, I. R. et al. Vibrotactile and electrotactile perception of time‐varying pulse trains. The Journal of the Acoustical Society of America 95, 1548–1558 (1994).
Kaczmarek, K. A., Tyler, M. E., Okpara, U. O. & Haase, S. J. Interaction of Perceived Frequency and Intensity in Fingertip Electrotactile Stimulation: Dissimilarity Ratings and Multidimensional Scaling. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 2067–2074, https://doi.org/10.1109/TNSRE.2017.2702628 (2017).
Szeto, A. Y. & Saunders, F. A. Electrocutaneous stimulation for sensory communication in rehabilitation engineering. IEEE Transactions on Biomedical Engineering, 300–308 (1982).
Kajimoto, H., Kawakami, N. & Tachi, S. Psychophysical evaluation of receptor selectivity in electro-tactile display. Proceeding of 13th International Symposium on Measurement and Control in Robotics (2003).
Kaczmarek, K. A., Webster, J. G., Bach-y-Rita, P. & Tompkins, W. J. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions on Biomedical Engineering 38, 1–16, https://doi.org/10.1109/10.68204 (1991).
Knibestöl, M. Stimulus-response functions of slowly adapting mechanoreceptors in the human glabrous skin area. The Journal of Physiology 245, 63–80, https://doi.org/10.1113/jphysiol.1975.sp010835 (1975).
Tezuka, M., Kitamura, N., Tanaka, K. & Miki, N. Presentation of various tactile sensations using micro-needle electrotactile display. PloS one 11, e0148410 (2016).
Jorgovanovic, N., Dosen, S., Djozic, D. J., Krajoski, G. & Farina, D. Virtual Grasping: Closed-Loop Force Control Using Electrotactile Feedback. Computational and Mathematical Methods in Medicine 2014, 13, https://doi.org/10.1155/2014/120357 (2014).
Shin, H., Watkins, Z. & Hu, X. Exploration of Hand Grasp Patterns Elicitable Through Non-Invasive Proximal Nerve Stimulation. Scientific Reports 7, 16595, https://doi.org/10.1038/s41598-017-16824-1 (2017).
Azbell, J., Park, J. K., Chang, S., Engelen, M. P. K. J. & Park, H. Closed-loop Tactile Augmentation by Transcutaneous Stimulation on either the Foot Sole or the Palm to Improve Lateral Postural Balance. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), 1072–1075, https://doi.org/10.1109/NER.2019.8717004 (2019).
Asakawa, D. S., Crocker, G. H., Schmaltz, A. & Jindrich, D. L. Fingertip forces and completion time for index finger and thumb touchscreen gestures. Journal of Electromyography and Kinesiology 34, 6–13, https://doi.org/10.1016/j.jelekin.2017.02.007 (2017).
This work was supported by the The Institute for Rehabilitation and Research (TIRR) foundation.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zhao, Z., Yeo, M., Manoharan, S. et al. Electrically-Evoked Proximity Sensation Can Enhance Fine Finger Control in Telerobotic Pinch. Sci Rep 10, 163 (2020). https://doi.org/10.1038/s41598-019-56985-9
This article is cited by
Longitudinal in-vivo quantification of tumour microvascular heterogeneity by optical coherence angiography in pre-clinical radiation therapy
Scientific Reports (2022)
Journal of NeuroEngineering and Rehabilitation (2021)