Minimally invasive robotic surgery allows for many advantages over traditional surgical procedures, but the loss of force feedback combined with a potential for strong grasping forces can result in excessive tissue damage. Single modality haptic feedback systems have been designed and tested in an attempt to diminish grasping forces, but the results still fall short of natural performance. A multi-modal pneumatic feedback system was designed to allow for tactile, kinesthetic, and vibrotactile feedback, with the aims of more closely imitating natural touch and further improving the effectiveness of HFS in robotic surgical applications and tasks such as tissue grasping and manipulation. Testing of the multi-modal system yielded very promising results with an average force reduction of nearly 50% between the no feedback and hybrid (tactile and kinesthetic) trials (p < 1.0E-16). The multi-modal system demonstrated an increased reduction over single modality feedback solutions and indicated that the system can help users achieve average grip forces closer to those normally possible with the human hand.
Since the introduction of the first robotic surgical systems in 1985, the field of surgical robotics has gone through many changes with newer systems demonstrating increased ease of operations as well as decreased operative times compared with typical laparoscopic surgery1. Benefits of robotic surgery over conventional laparoscopic surgery include motion scaling for finer motion control, stereoscopic vision, increased dexterity, and additional degrees of freedom in motion2,3,4,5,6. However, these benefits have also come at a cost. Compared to conventional open surgery and even laparoscopic surgery, robotic procedures suffer from a complete loss of haptic feedback. The loss of haptic feedback coupled with the inherent ability of robotic surgical systems to apply strong compressive and shear forces, have led to increased risk of tissue damage, reduced performance, and increased number of mistakes7,8,9,10. Ultimately, these risks translate to clinical outcomes, meaning, greater tissue damage, more pain, and longer recovery times for the patient11,12,13,14. As robotic surgery gains popularity, it becomes imperative that haptic feedback systems (HFS) become a standard feature of commercially available surgical robotic systems.
In the past decade, there have been significant changes to the field of haptics and numerous haptic feedback solutions have been investigated for surgical robotics13,15,16,17,18,19. More primitive force feedback implementations have been included as part of some of the more modern surgical robotic systems17,20. Kinesthetic force feedback (KFF) has been perhaps the most comprehensively researched area of haptics21,22,23,24,25 because of its relative ease of integration with the master controls of surgical robots. However, haptics is not limited to kinesthetic force feedback. In fact, the sense of touch in humans involves the synergistic activation of mechanoreceptors both in the skin (i.e. tactile feedback) and in the muscles (i.e. kinesthetic force feedback)23. Some research groups have focused on tactile feedback as an alternative to force feedback, targeting activation of mechanoreceptors in the fingertips through pneumatic tactile feedback26 and/or vibrotactile feedback27,28,29. Studies have shown that these feedback solutions can significantly impact the performance of surgeons during the procedure30,31,32 and reduce tissue damage10,12,16,33. More recently, some systems have attempted development of multi-modal HFS by coupling thermal feedback with tactile feedback34,35. However, no significant investigations have been conducted to evaluate impact of such solutions in MIS applications.
Despite extensive research in the field of haptics, current feedback technologies are far from replicating the natural sense of touch that a surgeon receives in conventional open and even laparoscopic surgery. One reason for this is that none of these systems attempted to restore the synergistic behavior that takes place when various mechanoreceptors in the skin and muscles simultaneously become activated in real touch. In fact, nearly all haptic feedback systems have targeted only a single modality of feedback. The perception of feedback is therefore hindered because the brain does not receive signals from all the sensory pathways normally involved in the sense of touch.
A multi-modal haptic feedback system refers to a HFS that seeks to sense and convey more than one aspect of touch. By targeting multiple classes of mechanoreceptors in the skin and muscles, and conveying both tactile and proprioceptive information, multi-modal HFS engenders the promise of a more natural and complete HFS, one that hopes to bring haptics one step closer to real touch. Due to the engineering challenges that arise from attempting to integrate multiple sensing and feedback modalities while still maintaining small footprints necessary for application in surgery, multi-modal HFS has not been extensively studied. In more recent years, some attempts have been made on the development of bi-modal haptic feedback systems for surgical applications15,33,36,37,38, however, conclusive outcomes have not yet been achieved.
The use of haptic feedback for reduction of tissue damage in robotic tasks involving average forces generated during grasping and manipulation of soft tissue has previously been extensively studied26. Despite evidence that both tactile and kinesthetic feedback are effective at reducing forces applied to the tissue12,39,40,41, the effects of combining both tactile and kinesthetic feedback into one system have not been evaluated11. We hypothesize that a multi-modal HFS, which can target both mechanoreceptors in the skin and the muscles and provide simultaneous tactile and kinesthetic force feedback, can be more effective in reducing excessive grip forces during robotic surgical tasks.
The goal of haptic feedback systems is to restore the sense of touch available when performing tasks with human hands. Figure 1 shows the results of a benchmark study, in which the average grip force when 6 subjects performed two-handed peg-transfers with their own hands, was compared to when 15 subjects performed the task without any feedback on the da Vinci robot (data from Fig. 2A). These data showed that average grip force when performing a peg-transfer with a human hand was a mere 0.88N with a standard deviation of 0.15N. On the other hand, the same task performed using the da Vinci system without HFS, resulted in grip forces averaging 2.78N (std. dev. 0.96N).
Bi-Modal Pneumatic Kinesthetic-Tactile HFS
The results from the experiments conducted using the pneumatic variation of the kinesthetic force feedback system show (Fig. 2A) that the average grip force is significantly lower (ANOVA42, p-value (p) = 2.4E-9, Degrees of freedom (Df) = 3, F-statistic (F) = 24.59) when feedback is provided. Compared to the no feedback condition, average forces were lower when tactile feedback (Tukey43, p = 0.017), kinesthetic feedback (Tukey43, p = 1.66E-6) or hybrid feedback (Tukey43, p < 1.0E-16) are provided. The bi-modal kinesthetic-tactile HFS also performs better than both tactile-only (Tukey43, p = 5.64E-8) and kinesthetic-only (Tukey43, p = 0.0027) feedback conditions.
With regards to the peak grip force, significant difference was also observed between the groups (ANOVA42, p = 0.0007, Df = 3, F = 6.876). While no significant improvement can be seen between the tactile feedback and the No Feedback condition, both kinesthetic (Tukey43, p = 0.0008) and hybrid HFS (Tukey43, p = 0.0001) conditions display a significant reduction in peak grip force compared to when no feedback was present.
The number of faults (Fig. 2A), i.e. the number of times the subject dropped the peg, appears to be significantly different among the groups as well (ANOVA42, p = 0.005, Df = 3, Chi-Square Statistic (χ2) = 12.702). While there are no significant differences between no feedback and the feedback conditions, a higher number of faults is observed when hybrid HFS was activated, compared to the tactile feedback group (Tukey43, p = 0.012).
The results from the peg transfer studies show the clear benefits of providing haptic feedback in reducing grip force. All feedback modalities performed significantly better than the no feedback condition. Most importantly, there is also a clear indication that the multi-modal kinesthetic-tactile feedback system is significantly better than both single-modality feedback solutions, benefiting from the synergistic relationship from the activation of mechanoreceptors in the skin and muscle. This leads to a more natural sense of touch, allowing the subjects utilizing multi-modal HFS to apply forces nearly 50% less than when no feedback is provided.
Ex-Vivo Evaluation of Bi-Modal Kinesthetic-Tactile HFS
The ex-vivo bowel run experiments were designed with the goal of allowing comparison against the single modality tactile feedback system demonstrated in our previous work26. The results of the ex-vivo study in novice subjects (Fig. 2B) show a significant reduction in average grip force (T-test44, p = 0.00025, Df = 9, T-Statistic (T) = 5.827) and peak grip force (T-test, p = 0.00024, Df = 9, T = 5.861) in the hand that received feedback from the bi-modal HFS.
Looking at the inter-subject variation (Fig. 2B), it can be clearly seen that with regards to both average and peak grip-force, nearly all novice subjects benefited from the presence of the bi-modal kinesthetic-tactile HFS. These results clearly highlight the advantage of the newly developed multi-modal HFS over the traditional single-modality implementations26. While these previous studies showed large inter-subject variation and limited conclusive evidence as to the benefits of tactile feedback, the results of the ex-vivo bowel run (Fig. 2B) clearly indicate the consistent benefits provided by the multi-modal HFS.26. While these previous studies showed large inter-subject variation and limited conclusive evidence as to the benefits of tactile feedback, the results of the ex-vivo bowel run (B) clearly indicate the consistent benefits provided by the multi-modal HFS.
In expert surgeons, the same trend followed (Fig. 2C) with the hand receiving feedback showing a significant reduction in average grip force (Wilcoxon45, p = 0.0234, V = 2). No significant difference was observed in peak grip force (T-test44, p = 0.677, Df = 7, T = 0.434) and the number of faults for expert surgeons. Looking at the inter-subject variation, most subjects benefited from the presence of HFS, though the impact was less apparent than in novices.
The effectiveness of the bi-modal kinesthetic-tactile HFS warranted further study of the multi-modal HFS, utilizing vibration to help reduce excessive peak grip forces.
The results of the study based on the tri-modal HFS show (Fig. 3) a significant reduction in average grip force when the tri-modal HFS is used compared to the bi-modal kinesthetic-tactile feedback for novice subjects using the da Vinci surgical system (T-test44, p = 0.00039, Df = 9, T = 5.473). The same pattern followed with the peak grip force where the tri-modal feedback system showed a lower peak grip force compared to the bi-modal HFS (T-test44, p = 0.0049, Df = 9, T = 3.702). Figure 3 also shows the average grip force for peg-transfers using human hands (data from Fig. 1A) in green overlaid on top of the data for this study as a point of comparison.
Robotic surgical systems have long suffered from a lack of comprehensive haptic feedback solutions and this limitation has had significant impact on the adoption of surgical robots and in clinical outcomes7,8,9,10,14,16. Research in areas of haptic feedback for surgical robotics has been abundant, with recent implementations of single-modality feedback solutions showing improvements in outcomes of robotic tasks7,10. This investigation aimed to expand beyond single modality feedback technologies and study the potential benefits of a multi-modal feedback solution over traditional uni-modal haptic feedback systems.
The results of the experiments using the multi-modal HFS clearly highlight the benefit of multi-modal feedback over single modality feedback solutions. The observed improvements confirm that great benefits can be gained by targeting multiple mechanoreceptors in the skin and muscles simultaneously. This synergistic activation leads to a more natural sense of touch, allowing subjects to perform the task with forces nearly 50% lower than when no feedback is present (Fig. 2A).
An interesting observation from the study that is worth discussing is the larger number of faults (i.e. peg drops) in the bi-modal HFS compared to the tactile-only condition (p = 0.012). Based on the large standard error mean, it is clear that there is also significant variation among subjects. The reason for this variation is in fact due to the way the pneumatic kinesthetic feedback functions. The high pressures used for higher feedback levels makes compressing the grasper at the surgeon’s console quite challenging due to increased resistance. When this resistance is coupled with a high tactile feedback level, it appears to result in the subject suddenly relaxing their hold on the graspers, and consequently often dropping the peg. This variation is more of a learned behavior most likely caused by a lack of experience with the feedback system. Even though training with the haptic feedback system can eventually eliminate this behavior (as we observed in a series of follow up bench tests), the correct way to ultimately deal with this issue is to develop an adaptable feedback system. Such a feedback system could learn from the user’s behavior and automatically lower the pressure levels to help reduce grasper resistance and hence the number of peg drops.
Beyond the peg-transfer experiments, the results of the ex-vivo studies help to highlight the benefits of multi-modal HFS in more life-like surgical conditions. As expected, the bi-modal HFS reduced applied grip force in both novices and experts. However, the results also showed that such a feedback system can have a much more significant impact on the less experienced novices. In fact, novices significantly benefited from the presence of feedback with regards to the applied peak grip force. On the other hand, for surgeons, the applied peak grip force was approximately 4N, with or without feedback. That is on par with the peak grip force in novices after feedback was provided. These results help highlight the importance of training and how expert robotic surgeons have learned to compensate for the lack of feedback through the use of visual cues and overall experience. Although experienced surgeons did not reduce peak grip force when receiving feedback, the reduction in average grip force is indicative of their extensive training in robotic surgery. The improvement solely in average grip force and not peak grip force for experienced surgeons reflects their prior training in reducing applied forces and completion of tasks using solely visual cues. Whereas novice subjects lacking extensive experience with robotic systems, lack the proper skills to complete surgical tasks with or without feedback, resulting in the greater room for improvement in average grip force and peak grip force. Having said that, the average grip force data for expert surgeons under the no feedback conditions show that there is still room for improvement. That is, even for experienced surgeons, the applied average and peak forces without HFS are still high enough to induce tissue damage, as prior studies found that average forces as low as 1.25N are able to induce sites of tissue damage with number of sites increasing with greater applied force46, making the multi-modal HFS valuable even for expert robotic surgeons due to the evidence of average force reduction.
Even with the bi-modal kinesthetic-tactile HFS, peak grip forces which lead to significant tissue damage remain an issue. The underlying issues are the lack of training and the initial grasping action from novices that involves completely closing the console’s grasper with maximal force. While additional training can help reduce these peak forces, an alternative approach was to utilize vibrotactile feedback as a warning system which quickly activates a negative feedback response in the brain. The results clearly show that tri-modal HFS performs even more effectively than bi-modal HFS. This further confirms the initial hypothesis that controlled recruitment of additional mechanoreceptors can lead to further improvements in effectiveness of HFS.
The overlaid data from the peg-transfers performed by human hands also tells an interesting story (Fig. 3). While the sensors installed on one finger may have slightly hindered the subject’s sense of touch, the data still served as a good first level benchmark for average grip force. These data clearly highlight how close the multi-modal haptic feedback system has come to achieving what may be considered the ideal grip force. Of course, peak grip forces are still significantly higher, even with the haptic feedback system, showing that there is room for further investigation and optimization of multi-modal HFS in robotic surgery.
Materials and Methods
Multi-Modal HFS System Architecture
The multi-modal HFS architecture was designed to allow significant flexibility in the way sensor data could be mapped to activation of feedback actuators. The goal was to develop a low latency HFS that could be easily reconfigured for application in various robotic surgical tasks without requiring any reprogramming or redesign of the sensor boards and/or control boards. An overview of this system architecture can be seen in Fig. 4.
In this design, a software engine (i.e. Haptics Manager) is developed using C# and the Net framework for use as the central processing unit in the multi-modal HFS (see Supplemental Materials Section A). The sensor board and control board functionalities are thereby simplified. The sensor board (Fig. 4B) would thus be responsible only for transmitting raw sensor data to the Haptics Manager software, using a ESP8266 WiFi module programmed for UART-to-TCP passthrough data transmission. The control board also relies on the same communication modality for receiving control packets from the Haptics Manager software. All logic and control decisions are made in the Haptic Manager software. The software handles filtering of sensor data, deciding what actuation modality and level setting should be invoked in response to the current sensor values. A control packet is then created with the target actuation levels and sent to the control board which activates the appropriate feedback modality at the actuators mounted to the console graspers.
Because of the small size of incisions in Minimally Invasive Surgery, robotic and laparoscopic instruments are generally designed to be able to move through trocars with diameters less than 12 mm. This means that any sensor that is designed to be installed on either the end effector or on the shaft of the tool must be significantly smaller than 12 mm wide (Fig. 4A). Furthermore, the range of forces applied by different instruments can vary significantly, from 0–5N for instruments such as the da Vinci Cadiere forceps to more than 20N for the da Vinci Prograsp forceps. To meet the necessary requirements, normal force sensing is achieved through commercially available Tekscan FlexiForce piezoresistive sensors.
Two types of tactile feedback modalities are investigated for targeting mechanoreceptors in the skin of the fingertips. Normal force pneumatic tactile feedback was provided using a multiplexed, 5-level solenoid valve array. Controlling the activation of each solenoid valve using analog switches allows for 5 distinct pressure outputs to actuators in contact with the operator’s fingertips. This technology has previously been discussed in the literature and evaluated as an effective method for reducing grip forces in robotic surgery26. An alternative, depressed-membrane feedback actuator design was used to reduce the impact of sensory adaptation31 and allow for installation of vibration motors which provide vibrotactile feedback (Fig. 4D)31.
Kinesthetic Force Feedback
Kinesthesia and proprioception are often interchangeably used in the literature to refer to the awareness of the position and movement of the limbs and the muscles. Proprioception is more strongly linked to the feedback mechanisms within the neuromuscular system, as it involves the process of sending information about the movement of the body to the brain in order to make proper adjustments to muscle movements. Proprioceptive Feedback, or as it is more commonly known, Kinesthetic Force Feedback (KFF), is therefore designed to trigger activation of mechanoreceptors in the muscles, particularly the Golgi tendon organ (GTO). The GTO is responsible for measuring tension in the muscles and sending this sensory information to the brain to help create a sense of resistance50.
Conventional implementations of KFF often rely on motors installed on the joints of the robotic controls to provide feedback. This approach, particularly for grasping applications, can be susceptible to a jerking phenomenon which results from the interaction of KFF with human reflex responses (see Supplemental Materials Section B). A pneumatic kinesthetic force feedback system was developed as an alternative, aiming to provide a more natural sense of kinesthesia. This solution relied on the placement of a pneumatic tube between the graspers of the master console.
The increase in air pressure (0–19 PSI) inside the tube would lead to constriction of the grasper’s ability to close, hence resisting the grasping action. The higher the pressure inside the tube, the more the grasper would resist being closed, indicative of stronger kinesthetic feedback.
One of the disadvantages of the motor-based kinesthetic feedback design is its dependency on modifications to the robotic console. The pneumatic feedback was instead designed as an add-on solution with compatibility in mind. The 3D printed pneumatic actuators, which already supported vibration feedback, were modified even further to allow installation of the kinesthetic feedback system directly on the same actuator (Fig. 4F).
In order to control the air pressure for this kinesthetic feedback system, it was necessary to investigate an alternative pressure regulation system compared to the one in use for the normal force tactile feedback. The reason for this change from the quantized levels of our previous tactile feedback system46 was the lack of scalability. With the utilization of the multiplexed solenoid valve array, each additional pressure level, for each actuator, required one additional solenoid valve. For this reason, providing a pressure control system that would simulate continuous pressure regulation would require an extremely large number of solenoid valves, a design which, due to cost and complexity of the system, would not be practical. A more compact pressure regulation system with a high number of pressure levels (i.e. simulating continuous pressure regulation), was critical for two reasons: (1) large pressure changes can reduce user performance, particularly in a kinesthetic feedback where sudden changes in feedback can result in the user dropping the peg, and (2) a user-based, adaptive feedback requires pressure levels to be variable and changeable electronically.
Previous research had investigated the possibility of utilizing electro-pneumatic pressure regulators. However, these systems generally have slow response times and produce significant vibration during pressure changes46. To resolve these issues, a dual-valve continuous pressure regulation system was developed that would provide continuous pressure regulation with low latency.
The dual-valve pressure regulation technology relied on an Asco Sentronic D electro-pneumatic valve placed in series with a SMC solenoid valve. The Sentronic D valve utilizes a direct acting proportional coil to control output pressure. The benefit of this type of electro-pneumatic valve is its stable pressure output. The stable response, however, comes at the expense of system response times. System response time can be reduced, but reducing the response time comes at the expense of large pressure overshoots when pressure levels are being changed. The Sentronic D’s output pressure is controlled by providing a voltage of 0–10 V to its analog control pin.
This large pressure overshoot could not be allowed to reach the pneumatic tubing. To resolve this issue, a dual-input solenoid valve was placed in series after the Sentronic D valve (Fig. 4F). The solenoid valve, when in the off position, would allow air flow from the first input to its output. When turned on, the solenoid valve would allow air flow from the second input to its output. The haptic feedback controller was then programmed to change the solenoid valve to its on position right before changing the pressure of the Sentronic D valve. By feeding the output of the solenoid valve back to its second input port, the system would maintain the output pressure until the Sentronic D’s output pressure had stabilized. At that time, the solenoid valve would be turned off, and the new pressure would flow to the actuators.
The Sentronic D valve has a built-in controller which can be programmed using Asco’s data acquisition software. By modifying various control parameters of the valve, a stable pressure output was achieved while also reducing the total response time of the dual-value system to <70 ms (Average: 65 ms, Std. Dev.: 7.5 ms, Max: 67 ms, Min: 50 ms).
Grip-Force Reduction Studies: Experimental Methods for Evaluation of Hybrid HFS
All methods were previously approved by the Institutional Review Board at UCLA and carried out in accordance with relevant university guidelines and regulations. Per approved IRB protocol, informed consent was obtained from all subjects.
Evaluation of Bi-Modal Pneumatic Kinesthetic-Tactile HFS
These experiments targeted the evaluation of a hybrid kinesthetic-tactile HFS using the pneumatic implementation of KFF on the da Vinci IS 1200 surgical system. The control system for the pneumatic kinesthetic-tactile hybrid HFS was implemented by configuring the logic engine of the Haptics Manager software to utilize the dual-value continuous pressure regulators as a means of changing the pressure in the pneumatic tube responsible for providing kinesthetic force feedback (Fig. 5A).
A total of 15 novice subjects with little to no experience with robotic surgery were recruited to perform single handed peg-transfer tasks using a da Vinci IS1200 surgical system. Subjects were given a 2-minute training period prior to the start of the study to familiarize themselves with the robotic system. This training period was sufficient in most cases since the use of clutch, camera and most other complex da Vinci operations were not allowed. In order to eliminate any bias toward either hand, nearly half (7) of the subjects received feedback on the right hand while the remaining subjects (8) received feedback on the left hand. All subjects were right handed. Each subject was asked to perform four peg transfers per trial. To eliminate any bias towards the position in which pegs may be dropped, if the subject dropped the peg, the proctor would quickly reset the peg to its original position.
For this study, a FlexiForce A201 sensor was installed on da Vinci Fenestrated Bipolar forceps. Each subject performed the peg transfer tasks four separate times as part of four trials, with each trial performed under different feedback conditions: (1) No Feedback (2) Normal Force Tactile Feedback (3) Pneumatic Kinesthetic Force Feedback (4) Hybrid Kinesthetic-Tactile Feedback.
During the trial, the number of faults (i.e. number of times the subject dropped the peg), time-to-completion, and the grip force were recorded. Statistical analysis of average grip force was performed using Repeated Measures ANOVA42 after a Log2 transform was used to meet the normality assumption. Repeated Measures ANOVA42 was also used for analysis of peak grip force. Statistical analysis of the number of faults was conducted using Ordinal Repeated Measures ANOVA. Follow up post-hoc analysis using Tukey correction43 were performed when p-value was less than 0.05.
Ex-Vivo Evaluation of Bi-Modal Kinesthetic-Tactile HFS
In order to evaluate the bi-modal kinesthetic-tactile HFS under conditions more representative of real-world surgical applications, an ex-vivo study was designed. Previous in-vivo work on tactile feedback systems have shown that there is a clear correlation between applied grip force and the number of damaged tissue sites46. Since the correlation of applied grip force with tissue damage already exists, it is therefore possible to rely on ex-vivo experiments for evaluation of the kinesthetic-tactile HFS.
These experiments relied on the same control system as the previous pneumatic kinesthetic-tactile HFS. FlexiForce A201 sensors were installed on two da Vinci Cadiere forceps. Feedback actuators were installed on both left and right controls of the da Vinci system console. Only one hand however received feedback throughout the trial. This experimental design choice allowed one hand to act as a control for the other. To eliminate any bias toward either hand, half of the subjects performed the task with feedback on one hand and half with feedback on the other.
Subjects were asked to run a porcine bowel, approximately 30 cm in length (Fig. 5B). Each subject performed the task only once. For this investigation, two groups of subjects were recruited. The first study was performed by recruiting 10 novice subjects with little to no experience with robotic surgery. The second group of subjects were expert robotic surgeons recruited from Ronald Reagan Hospital at the University of California, Los Angeles. A total of 8 expert surgeons were recruited for participation in this study.
For all subjects, applied grip force was recorded throughout the study. For expert subjects, the number of times the surgeon dropped the tissue with either hand was also recorded to determine any impact of the hybrid HFS on proper handling of the tissue. This parameter (number of faults) was not recorded and analyzed for novices. This experimental design choice was made because of the difficulty that many novice subjects experienced picking up and handling the bowel tissue. This lack of experience and the non-homogeneity of the tissue made this parameter too variable in novices to be valuable without requiring a very large number of subjects to be recruited. For novice subjects, the study controlled for utilization of the clutch30 by prohibiting subjects from using the clutch operation on the da Vinci. The proctor instead adjusted the controls such that no visual-perceptual mismatch51 was present and subjects could perform the tasks without requiring access to the clutch and camera operations.
Statistical analysis for any one of the measured metrics (ex. force, faults, etc.) was performed using a standard student’s paired t-test44 when normality assumption was met, and a non-parametric, paired Wilcoxon Signed Rank test45 when the data were not normally distributed.
Evaluation of a Tri-Modal HFS
A tri-modal haptic feedback system was utilized to provide a third modality of feedback when applied forces moved beyond a certain threshold. Building upon previous understanding of the effectiveness of vibrotactile feedback as a warning system30, these experiments aimed to determine the potential benefits of a tri-modal HFS over the previously tested bi-modal kinesthetic-tactile feedback system in reducing average and peak grip forces in robotic minimally invasive surgery tasks.
The control system for a tri-modal feedback system was implemented by configuring the logic engine of the Haptics Manager software (Fig. 5C).
A total of 10 subjects were recruited to perform a two-handed peg transfer task using a da Vinci IS1200 surgical system. Two Cadiere forceps were installed with FlexiForce A201 normal force sensors. Subjects received bi-modal kinesthetic-tactile feedback on one hand and tri-modal feedback on the other. The tri-modal feedback rules were the same as the bi-modal ones with the difference that vibration feedback was provided beyond level 3 of normal force tactile feedback (2N). A more intense vibration was provided beyond level 4 of normal force feedback (3N). To eliminate bias towards either hand, the hand receiving tri-modal feedback was switched for half of the subjects.
For all subjects, the study controlled for utilization of the clutch by prohibiting subjects from using the clutch operation on the da Vinci. The proctor instead adjusted the controls such that no visual-perceptual mismatch was present and subjects could perform the tasks without requiring access to the clutch operation.
Subjects were asked to perform four peg transfers. In each transfer, the subject would pick up the peg on one side of the field with the arm closest to the peg, pass it to the other arm, and then place it back down on the other side of the field. The applied grip force and the number of peg drops with each hand was recorded throughout the trial. Statistical analysis was performed using a standard Student’s paired t-test44 when normality assumption was met and a Wilcoxon Signed Rank test45 when the data were not normally distributed.
To allow a qualitative comparison of the system with the normal sensory feedback system of the human hand, 6 subjects were asked to perform the same peg-transfer with their own hands (thumb and index finger only) with the sensors used in the study installed on their index fingers. The force values during 10 consecutive peg two-handed peg transfers were used to calculate the average grip force when performing this task using normal human senses.
Kumar, R. & Nayak, B. Robotic versus conventional laparoscopic pyeloplasty: A single surgeon concurrent cohort review. Indian J. Urol. 29, 19–21 (2013).
Munz, Y. et al. The benefits of stereoscopic vision in robotic-assisted performance on bench models. Surg. Endosc. 18, 611–6 (2004).
Moorthy, K. et al. Dexterity enhancement with robotic surgery. Surg. Endosc. 18, 790–795 (2004).
Ballantyne, G. H. Robotic surgery, telerobotic surgery, telepresence, and telementoring. Surg. Endosc. 16, 1389–1402 (2002).
Marescaux, J. et al. Transatlantic robot-assisted telesurgery. Nature 413, 379–80 (2001).
Lanfranco, A. R., Castellanos, A. E., Desai, J. P. & Meyers, W. C. Robotic surgery: a current perspective. Ann. Surg. 239, 14–21 (2004).
Van Der Meijden, O. A. J. & Schijven, M. P. The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: A current review. Surg. Endosc. Other Interv. Tech. 23, 1180–1190 (2009).
Kirkpatrick, K. Surgical robots deliver care more precisely. Commun. ACM 57, 14–16 (2014).
Nayyar, R. & Gupta, N. P. Critical appraisal of technical problems with robotic urological surgery. BJU Int. 105, 1710–1713 (2009).
Enayati, N., De Momi, E. & Ferrigno, G. Haptics in robot-assisted surgery: Challenges and benefits. IEEE Rev. Biomed. Eng. 9, 49–65 (2016).
Okamura, A. M. Haptic Feedback in Robot-Assisted Minimally Invasive Surgery. Curr Opin Urol 19, 102–107 (2009).
Wottawa, C. R. et al. Evaluating tactile feedback in robotic surgery for potential clinical application using an animal model. Surg. Endosc. 30, 3198–209 (2016).
Tavakoli, M., Patel, R. V. & Moallem, M. Haptic interaction in robot-assisted endoscopic surgery: a sensorized end-effector. Int. J. Med. Robot. Comput. Assist. Surg. 1, 53–63 (2005).
Westebring – van der Putten, E. P., Goossens, R. H. M., Jakimowicz, J. J. & Dankelman, J. Haptics in minimally invasive surgery – a review. Minim. Invasive Ther. Allied Technol. 17, 3–16 (2008).
Kitagawa, M., Dokko, D., Okamura, A. M. & Yuh, D. D. Effect of sensory substitution on suture-manipulation forces for robotic surgical systems. J. Thorac. Cardiovasc. Surg. 129, 151–158 (2005).
Bethea, B. T. et al. Application of Haptic Feedback to Robotic Surgery. J. Laparoendosc. Adv. Surg. Tech. 14, 191–195 (2004).
Wagner, C. R., Stylopoulos, N., Jackson, P. G. & Howe, R. D. The Benefit of Force Feedback in Surgery: Examination of Blunt Dissection. Presence Teleoperators Virtual Environ. 16, 252–262 (2007).
Demi, B., Ortmaier, T. & Seibold, U. The touch and feel in minimally invasive surgery. HAVE 2005 IEEE Int. Work. Haptic Audio Vis. Environ. their Appl. 2005, 33–38 (2005).
Martell, J., Elmer, T., Gopalsami, N. & Park, Y. S. Visual Measurement of Suture Strain for Robotic Surgery. Comput. Math. Methods Med. 2011, 1–9 (2011).
McKinley, S. et al. A single-use haptic palpation probe for locating subcutaneous blood vessels in robot-assisted minimally invasive surgery. IEEE Int. Conf. Autom. Sci. Eng. 2015-Octob, 1151–1158 (2015).
Craig, J. C. & Rollman, G. B. Somesthesis. Annu. Rev. Psychol. 50, 305–331 (1999).
Bicchi, A., Canepa, G., De Rossi, D., Iacconi, P. & Scillingo, E. P. A sensor-based minimally invasive surgery tool for detecting tissue elastic properties. Proc. IEEE Int. Conf. Robot. Autom. 1, 884–888 (1996).
Schostek, S., Schurr, M. O. & Buess, G. F. Review on aspects of artificial tactile feedback in laparoscopic surgery. Medical Engineering and Physics 31, 887–898 (2009).
Bholat, O. S., Haluck, R. S., Murray, W. B., Gorman, P. J. & Krummel, T. M. Tactile feedback is present during minimally invasive surgery. J. Am. Coll. Surg. 189, 349–355 (1999).
Benali-Khoudja, M., Hafez, M., Alexandre, J.-M. & Kheddar, A. Tactile interfaces: a state-of-the-art survey. Int. Symp. Robot. 31, 23–26 (2004).
Wottawa, C. R. An Investigation into the Benefits of Tactile Feedback for Laparoscopic, Robotic, and Remote Surgery. at, http://escholarship.org/uc/item/7w74q3wh (2013).
Debus, T., Becker, T., Dupont, P., Jang, T. & Howe, R. Multichannel vibrotactile display for sensory substitution during teleoperation. 2001 SPIE Int. Symp. Intell. Syst. Adv. Manuf. 28–31, https://doi.org/10.1117/12.454744 (2001).
Kaczmarek, K. A., Webster, J. G., Bach-y-Rita, P. & Tompkins, W. J. Electrotactile and Vibrotactile Displays for Sensory Substitution Systems. IEEE Trans. Biomed. Eng. 38, 1–16 (1991).
Massimino, M. J. Improved force perception through sensory substitution. Control Eng. Pract. 3, 215–222 (1995).
Abiri, A. et al. Visual–perceptual mismatch in robotic surgery. Surg. Endosc. 31, 1–8 (2016).
Abiri, A. et al. Depressed-Membrane Pneumatic Actuators for Robotic Surgery. In EMBC 2016 (2016).
Abiri, A. et al. Tensile strength and failure load of sutures for robotic surgery. Surg. Endosc. Other Interv. Tech. 1–13, https://doi.org/10.1007/s00464-016-5356-1 (2016).
Westebring-van der Putten, E. P., Goossens, R. H. M., Jakimowicz, J. J. & Dankelman, J. Haptics in minimally invasive surgery - a review. Minim. Invasive Ther. Allied Technol. 17, 3–16 (2008).
Santos-Carreras, L., Sengul, A., Vollenweider, M. & Bleuler, H. Multimodal haptic interface for surgical robotics. In 4th International Conference on Human System Interaction, HSI 2011, 60–63, https://doi.org/10.1109/HSI.2011.5937343 (2011).
Gallo, S. et al. Towards multimodal haptics for teleoperation: Design of a tactile thermal display. In International Workshop on Advanced Motion Control, AMC, https://doi.org/10.1109/AMC.2012.6197145 (2012).
Okamura, A. M., Verner, L. N., Reiley, C. E. & Mahvash, M. Haptics for robot-assisted minimally invasive surgery. Springer Tracts Adv. Robot. 66, 361–372 (2010).
Schoonmaker, R. E. & Cao, C. G. L. Vibrotactile force feedback system for minimally invasive surgical procedures. 2006 IEEE Int. Conf. Syst. Man Cybern. 3, 2464–2469 (2006).
Lindeman, R., Yanagida, Y., Sibert, J. L. & Lavine, R. Effective vibrotactile cueing in a visual search task. Proc. ninth IFIP TC13 Int. Conf. Human-computer Interact. (INTERACT 2003) 89–96 at, http://books.google.com/books?hl=en&lr=&id=PTg0fVYqgCcC&oi=fnd&pg=PA89&dq=Effective+Vibrotactile+Cueing+in+a+Visual+Search+Task&ots=O8SKwzlyA_&sig=zW0TJW7PWU_5qhyYRH6QPBVjscI%5Cnhttp://books.google.com/books?hl=en&lr=&id=PTg0fVYqgCcC&oi=fnd&pg=PA89&dq=Eff (2003).
King, C.-H. et al. Tactile Feedback Induces Reduced Grasping Force in Robot-Assisted. Surgery. IEEE Trans. Haptics 2, 103–110 (2009).
Ortmaier, T. et al. Robot Assisted Force Feedback Surgery. at, http://elib.dlr.de/52977/1/KüblerMedicalSpringer.pdf.
Wagner, C. R. & Howe, R. D. Force Feedback Benefit Depends on Experience in Multiple Degree of Freedom Robotic Surgery Task. IEEE Trans. Robot. 23 (2007).
Ellen R. Girden. ANOVA: Repeated Measures, Issue 84. SAGE PUBLICATIONS at, https://books.google.com/books?id=JomGKpjnfPcC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false (1991).
Surhone, L. M., Timpledon, M. T. & Marseken, S. F. Tukey’s Range Test. (VDM Publishing, 2010). at, https://books.google.com/books?id=b_OycQAACAAJ&dq=Tukey+statistical+test&hl=en&sa=X&ved=0ahUKEwibmZnM54ndAhVIOKwKHZtuD10Q6AEILzAB.
Urdan, T. C. Statistics in Plain English. at, https://books.google.com/books?id=ai75P2UjvtUC&pg=PA89&dq=student%27s+t+test&hl=en&sa=X&ved=0ahUKEwjqx4L7gIrdAhVHb60KHVsLBNAQ6AEISTAH#v=onepage&q=student’s t test&f=false (Psychology Press, 2001).
Surhone, L. M., Timpledon, M. T. & Marseken, S. F. Wilcoxon Signed-Rank Test. at, https://books.google.com/books?id=gclXYgEACAAJ&dq=Wilcoxon+statistical+test&hl=en&sa=X&ved=0ahUKEwid_NLH6IndAhVLEawKHVNjCCsQ6wEIMjAB (VDM Publishing, 2010).
CR., W. An investigation into the benefits of tactile feedback for laparoscopic, robotic, and remote surgery. at, https://escholarship.org/uc/item/7w74q3wh.pdf (University of California, Los Angeles, 2013).
Covidien. Covidien Trocar. at, https://abm-website-assets.s3.amazonaws.com/surgicalproductsmag.com/s3fs-public/Covidien.jpg.
Intuitive Surgical. Cadiere Single Straight up. at, https://www.intuitivesurgical.com/test-drive/assets/images/instruments/Cadiere_Single_Straight_up.jpg.
King, C. H. Cadiere graspers with pressure sensor. at, http://doi.ieeecomputersociety.org/cms/Computer.org/dl/trans/th/2009/02/figures/tth20090201031.gif.
Hale, K. S. & Stanney, K. M. Design Guidelines from Human Physiological, and Neurological. Haptic Render. - Beyond Vis. Comput. 33–39, https://doi.org/10.1109/MCG.2004.1274059 (2004).
Abiri, A. et al. Visual-Perceptual Mismatch in Robotic Surgery. Surg. Endosc (2017).
This research was partially supported by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award number R01EB019473-01. We thank our colleagues who provided insight and expertise that greatly assisted the research.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Abiri, A., Pensa, J., Tao, A. et al. Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic Surgery. Sci Rep 9, 5016 (2019). https://doi.org/10.1038/s41598-019-40821-1
This article is cited by
Soft actuators-based skill training wearables: a review on the interaction modes, feedback types, VR scenarios, sensors utilization and applications
ROBOMECH Journal (2023)
Scientific Reports (2022)
Journal of Robotic Surgery (2022)
Validation of the portable virtual reality training system for robotic surgery (PoLaRS): a randomized controlled trial
Surgical Endoscopy (2022)
Robotic hepatectomy and biliary reconstruction for perihilar cholangiocarcinoma: a pioneer western case series
Updates in Surgery (2021)