Effect of Cutaneous Feedback on the Perception of Virtual Object Weight during Manipulation

Haptic interface technologies for virtual reality applica have been developed to increase the reality and manipulability of a virtual object by creating a diverse tactile sensation. Most evaluation of the haptic technologies, however, have been limited to the haptic perception of the tactile stimuli via static virtual objects. Noting this, we investigated the effect of lateral cutaneous feedback, along with kinesthetic feedback on the perception of virtual object weight during manipulation. We modeled the physical interaction between a participant’s finger avatars and virtual objects. The haptic stimuli were rendered with custom-built haptic feedback systems that can provide kinesthetic and lateral cutaneous feedback to the participant. We conducted two virtual object manipulation experiments, 1. a virtual object manipulation with one finger, and 2. the pull-out and lift-up of a virtual object grasped with a precision grip. The results of Experiment 1 indicate that the participants felt the virtual object rendered with lateral cutaneous feedback significantly heavier than with only kinesthetic feedback (p < 0.05 for mref = 100 and 200 g). Similarly, the participants of Experiment 2 felt the virtual objects significantly heavier when lateral cutaneous feedback was available (p < 0.05 for mref = 100, 200, and 300 g). Therefore, the additional lateral cutaneous feedback to the force feedback led the participants to feel the virtual object heavier than without the cutaneous feedback. The results also indicate that the contact force applied to a virtual object during manipulation can be a function of the perceived object weight p = 0.005 for Experiment 1 and p = 0.2 for Experiment 2.

The recent growth of the virtual reality industry calls for the means to let a user vividly feel and naturally interact with the virtual environment. In this context, various haptic technologies have been developed to increase the reality and manipulability of a virtual object by creating a diverse tactile sensation. A typical way of providing a tactile sensation to the user is kinesthetic feedback, which applies an interaction force between the user and a virtual object with a robot arm, called a force feedback interface. Most of force feedback interfaces allow the user a 3-degree-of-freedom (3DOF) or 6-degree-of-freedom (6DOF) motion and provide kinesthetic feedback to a tool or a stylus/stylus that s/he is holding during virtual interaction [1][2][3][4] . A specific form of the force feedback interface is a glove-type or exoskeleton force feedback interface, which can represent contact force to the user's fingertips [5][6][7][8][9] . This type of haptic interface has the advantage of letting the user directly feel the contact at his/her fingers while minimizing the constraint on hand movement. Another form of haptic interface is a cutaneous or fingertip haptic interface that stimulates the user's skin at the fingertip 10 . This type of haptic device is developed to emulate the sensation of touching a real object by stimulating RA, SA1, and SA2 type mechanoreceptors [11][12][13][14] . Thus, tactile cues imparted from a cutaneous interface include contact location information [15][16][17] , the orientation of contact area 18,19 , and the change of contact area 20,21 . Most of the aforementioned haptic feedback interfaces have been evaluated in terms of improving the perception of a static virtual object's tactile properties. On the contrary, little is known about the effect of haptic feedback on the tactile perception during object manipulation.
There are theoretical and experimental grounds that the addition of cutaneous stimulus can lead to a change in the perception of an object's physical properties. According to the studies on multi-modal sensory integration, the addition of sensory cue with different modality can result in the shift of percept when the different types of information integrate optimally. Such optimal sensory integration is found in various combination of sensory cues, e.g., visual-haptic [22][23][24] , intra-visual 25 , or audio-visual integration 26,27 . Narrowing the range of modalities to only haptic cues, additional cutaneous feedback, along with kinesthetic cues, are known to affect the haptic perception of physical properties such as friction 28 or surface stiffness 21,29 or object size 30 , as well as hand kinesthesia 31,32 . However, fewer studies have been conducted to evaluate the relative role of cutaneous information in object weight perception, which is a crucial component in manipulation [33][34][35] .
Multiple studies indicate that lateral cutaneous feedback at fingertip provides information on an object's weight. When one lifts an object with a precision grip, tangential torque is applied to fingertip skin depending on object weight [36][37][38] . Noting that the load torque is a function of object weight, a strong correlation is expected between weight perception and tangential torque at the fingertip. Also, cutaneous feedback to the fingertip is known to create an illusory perception of force with or without kinesthetic feedback [39][40][41][42] . Considering its strong correlation to force sensation, we can expect weight perception is affected by cutaneous feedback. The focus of the studies above was mainly on either the relation between object weight and tangential torque profile or the effect of lateral skin stretch feedback that substituted kinesthetic feedback. In the meanwhile, it is hard to find studies on the relative role of lateral cutaneous feedback on object weight perception during manipulation in the view of multi-modal sensory integration.
This study explores the extent that additional lateral cutaneous feedback to kinesthetic feedback affects the perception of virtual object weight during manipulation. Previous studies on haptic feedback mainly focused on its effect on haptic perception of static object's surface properties. On the other hand, the present study analyzes the effect of cutaneous feedback on the weight perception during object manipulation. If we assume that the tactile information from cutaneous and kinesthetic feedback integrate optimally, rendering both cutaneous and kinesthetic feedback can shift the perceived object weight from the one represented only with kinesthetic feedback. We, therefore, hypothesize that the perception of a virtual object weight during manipulation can be affected by the addition of lateral cutaneous feedback. Also, varying the rate of providing cutaneous feedback may affect the perception of virtual object weight. We tested our hypothesis by evaluating the effect of cutaneous feedback on the perceived weight of a virtual object. The experimental task is to judge the perceived weight of a virtual object, manipulated with one or two fingers (Fig. 1). A participant of the experiment could feel the weight of the virtual object rendered with both lateral cutaneous feedback and kinesthetic feedback. Then, we evaluated the effect of the additional lateral cutaneous feedback on the perceived weight. The next section describes the experimental setup for virtual object manipulation and weight perception.

Methods
The experimental tasks of the present study were to move a virtual object with an index finger laterally, and to pull-out and lift-up a virtual object grasped with a precision grip. Since tactile information imparted to a user differs by the type of manual task, we built two different experimental setups. This section describes the system overview and haptic rendering algorithm for the two experiments. Then, we illustrate the experimental paradigm and procedure to evaluate the effect of lateral cutaneous feedback on the perceived object weight.
All methods were previously approved by the KIST (Korea Institute of Science and Technology) Institutional Review Board (IRB approval No. 2019-007) and carried out in accordance with the Declaration of Helsinki for research involving human subjects. Informed written consent was obtained from all participants involved in the experiments. Experiment 1: rendering haptic feedback during virtual object manipulation with one finger. For the experiment with one finger object manipulation in Fig. 1(a), we set up a virtual environment where a user can manipulate a virtual cube with one finger by feeling the contact force with a haptic system. Figure 2 shows the overall experimental system architecture, including the virtual environment, controller, and physical environment. The haptic interface consists of a force feedback interface and a custom-built cutaneous feedback interface ( Fig. 2(a)). The cutaneous feedback interface is worn around the user's index fingertip beneath which a contact plate is located. When the user laterally moves a virtual object, the fingertip skin is deformed by moving the contact plate with a servo motor (model HV75K, MKS Servo Tech, Ilan, Taiwan). The rotation of a joint of the cutaneous interface or the motor is tracked with a magnetic rotary encoder (RM08, RLS, Slovenia). Being attached to the force feedback interface, the user can feel both cutaneous and kinesthetic feedback simultaneously.
During the experiment, a user's fingertip is mapped to a virtual environment and represented as a fingertip avatar ( Fig. 2(b)). When the fingertip avatar touches a virtual object, the normal contact force F n is calculated based on a virtual proxy based haptic rendering model 43,44 as follows: where K, x p , and x f denote virtual object surface stiffness, fingertip proxy contact position, and the fingertip contact position, respectively. Haptic rendering of the lateral friction force is calculated from a contact model based on a Coulomb friction law. A direct application of the Coulomb friction model results in a sudden change of friction force and thus can cause the haptic feedback system unstable. Thus, we modeled the state of static contact and sliding by considering the Coulomb friction model and the velocity of the fingertip based on the piecewise linear approximate Coulomb friction model 45,46 . We calculate the relative lateral speed of fingertip v fo,t to the virtual object at timestamp t as where v f,t and v fo,t−1 are fingertip speed at timestamp t, and virtual object speed at timestamp t − 1, respectively. By incorporating the Eqs. 1 and 2 with a piecewise linear approximate Coulomb friction model, the following friction state model is derived as: where μ a s , , μ a d , , and v th are static and dynamic friction coefficients between the virtual fingertip, and state switching threshold speed, respectively. We modeled the tangential force between the virtual object and the virtual surface based on the Coulomb friction law as follows: where m o bj is the virtual object weight. Equations 3 and 4 decide whether the virtual objects moves or not. When the virtual object moves, its position is updated along with its speed. The force to be exerted by the force feedback, f is a vector summation of the lateral and normal contact force, calculated as www.nature.com/scientificreports www.nature.com/scientificreports/ On each servo loop, the calculated lateral contact force F kin is fed to the controller and the force feedback interface. The desired displacement of the contact plate on the cutaneous interface is calculated as Then, the contact plate is positioned to the desired displacement with a PID controller. The output torque from the controller is converted to a PWM signal generated by a multifunction board (Model 826, Sensoray Co. Inc., OR, U.S.A) and is sent to a motor controller, connected to the servo motor of the cutaneous interface. Figure 2(c) shows the experimental apparatus that can render both lateral cutaneous feedback and force feedback, as a user moves a virtual object with an index finger. The interaction force due to the friction between a fingertip avatar and a virtual object was rendered for force and cutaneous feedback ( Fig. 2(c)). The force feedback interface directly applied the calculated force F kin with a commercially available force feedback interface (PHANToM Premium 1.0, 3D Systems, SC, USA). The custom-built haptic interface ( Fig. 2(a)) rendered lateral cutaneous feedback by moving a contact plate placed beneath a participant's fingertip. Experiment 2: rendering haptic feedback during virtual object manipulation with a precision grip. For the experiment with two-finger object manipulation in Fig. 1(b), we set up a virtual environment where a user can feel when lifting a virtual cuboid grasped with a precision grip. Figure 3 shows the overall haptic system architecture, including virtual environment, controller, and physical environment. Two force feedback interfaces provides a user with kinesthetic feedback as s/he grasps a virtual object with a precision grip. A cutaneous feedback interface ( Fig. 3(a)) was attached to the end-effector of each force feedback interface and a contact skin rotates fingertip skin, which emulates the torsional skin deformation as an object rotates due to its weight. A magnetic rotary encoder (RM08, RLS, Slovenia) with the resolution of +0.3 deg is attached at the bottom of the contact disk to track the disk's rotation angle. A pair of springs tightly fixes the haptic interface to a user's fingertip at the distal interphalangeal (DIP) joint.
Haptic rendering of the force and cutaneous feedback for manipulating an object with a precision grip was calculated in a similar way to Experiment 1. When a user holds a cuboid with his/her index finger and thumb ( Fig. 3(b)), we can calculate the frictional torque at the surface of the contact point (red circle in the figure) as (μ s : friction coefficient, r disk : rotating disk radius, and F n : normal contact force) 47 . Then, applying the Coulomb friction model 48 on the torque, we get the friction model as follows: where τ e and x r are the torque due to the object weight and the displacement vector from the contact point to the center of mass, respectively. F n is the normal contact force in Eq. 1. Then, total torque at the fingertip contact point τ tot is We can calculate the rotation of the contact plate from τ α = I tot (I: the moment of inertia and α: angular acceleration), and ω with Euler integration. The rotation of the contact plate is then updated as follows: to avoid the effect of haptic weight interface on the perception of virtual object weight. Then, total force to be exerted by each force feedback interface F kin is, kin v n where F n is the normal contact force calculated by the Eq. 1. Figure 3(c) shows the experimental apparatus that can provide kinesthetic feedback and cutaneous feedback to the participant. Two commercially available force feedback interfaces (Touch, 3D Systems, SC, USA) directly rendered the force vector F kin as a vector sum of normal-directional contact force F n and the weight of the virtual cuboid, F v . A cutaneous interface (Fig. 3(a)) is attached to the end-effector of each force feedback interface to provide cutaneous feedback by rotating contact plate.
Friction coefficients of real object surfaces. We measured the friction coefficients between real objects to estimate the degree of slipperiness/roughness of the virtual friction coefficient for the experiments. The friction coefficient was measured for three surface materials (metal, plastic, and silicon), and two sliding object materials (metal, and silicon). We employed a typical inclined slope method to measure the static friction coefficient, where the friction coefficient was decided as a tangent of the angle that the object begins to slide. For the sake of measurement accuracy, we attached markers on the sides of the slope, video-recorded the sliding, and derived the sliding angle by analyzing the image at the moment of sliding. The measurement was conducted three times for each condition. The static friction coefficients between two materials were 0.21 ± 0.01 (metal-metal), 0.16 ± 0.03 (metal-plastic), 0.65 ± 0.12 (metal-silicon), 0.43 ± 0.03 (silicon-plastic), and 0.55 ± 0.11 (silicon-silicon).
Experimental procedure. For the experiments, we employed a one-up one-down adaptive procedure to match the perceived weight of a virtual object rendered with both cutaneous and kinesthetic feedback to that rendered only with kinesthetic feedback 49 . The adaptive procedure estimates the PSE of comparison stimuli that is perceived to be equivalent to the reference stimulus.
Each participant conducted four (six) experimental runs by reference object weight and l f for Experiment 1 (μ s for Experiment 2). The order of reference stimuli was randomized for each participant. On each trial, a participant was presented with two virtual objects: a reference object rendered both with cutaneous and kinesthetic feedback and a comparison object rendered only with force feedback. If the participant responded that the reference object felt heavier than the comparison plane, the weight of the comparison plane decreased. Otherwise, the weight of the comparison plane increased. The initial weight was 400 g and 300 g for Experiments 1 and 2, respectively, considering the stability of the force feedback interface used for each experiment. The step size of increasing/decreasing the weight changed from 100 g to 25 g after the first three reversals of the responses. Each experimental run terminated after 12 reversals of the responses at the smaller step size. The total number of trials for one experimental run typically ranged between 30 and 40.
Before the main experiment, a participant was seated in front of a computer and asked to wear noise-canceling headphones (MDR10RNC, Sony, Tokyo, Japan). Then, s/he inserted his/her index finger inside cutaneous interface (Figs. 2(a) and 3(a)). Then, a training session was initiated where the participant could feel the weight of a virtual object rendered with both cutaneous and force feedback or only with force feedback. Virtual fingertips were visually displayed as green spheres along with the virtual object. When the participant felt ready for the main experiment, the training session terminated.
In the main experiment, the order of presenting reference and comparison stimuli was randomly decided on each trial. In the beginning, fingertip avatars and a virtual object were displayed on the screen. In Experiment 1, a participant was asked to laterally move the virtual object with an index finger and feel the object weight. In Experiment 2, a participant was asked to grasp and lift up the cuboid laid on a virtual plane with a precision grip and feel the object weight. During the main experiment, white noise was played on headphones to block possible audio cues from the haptic interface. By hitting the enter key, the participant moved to the next phase to feel the other stimulus. Then, the participant was asked to indicate which stimulus felt heavier. For each trial,

Results
We conducted two psychophysical experiments by how to manipulate the object (Fig. 1), to investigate the effect of the cutaneous feedback on the perceived object weight. In the first experiment, participants moved a virtual object on a floor sideways by feeling cutaneous and kinesthetic feedback, to perceive the object weight. The second experiment evaluated the effect of cutaneous feedback on the perceived weight of a virtual object grasped and lifted with a precision grip. For both experiments, we also investigated how the rate of providing cutaneous feedback affects the perceived weight of the virtual object. The following subsections describe the experimental setup and report experimental results. Experiment 1: Perceived weight of a virtual object manipulated with one finger. The goal of this experiment is to evaluate the effect of cutaneous feedback on the perceived weight of a virtual object manipulated with an index finger. Total 12 subjects participated in the experiment which matched the perceived weight of a virtual object rendered both with cutaneous and kinesthetic feedback to the one rendered with kinesthetic feedback only.
In the main experiment, a participant laterally manipulated a virtual cube (dimension: 3 cm × 3 cm × 3 cm) with a weight of 100 and 200 g, which is about as heavy as a hand-held device, with an index finger. We decided the weight of the virtual object to be around that of a handheld device while conserving the stability of the force feedback interface. The friction coefficients were 0.74, 0.57, and 0.25 for fingertip-object static friction (μ a,s ), fingertip-object dynamic friction (μ a,d ), and object-floor friction respectively (μ b ). We intentionally chose the friction coefficients for fingertip-object (∼silicon-metal) to be higher than that for object-floor (∼metal-metal). Then, the object can slide when sufficient normal and lateral force is applied. For the rendering of cutaneous feedback, we define l F , which is the ratio of contact plate displacement to the lateral force. We used two l F values, 0.005 and 0.01 m/N for the experiment to evaluate the effect of cutaneous feedback intensity on the perceived weight of a virtual object.
A one-up one-down adaptive procedure was employed to derive the point of subjective equality (PSE) for a reference stimulus rendered with cutaneous and kinesthetic feedback, matched to the weight of a virtual object rendered only with kinesthetic feedback. Both cutaneous and kinesthetic feedback was provided to a participant for the reference stimuli while only kinesthetic feedback was provided for the comparison stimuli. Figure 4(a) shows the average PSE estimate of a virtual object weight plotted as a function of reference object weight by two l F values. We compared the PSE estimate to the reference object weight, to see the effect of cutaneous feedback on the perceived object weight. The results of one-sample t-test ( ]. Overall, the addition of lateral cutaneous feedback led the participants to feel a virtual object heavier than the one rendered with only kinesthetic feedback. The modulation of l F did not affect the perception of virtual object weight. Figure 4(b) shows the mean of the maximum normal-directional contact force by l F as a function of reference object weight, m ref . We conducted a two-way repeated measure ANOVA to determine the effects of l F and m ref on the maximum normal-directional contact force. We found no significant interaction of the two factors [ = .
= . ]. In Fig. 4(c), the mean of the maximum lateral contact force is shown by l F as a function of reference object weight. When we conducted a two-way repeated measure ANOVA with the factors l F and m ref , no significant two-way interaction was found between the two factors [ = . F(1, 11) 0 046, η = .
= . ]. Overall, the addition of lateral cutaneous feedback to force feedback resulted in the increase of the perceived weight of a virtual object during one-finger manipulation. It is also notable that the reference object weight significantly affected both the normal and lateral force contact force. We discuss the implication of the findings in the discussion section. Experiment 2: Perceived weight of a virtual object grasped and lifted with a precision grip. The goal of this experiment is to examine the effect of torsional cutaneous feedback on the weight perception of a virtual object held with precision grip during object pull-out and lift-up task. A new group of 12 subjects participated in the experiment, which matched the perceived weight of a virtual cuboid with the dimension of 6 cm × 5 cm × 14 cm, rendered both with cutaneous and kinesthetic feedback to the one rendered with kinesthetic (2020) 10:1357 | https://doi.org/10.1038/s41598-020-58247-5 www.nature.com/scientificreports www.nature.com/scientificreports/ feedback only. On each trial of the main experiment, a participant was asked to pull out a virtual cuboid attached to a virtual wall, by grasping the object with a precision grip. Then, the virtual object tilted forward, and the participant could feel its weight rendered with kinesthetic and cutaneous feedback.
The weight of the virtual cuboid is 100, 200, and 300 g, which are about as heavy as a hand-held electronics including a smartphone or a tablet. To see the effect of varying cutaneous feedback on the perceived object weight, we used two values for the friction coefficient μ s between a participant's fingertip and the cuboid, 0.35 (∼silicon-plastic) and 0.7 (∼silicon-metal). As was in the first experiment, we used a one-up one-down adaptive procedure to derive the point of subjective equality (PSE) for a reference stimulus rendered with cutaneous and kinesthetic feedback, matched to the weight of a virtual object rendered only with kinesthetic feedback.
In Fig. 5(a) are shown the average PSE estimates of a virtual object weight plotted as a function of reference object weight by the two μ s values. When we compared the PSE estimates to the reference object weight with one-sample t-test (  The results of Experiment 2 show a similar trend to those of Experiment 1. Additional torsion feedback led the participants to feel the weight of a virtual object heavier than when there was only kinesthetic feedback, during www.nature.com/scientificreports www.nature.com/scientificreports/ the pull-out and lift-up task. It is notable that varying μ s resulted in the change of both perceived weight, and the normal contact force at m ref = 100 g. We discuss the implications of the findings in the next section.

Discussion
The results of the experiments show that the addition of cutaneous feedback to kinesthetic feedback during manipulation led participants to feel virtual object weight as being significantly heavier than that rendered with kinesthetic feedback only. We varied the rate of providing cutaneous feedback for the two experiments. We modulated the rate of the contact plate's lateral motion to the calculated force for Experiment 1. For Experiment 2, we varied the friction coefficient of rotating the contact plate during manipulation. The experimental results indicate that the effect of varying the rate of cutaneous feedback was insignificant except for the reference weight of 100 g in Experiment 2. We also observed that the grip force was affected by the rate of cutaneous feedback for the reference weight of 100 g.
One common feature of the two experiments's results is that the addition of cutaneous feedback to a participant's fingertip led the participants to feel a virtual object heavier during manipulation, than with kinesthetic feedback only. This result can be partly explained with the optimal integration model for multi-modal sensory input. Then, the weight perception can be estimated with the following model: weight K K C where Ŝ K and Ŝ C are the estimates of weight from kinesthetic and cutaneous information, respectively, and w K and w C are their respective weights. If we assume the distribution of Ŝ C is larger than that of Ŝ K , Ŝ weight with the addition of cutaneous feedback will be larger than Ŝ K , as is the case with the present study. To substantiate the model, however, additional experimental grounds are needed to show that Ŝ C > Ŝ K . The effect of the friction coefficient on the perceived object weight in Experiment 2 can be explained with the results of previous studies that investigated the perception of fingertip slip perception during tangential loading. Barrea et al. demonstrated that the participants of their experiment could perceive the slippage at the fingertip with the partial skin deformation due to passive tangential loading 50 . Similarly, Srinivasan et al. showed that the tactile afferents could encode surface slippage at the fingertip 51 . In other words, Ĉ in 10 is a function of the cutaneous feedback. Then, their results suggest that the cutaneous feedback in the present study generated the tactile information indicating the existence of tangential loading. For Experiment 2, the slippage at the fingertip due to cutaneous feedback could have been more significant for the lower friction coefficient of μ s = 0.35 because the virtual object would have rotated more easily. In the meanwhile, the difference of the perceived object between the two friction coefficient was insignificant for larger reference weight, m ref = 200 and 300 g. This result can ascribe to the insensitivity to smaller cutaneous stimulus change as the overall perceived stimulus, Ŝ weight increases with the increase of the reference weight by Weber's law.
The experimental results suggest that the force applied to virtual object during manipulation can be a function of perceived object weight. The maximum contact force in Experiment 1 shows an increasing trend along with the increase of the perceived weight of the virtual object. Also, when we compare the maximum contact force of Experiment 2 ( Fig. 3(b)) to the PSE estimates ( Fig. 3(a)), the effect of varying μ s shows the same trend that the difference is significant for low m ref value of 100 g. Previous studies 36,38 indicate that when one is manipulating an object with a precision grip, s/he applies normal force being proportional to tangential torque at the fingertip. Their findings coincide with the results of Experiment 2 where participants felt an object heavier with larger torsional cutaneous feedback (lower μ s ) and thus exerted larger grip force. Thus, the contact force during object manipulation can be modeled as a function of the perceived object weight, and thus depend on kinesthetic and cutaneous information as follows: www.nature.com/scientificreports www.nature.com/scientificreports/ where Ŝ weight , I K , and I C are the perceived weight, kinesthetic, and cutaneous information, respectively. To summarize, the present study provides grounds that the addition of lateral cutaneous feedback to kinesthetic feedback during object manipulation can increase the perceived weight of the object. Also, the experimental results suggest that the contact force during object manipulation can be a function of the perceived object weight.