Better grip force control by attending to the controlled object: Evidence for direct force estimation from visual motion

Estimating forces acting between our hand and objects is essential for dexterous motor control. An earlier study suggested that vision contributes to the estimation by demonstrating changes in grip force pattern caused by delayed visual feedback. However, two possible vision-based force estimation processes, one based on hand position and another based on object motion, were both able to explain the effect. Here, to test each process, we examined how visual feedback of hand and object each contribute to grip force control during moving an object (mass) connected to the grip by a damped-spring. Although force applied to the hand could be estimated from its displacement, we did not find any improvements by the hand feedback. In contrast, we found that visual feedback of object motion significantly improved the synchrony between grip and load forces. Furthermore, when both feedback sources were provided, the improvement was observed only when participants were instructed to direct their attention to the object. Our results suggest that visual feedback of object motion contributes to estimation of dynamic forces involved in our actions by means of inverse dynamics computation, i.e., the estimation of force from motion, and that visual attention directed towards the object facilitates this effect.


S1.1 Summary
Sarlegna et al. 1 reported changes in grip force pattern caused by a delay in visual feedback as evidence for visual contribution to grip force adjustments.
Here we show that two potential visual contributions to load force estimation can both account for this finding; one based on visual estimation of inertia from object motion 1,2 and the other based on perturbation force estimation based on prediction error of hand position, the latter process often assumed in contexts of motor learning 3,4 and human-computer interaction 5,6 . Importantly, the former contribution is based on inverse dynamics computation whereas the latter is based on forward dynamics computation. Therefore, the distinction relates to the fundamental question on whether the human brain also solves the computational problem of inverse dynamics 2 ; the estimation of force from motion. In the following sections, we will first describe the finding reported by Sarlegna et al. (S1.2) and then describe the two possible hypotheses (S1.3 and S1.4). Finally we will describe the critical distinction between the two hypotheses which we have addressed in our study (S1.5).

S1.2 Finding
Top and bottom panels in Fig. S1 illustrate the setup used in the study reported by Sarlegna et al. 1 . Participants pinched a force sensor which was attached to a wall with an elastic cord. They repeatedly stretched the cord while observing a cursor on a computer monitor that indicated the position of the pinched sensor.
When the motion of the cursor was delayed relative to the motion of the sensor, grip force applied to the sensor tended to precede the load force. The temporal precedence increased as the delay increased within the observed range (phase delay of 03   [rad]).

S1.3 Cursor-inertia hypothesis 1
In order to explain the observed temporal shift of grip force pattern, Sarlegna If we assume that the virtual inertia is significantly smaller compared to the stiffness of the elastic cord such that

S1.4 Perturbation model
While the theory mentioned above explains the temporal precedence of the grip force from the load force, we noticed that a simple assumption, often made in models of force-field adaptation 3,4 , also explains the phenomenon. This assumes   . Therefore, the model also explains the temporal precedence of the grip force.

S1.5 Relation between the two hypotheses
The force estimation process assumed by Sarlegna et al. would require an inverse dynamics model to estimate the inertial force from the motion of the delayed cursor. Meanwhile, the force estimation process described in section S1.3 would require a forward dynamics model to predict the cursor position from which the error would be calculated. The critical difference between the two processes is whether the cursor is interpreted as an object motion or as a hand motion. Our study reported in the main text suggests that the cursor is interpreted as object motion in the context of force estimation and therefore supports the former hypothesis.

Fig. S1
Hypotheses for the result reported by Sarlegna et al.

S2.1 Summary
Considering the earlier finding 7 that visual attention and saccadic eye movements are not completely independent, one could claim that participants may have made saccades to the cursors when asked to direct their attention to them in experiment 2. While the fixations of the participants were visually monitored by the experimenter throughout the experiment, it was difficult to detect small deviations from the fixation or instantaneous saccades to the cursor.
To rule out the possibility that any difference in eye movements or visual input caused the improvement in grip force timing, we replicated the results of experiment 2 using a different set of participants while their fixations were monitored using an eye tracker (SR Research EyeLink II).

S2.2 Method
Ten participants (1 male and 9 female; mean age = 36.5 years old) contributed to the experiment. The task given to the participants were identical to that in     Asterisks along the vertical bars indicate significant difference between the visual conditions. Multiple comparison was controlled based on the Ryan's method. ***, and **** denote p < 0.005, and p < 0.001, respectively.

S3. Bayesian estimation theory for explaining the grip-load force coupling
Körding et al. 8 showed that magnitude estimation of perturbation forces applied to a moving hand depends on Bayesian prior which reflects the statistical distribution of the experienced force magnitudes. The precedence of grip force relative to the load force, examined in our two experiments, can be explained by extending this theoretical framework to multimodal estimation of temporal patterns of load forces linked to self-motor action. Figure S3 illustrates the hypothesis. The task we consider for grip force control is to estimate the temporal pattern of load force associated with the hand action. First, we assume a Bayesian prior of the load force pattern. Considering that we often move a rigid object with a pinch grip, we assumed a Bayesian prior of the load force pattern which corresponds to that of a rigid object held in our hands. This predicts a load force synced with the hand acceleration. Integration of such a prior with a somatosensory estimate of the load force pattern, which should generally represent the actual pattern, predicts that the grip force would sync with the load force when we move a rigid object, but will precede the load force when moving an object with a lagged trajectory. This was actually what we observed. Earlier studies also reported that grip precedence is observed when moving a non-rigid object 9,10 . The framework also explains why the precedence was larger in the large delay condition (note that the difference from the prior is larger in this condition). We also found that grip precedence, averaged across all conditions, gradually decreased during the experiment (statistically significant correlation between trial number and grip precedence averaged across all conditions and participants in the 2 nd experiment; ). The decrease may represent a gradual update of the prior, although it may also reflect muscle fatigue. Secondly, we assume that the visual motion of the object provides an estimation of the load force pattern which is proportional to its acceleration 1,2 (inertial force of the cursor) when attention is directed to it. Since this is roughly equal to the actual load force pattern in our case, Bayesian integration of the visual estimate should decrease the precedence of the grip force. This explains the improvement in the object condition of experiment 1 and the attend-to-object condition in experiment2. Finally, the theory also predicts that delaying the cursor motion would increase the grip precedence when the actual load force acts in the opposite direction compared to the visually implied inertia of the cursor. This was the case in the study by Sarlegna et al. 6 and the increase in the grip precedence was actually observed.
While this theory remains to be tested, earlier studies have confirmed that our brain uses Bayesian integration to estimate the magnitude of forces applied to the reaching hand 8 , grip force control relies on the statistical distribution of the experienced load forces 11 , and the Bayesian computation also applies to timing estimation based on visual cues 12 .

Figure S3
Bayesian account of grip force precedence. Top and bottom panels illustrate how the time of load force peak is estimated from multiple factors.
The horizontal axis represents the phase lag of the load force peak relative to the hand acceleration. Distribution illustrated with black, orange, and blue lines illustrate the distributions of prior, somatosensory estimate, and visual estimate of the time of the load force peak. The distribution illustrated in green represents the Bayesian estimate of the load-force-peak time.