Human pointing motion during interaction with an autonomous blimp

We investigate the interaction between a human and a miniature autonomous blimp using a wand as pointing device. The wand movement generated by the human is followed by the blimp through a tracking controller. The Vector Integration to Endpoint (VITE) model, previously applied to human–computer interface (HCI), has been applied to model the human generated wand movement when interacting with the blimp. We show that the closed-loop human–blimp dynamics are exponentially stable. Similar to HCI using computer mouse, overshoot motion of the blimp has been observed. The VITE model can be viewed as a special reset controller used by the human to generate wand movements that effectively reduce the overshoot of blimp motion. Moreover, we have observed undershoot motion of the blimp due to its inertia. The asymptotic stability of the human–blimp dynamics is beneficial towards tolerating the undershoot motion of the blimp.


Scientific Reports
| (2022) 12:11402 | https://doi.org/10.1038/s41598-022-15016-w www.nature.com/scientificreports/ undershoot blimp motion can be tolerated by the human. Furthermore, the GT-MAB tends to bypass the target position more easily than a mouse cursor displayed on a monitor. We discover that the VITE model may serve as a reset controller used by the human to reduce the overshoot of the blimp motion. This work is a significant extension to our conference paper 19 , where we show preliminary results on modeling human pointing motion with the VITE model, and provide stability analysis. This work presents validation of modeling the pointing motion using the VITE model with experiment data collected by more human subjects. Further, we establish a connection between the reset control method and the VITE model, and discuss the benefits and limitations of the VITE model as a feedback controller with resetting operator, comparing with a linear feedback control law. These discoveries have not been reported in our conference paper 19 , as well as the literature reviewed. The rest of the paper is organized as follows. "Background: VITE model" section provides a brief introduction on the background of modeling human pointing behavior. The problem formulation is presented in "Problem formulation" section. "Stability analysis" section presents stability analysis of the closed-loop system. Experiment setting and experiment results are shown in "Parameter identification and validation" section, and the discussion on human behavior analysis is presented in "Human behavior analysis" section. "Conclusion" section describes the conclusion of the paper based on the experiment results.

Background: VITE model
The VITE model is a second order dynamic model for human pointing motion 20 . Consider a human controls the position of a pointing device, such as a computer mouse, which displays a visible pointer such as a cursor on a monitor. We assume the human intends to move the pointer to a desired location. Let y(t) represent the position of the pointing device under human control. Let u(t) represent the perceived position of the pointer, r t denote the desired position of the pointer, then r t − u(t) represents a difference vector describing the difference between the displayed pointer position and the desired position. The VITE model describes the motion of the pointing device as follows: where η(t) represents an internal state describing how the human perceives the difference vector, which cumulatively integrates the difference vectors over time with a constant gain γ . The operator [·] + d is used to switch the pointing motion off when the displayed pointer bypass its target. It is defined by the following equation where d defines the direction from the pointer to the target position at initial time t = 0. The parameter g is called the go signal, which is a feedback gain describing how the internal state η(t) results in the pointer motion y(t). The signature property of the VITE model is that the displayed pointer tends to move beyond the target, generating an overshoot. This has been confirmed to agree with human behaviors by experimental work on human pointing motion 22,23 .
The pointing task is modeled as a feedback control that generates the wand position as the input to the blimp system. The blimp position is the output that will be controlled to a desired position. This is considered as an output regulation problem for controller design. The VITE model is viewed as a feedback control law that produces the wand position y(t) based on the difference vector r(t) − u(t) . In the context of human computer interface using a computer mouse, the human drives the displayed cursor position u(t) to the target position r t , by controlling the position of the mouse y(t). We notice that the VITE model bears similarity with the reset controller in control theory literature 24,25 . Comparing to regular feedback controllers, a reset controller typically defines a reset condition. At the time that the reset condition is met, the control effort is set to zero. The control effort of the reset controller switches back to non-zero values if the reset condition is not met. Due to the switching condition in (2), the VITE model can be viewed as a variant of the reset controller. However, the human considers the pointing task fulfilled when the reset condition is satisfied, and the control effort y(t) stays at zero for all time after the reset condition is met. It is known that a reset controller might reduce overshoot in the step response of a feedback control system 26 . However, such effect has not been reported in human robot interaction through pointing motion.

Problem formulation
We investigate human pointing motion when interacting with the GT-MAB. as illustrated by Fig. 1. The human operator observes blimp position and then moves a marked wand with its position followed by the GT-MAB. The human will control the blimp towards a target position that is unknown to the blimp. We perform a series of pointing experiments where data on the wand movements and blimp movements are collected.
To simplify the dynamics model, we introduce the Assumption 1 in 19 , that the horizontal position of the target that the human specifies is the same as the horizontal position of the blimp. Due to this Assumption, we use y(t) to represent the vertical location of the wand, u(t) to represent the vertical position of the blimp, and r(t) to represent the vertical location of the target. We will then attempt to model the human pointing motion using the VITE model.

Assumption 1
We assume that when interacting with the blimp, the wand movements can be modeled by (1).  27 . VITE model has been adopted for modeling human pointing motion in many existing work on HCI, e.g. 21 . Hence we assume that the human blimp interaction behavior can be described by the VITE model as well. Experimental justification of this assumption is provided in Section Parameter identification and validation.
From our previous work 14 , the vertical motion of GT-MAB can be described as where m = m RB + m Az , m RB and m Az are the rigid-body mass of the blimp and the added mass for the vertical motion. The term F z (t) = Du(t) where D is the aerodynamic drag coefficient for the vertical movement. As justified in our previous paper 28 , D can be well approximated as a positive constant for indoor miniature blimps. The term f z (t) represents the thrust force in the vertical direction. Feedback controllers are implemented to control the position and heading of the blimp. For convenience, GT-MAB keeps the same heading angle through the entire experiment, and the reference setpoint for horizontal movement is set to a fixed position, so that it will remain at the same horizontal position throughout the experiment. Setpoint of the vertical motion is set to the height of the wand, thus enabling the blimp to track position of the wand. The height controller can be described as f z (t) = k p e(t), where e(t) = y(t) − u(t) represents the difference between the height of the blimp and the height of the wand. k p > 0 denotes the feedback gain. The blimp dynamics and the human pointing dynamics are connected to form a closed-loop feedback system as shown in Fig. 1. The first objective of this paper is formulated as the following problem: Problem 1 Prove that the closed-loop system formed by the human pointing dynamics and the blimp dynamics is asymptotically stable.
Concerns on stability might not be of high priority for human computer interface because mouse movement is agile. For the GT-MAB, however, due to inertia and air drags, the bimp might not react instantaneously to wand movement. A theoretical justification of the asymptotic stability of the closed-loop system will ensure that the human-blimp interaction experiments can be carried with meaningful and predictable results.
After collecting the wand movement data from the experiments, we will identify the parameters of the VITE model for each human subject. The parameters of the VITE model include the unknown gains g, γ , and the unknown reference target positions r t . The problem is formulated as follows: Problem 2 Given a set of wand trajectories, denoted as y(t) collected from multiple experiments, identify the unknown VITE parameters g, γ , and the unknown reference signal r t .
Solution to this problem will also be used to justify that the VITE model can capture the features of human pointing motion in the context of human-blimp interaction.
We have noticed that the VITE model can be viewed as a reset controller used by the human to reduce the difference between the blimp position and the target position. Given many possible feedback controllers that are able to achieve the same goal, we are interested to know why the VITE model was adopted by human. Specifically, we formulate the following problem to compare the VITE model with a linear feedback law: Problem 3 Identify the benefits and limitations of the VITE model as a reset controller in the context of human blimp interaction, comparing to a linear feedback controller without the reset operator.

Stability analysis
In this section, we show that the closed-loop dynamics formed by the VITE model and the blimp dynamics are asymptotically stable. We assume that the target position is located at the origin, and blimp's initial position is lower than the target position. Detailed description and justification of this assumption is provided in 19 .
Given the blimp dynamics described in (3), and the human pointing motion dynamics as in (1), we introduce an augmented state x ∈ R 4 , where elements of x are defined as x 1 x 2 x 3 x 4 T u ηu y T , then the closedloop system dynamics can be written as www.nature.com/scientificreports/ where x + 2 denotes the non-negative portion of x 2 , that is, x + 2 = x 2 when x 2 ≥ 0 , and x 2 = 0 when x 2 < 0 . The closed loop dynamics has the following equilibrium set: E = {x 3 = 0, and x 1 = x 4 = −x 2 , and x 2 ≤ 0}.
We will now examine the stability of the closed-loop system dynamics in (4) considering two cases, the first case where x 2 ≥ 0 , and the second case x 2 < 0 . In both cases, the system dynamics is linear, while a switch in system dynamics happens when x 2 goes from Thus, ż 4 = 0 , and the dynamics of z can be described as For the first case, the following Lemma holds.

Lemma 1 The closed-loop system
Proof We will prove the above lemma using Routh's stability criterion. The characteristic polynomial is The matrix A 1 is Hurwitz if the first column of the Routh array is positive. Denote the first column of the Routh array as a 0 , a 1 , b 1 , c 1 , d 1 . a 0 , a 1 , b 1 and d 1 are guaranteed to be positive given any choice of D, γ , g, k p . If , then c 1 > 0 and the system is stable by the Routh stability criteria.
Now consider the second case where x 2 < 0 . In this case, z 4 has already converge to zero, we will consider the stability of the subsystem ż =Ã 2z , where z = [z 1 , z 2 , z 3 ] T , Ã 2 is the third order leading principal submatrix of A 2 .
Proof The characteristic polynomial is det( For all D ωz , g, γ , k p > 0 , the first column of the Routh array is positive. Therefore, the closed-loop system is exponentially stable.
The above two Lemmas lead to the following Theorem. Detailed proof can be found in 19 .
, if the states start from x 2 > 0 , the states will exponentially converge to the equilibrium set E.

Parameter identification and validation
In this section, we describe the experiment setting for human blimp interaction, and process the experimental data to justify Assumption 1 and to identify the unknown parameters of the VITE model. The stability of the closed-loop system is verified wit the parameters identified from the experimental data. Experiment setting and data processing. Participants Five unpaid participants (2 female, mean age 24 years old) are recruited for the study, all with normal eyesight, and all practiced to be familiar with GT-MAB dynamics.
Task We asked the participant to determine two target positions at different heights at their will, and then drive a GT-MAB from its initial position to the first target position by moving the wand. Once the participant is satisfied with the blimp position, he/she will change the direction of wand movement and drive the blimp towards the second target position. The participant was asked to keep repeating this task for 60 s. One video of the experiment can be found at https:// youtu. be/ 4JavP aOVKio. Note that in some of the experiments, Assumption 1 in 19  Data preprocessing It has been shown that when human performs the pointing motion, the pointer velocity data is noisy 29 . To address this issue, we filter the wand velocity using a Savitzky-Golay filter with a 4th order polynomial and a window size of 81 (8 s), and calculate the smoothed wand trajectory by integrating the filtered velocity. The smoothed wand trajectory, denoted as ỹ(t) , is used as the input data for solving the parameter identification problem.
We divide the smoothed wand trajectory ỹ(t) into several going-up and going-down segments, according to the movement of the wand. Since the trial is considered to be finished once the human stops moving the wand, indicating that the human is satisfied with the blimp height, the time interval of the experiment can be divided based on the up/down motion of the wand, where T − n and T + n denote the starting and ending time of the nth trial, T + n−1 = T − n . The ending time of each trial is the timestep when the wand stops moving, ẏ (t) = 0 . In each set of experiment data, we delete the data points that do not belong to a complete trial. Figure 2 (Right) shows one set of blimp and wand trajectories divided into up/down sections.

Parameter estimation
Let � u = {n ∈ Z|ẏ(t) > 0, t ∈ [T − n , T + n ]} represent the set of all going-up sections. Similarly, let � d = {n ∈ Z|ẏ(t) < 0, t ∈ [T − n , T + n ]} denote the set of going-down sections. Let r t,u and r t,d denote the target position of all the going-up sections and going-down sections respectively. We represent the set of unknown parameters in the VITE model as � = [ĝ,γ ,r t,u ,r t,d ] . Given a set of parameters as well as the actual blimp trajectory u(t) from the experiment, a simulated wand trajectory ŷ(t) can be derived from the following initial value problem, where the wand trajectory is simulated by the VITE model. Further, at the start and the end of each segment, since the human stops moving the wand, the human internal state must be zero. www.nature.com/scientificreports/ Hence we formulate the parameter identification problem as a constrained optimization problem. The optimization problem is minimizing the difference between the simulated and the smoothed wand wand trajectory. To solve the optimization problem, we convert the terminal constraint that the human internal state is zero to a penalty term in the cost function, as follows: where β ∈ R is a parameter chosen to balance the penalty term and the modeling error. We set β = 50 , and solve the above optimization problem using the interior point method 30 . Since the cost function is non-convex with respect to the parameters, we apply the global optimization technique 31 to avoid the solution converging to a local minimum. The identified parameters are given in Table 1. We identify the VITE model parameters γ ,ĝ for both participants. The two participants perform the pointing motion with different model parameters. Further, the identified parameters of each of the participants in different trials show consistency, which is indicated by the small variance of the identified parameters in different experiments.

Justification of Assumption 1. To verify whether the VITE model is applicable to human blimp interac-
tion, we divide the up/down sections into two sets, the training set and the testing set. The training set is the set of complete going-up and going-down segments in the first 35 s, while the test set contains the rest of the trajectory. We first compare the smoothed wand trajectory in experiment with the wand trajectory simulated by the VITE model with the identified parameters. Since the VITE model is a second order dynamical system, we also compare the smoothed wand velocity in experiment with the VITE model simulated wand velocity.
Justification using the wand position data The Root Mean Square Error (RMSE) between the true and the simulated wand trajectory and its variance is given in Table 1. For the two participants, the RMSE for the training set is less than 7.2% of the total change-of-height of the blimp throughout the experiment. The RMSE of the test set is larger than the training set, and is about 11.6% of the total change-of-height of the blimp. The small RMSE and the low variance indicate that the VITE model can accurately describe the human blimp interaction, and it can correctly predicts the human behavior. Figure 3 (Left) shows one set of comparison between the measured and the reconstructed wand trajectory. It can be seen that the two trajectories match well in both the training and the test set. This indicates that the VITE model simulated trajectory is able to describe and predict the human pointing behavior in human blimp interaction.

Justification using the wand velocity data
We also provide justification of the VITE model by computing the RMSE between the true and simulated wand velocity. The RMSE is given in Table 1. The RMSE for the training set is less than 8.5% of the maximum wand velocity throughout the experiment, while the RMSE is less than 15.2% of the maximum wand velocity in the test set. Figure 3 (Right) shows one set of the measured, smoothed and simulated wand velocity, where the simulated wand velocity is computed by ĝη(t) . It can be seen that the simulated wand velocity matches with the smoothed wand velocity, which shows that the VITE model can qualitatively reproduce the wand velocity. The VITE model matches with the wand velocity better at the beginning of each segment, in the acceleration phase. There is relatively larger error when the blimp is about to reach the target position, and the human decelerates the wand. One possible explanation for the modeling error in the deceleration phase is that in the experiment, human is not given a specific target position. Hence the participant does not pay attention to the accuracy of the pointing motion, and tends to drive the blimp towards the target with higher speed than the VITE model predicts.
From the error analysis, we observe that the human blimp interaction can be described and predicted by the VITE model. This is an expected outcome since the VITE model describes a general relationship between the distance from the target to the pointer and the neural commands of muscles contraction for human pointing motion in different interfaces.

Human behavior analysis
In this section, we describe human and blimp behaviors during the experiments. We will compare these behaviors with the behaivors observed in human computer interface experiments. In order to describe observations of the experiment, we use the following visualization techniques in existing literature on HCI: Time-series plots We plot the wand position and velocity over time, to describe the human behavior in time.

Phase space plots
The phase space trajectory describes the evolution of the system state in the VITE model with respect to the system input, which is the blimp position.
Wand and blimp movements. Figure 2 (Right) shows one set of experimental data. We observe that the human moves the wand towards the target with accelerated motion at the beginning. As the blimp goes near the target position, the human slows down the pointing motion, until he/she is satisfied with the blimp position. After the human stops moving the wand, the blimp slows down, and keeps moving up/down for a short period of time before its speed reduces to zero. As shown in Fig. 4, comparing to the identified target position with the blimp trajectory, the stopping position of the blimp in each going up/down segment goes over the identified target position. This overshoot can be explained by the VITE model. Let's take one going-up segment as an example. Suppose the internal state has not reached zero, then the human internal state can be described by integrating the system input, Suppose at time step t ′ , u(t ′ ) reaches r t . This is the timestep that the input to (8) goes to zero, meaning that the human stops accelerating the wand. However, at this timestep, Table 1. Identified parameters and RMSE between the actual and simulated wand trajectory. S1-S5 represent the two human participants, while D1-D15 are the 15 datasets collected in the experiments. traj RMSE m shows the RMSE between the true and the simulated wand trajectory in the training set, and traj RMSE t presents the RMSE in the test set (in m). traj Var m and traj Var t are the variance of the difference between the true and the simulated wand trajectory, for the training set and the test set, respectively. Similarly, vel RMSE m and vel RMSE t shows the RMSE between the true and the simulated wand velocity, in the training set and the test set, respectively (in m/s). www.nature.com/scientificreports/ acceleration is positive, r t − u(t) > 0, ∀t ∈ [0, t ′ ) . Hence the human does not stop the wand movements when the blimp reaches the target position. Instead, when human stops moving the wand, the blimp has already moved over the target position. This explains the overshoot observed in Fig. 4. Moreover, we observe an "undershoot"in the blimp trajectory. Figure 5 (Left) shows the trajectories of the closed-loop pointing system. The human internal state is plotted against the blimp position. From the plot, It can be seen that the blimp first move in the opposite direction of the target position, i.e., in the going-up segments, the blimp first goes down before going up towards the equilibrium set. The system state first moves in the "wrong"initial direction, but then it eventually reverses course and reaches the desired steady state. As shown in  , since the blimp is a second-order system, direction change will take more time than a computer mouse. Hence, after the human stops the wand and switches to the next target, the blimp takes additional time to reduce its speed to zero, before changing its direction to go to the next target. At the initial time of each goingup/down segment, the initial condition of the blimp is not zero. The wrong initial direction of the blimp is the natural response of the blimp driven by the system's initial states. Since the closed-loop system is exponentially stable, the blimp catches up with the wand trajectory after a short period of time, as shown in Fig. 5 (Right). The initial response in the "wrong"direction is an undesirable effect in practice. In human blimp interaction experiments, the user might get confused whether the blimp has finished reaching the previous target, and may suspect that the blimp is malfunctioning. In such cases, incorporating feedback from the blimp in reaction to the human user may provide better interaction experience 18 . Benefit and limitation of the VITE model. We refer to the VITE model without the reset term as the baseline controller, ÿ(t) = gγ (−η(t) + r t − u(t)). It is easy to prove that the equilibrium of the closed-loop system formed by the base controller and the blimp is the origin. When the blimp reaches the target position, we have u = r t . In this section, we aim to identify the benefits and limitations of the VITE model by comparing it against the baseline controller.
Compared with the baseline controller, the major benefit of the VITE model is that it reduces the overshoot of blimp motion. Figure 6 shows comparison between the tracking error of using the VITE model and the baseline controller to track a fixed target position. Under the VITE control, the blimp motion settles at the equilibrium much faster than the baseline controller, with significantly less overshoot. It has been theoretically justified that the reset controller reduces overshoot and settling time of a system under control 25 .
However, the equilibrium of the closed-loop system formed by the VITE model and the blimp is not the origin. Once the reset condition η = 0 is satisfied, the human perceives the task finished, and stops wand movement. Therefore, the VITE model will not drive the tracking error to zero. On the other hand, the baseline controller will drive the blimp position to the origin, achieving better accuracy than using the VITE model.
The use of the reset controller indicates that when interacting with the blimp, human prefers to fulfill the task in shorter time with less overshoot while sacrificing accuracy. This phenomenon is also seen in pointing motion across various interfaces, i.e. HCI using a mouse 29 . The human adopts a two-phase mechanism to fulfill the pointing motion, the surge phase and the corrective phase. The surge phase denotes the initial movement towards the target. In this phase human points in an accelerated motion. After the pointer is adjacent to the target, human uses a slower corrective motion to let the tracking error reach zero 32 .

Conclusion
We investigate interaction between a human user and an autonomous blimp by letting the human control the position of the blimp through wand movements. We verify that the VITE model can describe human generated wand motion when interacting with the blimp. We show that the closed-loop system describing the human-blimp interaction is exponentially stable. The exponential stability tolerates the undershoot behavior of the blimp caused by its inertia. Moreover, the study suggests that the VITE model, as a special reset controller, reduces the overshoot of the blimp motion in human-blimp interaction.