Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Motor learning in real-world pool billiards

## Abstract

The neurobehavioral mechanisms of human motor-control and learning evolved in free behaving, real-life settings, yet this is studied mostly in reductionistic lab-based experiments. Here we take a step towards a more real-world motor neuroscience using wearables for naturalistic full-body motion-tracking and the sports of pool billiards to frame a real-world skill learning experiment. First, we asked if well-known features of motor learning in lab-based experiments generalize to a real-world task. We found similarities in many features such as multiple learning rates, and the relationship between task-related variability and motor learning. Our data-driven approach reveals the structure and complexity of movement, variability, and motor learning, enabling an in-depth understanding of the structure of motor learning in three ways: First, while expecting most of the movement learning is done by the cue-wielding arm, we find that motor learning affects the whole body, changing motor-control from head to toe. Second, during learning, all subjects decreased their movement variability and their variability in the outcome. Subjects who were initially more variable were also more variable after learning. Lastly, when screening the link across subjects between initial variability in individual joints and learning, we found that only the initial variability in the right forearm supination shows a significant correlation to the subjects’ learning rates. This is in-line with the relationship between learning and variability: while learning leads to an overall reduction in movement variability, only initial variability in specific task-relevant dimensions can facilitate faster learning.

## Introduction

Motor learning is a key feature of our development and daily lives, from a baby learning to crawl, an adult learning a new sport, or a patient undergoing rehabilitation after a stroke. The process of learning a real-world motor skill is usually long and complex, and difficult to quantify as tasks are naturally unconstrained and highly variable. Most of the motor learning literature focuses on relatively simple tasks, performed in a laboratory setup or even within an MRI scanner, such as force-field adaptations1,2,3,4, visuomotor perturbations5,6,7,8,9, and sequence-learning of finger tapping or pinching tasks10,11,12,13. These laboratory-based learning tasks enable us to isolate specific features of motor learning and dissect them individually, thus provide elegant experiment designs to verify the experimenters’ hypothesis. As a result, motor learning in the real world is rarely studied. While laboratory-tasks play an important role in our understanding of sensorimotor control and learning, they address a very restricted range of behaviours that do not capture the full complexity of real-world motor control and may overlook fundamental principles of motor control and learning in real-life14,15.

Neurobehavioral mechanisms are subject to evolutionary selection pressures and survive only if they are relevant in natural tasks. Thus, studying operation in natural contexts allows us to evaluate mechanisms the nervous system has been designed for16,17. Example in sensory neuroscience the use of natural sensory stimuli has led to a revolution of our mechanistic understanding of perception18,19. Over the past decade, there were a few notable efforts to study motor learning in unconstrained tasks. One line of research devised more complex tasks for skill learning20,21,22 (e.g. skittles) which were implemented as computer-based gamified tasks that emulate real-world tasks. Others moved away from the computer screen but were still highly constrained; e.g. throwing a frisbee while the subject’s trunk is strapped to the chair to prevent trunk movement23. Another line of inquiry used free-behaviour in real-world tasks such as tool-making or juggling17,24,25,26,27. In these studies, we and others focused on anatomical and functional MRI measurable changes following learning. In a 3-year long complex tool-making apprenticeship experiment17 we were able to quantify changes in motor control precision and improvements of task outcomes, but given the 100 s of hours of training involved and complexity of the task itself, we were not able to record trial-by-trial learning effects. Therefore, the findings and insights of learning studied in the computational motor control literature—at the level of actual changes in motor coordination and control policies—has received little attention in previous real-world learning studies.

To tackle the complexity of the high dimensional full-body motor control and task-space (game objects) movement, we recorded continuously the full-body movement during the entire learning period (about an hour and a half) and measured balls movements automatically on the table. EEG activity was also recorded during the task via mobile brain imaging, but to focus here on the motor kinematics learning we chose to report the neural activity results elsewhere28. We quantify the trends in full-body movement and task performance separately during the entire learning process, and look for correlations between the changes in the body movement and the performance in the task.

We structured the results as follows: We ground our results in previous work on laboratory-tasks, to show that our unconstrained task and its task goal (directional error of the target ball relative to the pocket it is meant to go in) displays the well-known features of human motor learning, namely learning curves with characteristic double exponential shape. We then characterize full-body movement structure during the task, and how learning changes the kinematics of every of the measured 18 joints. In our analysis, we will alternate between taking a data-driven view that attempts to be task-ignorant to identify underlying simplicities indicative of biological mechanisms in the data, and a task-based view to interpret the data-driven findings by using task-domain knowledge. Finally, we compare across subjects to characterize how their performance, motor variability, and learning rates are linked.

## Results

30 right-handed volunteers, with little to no previous experience playing billiards, performed 300 repeated trials (6 sets of 50 trials each with short breaks in-between) where the cue ball and target ball were placed in the same locations, and subjects were asked to shoot the target ball towards the far-left corner pocket (Fig. 1A). During the entire learning process, we recorded the subjects' full-body movements with a ‘suit’ of inertial measurement units (IMUs; Fig. 1B), and the balls on the pool table were tracked with a high-speed camera to assess the outcome of each trial (Fig. 1C).

### Movement and learning in a real-world pool task

The ball tracking data showed learning curve for the decay in the directional error of the target ball (relative to the direction from its origin to the centre of the target pocket) over trials (Fig. 1D). This learning curve was best fit with a double exponential curve (Supplementary Fig. S1). The direction of the error in the initial trials was consistent across subjects as they tended to hit the centre of the target ball and shot it forward towards the centre of the table. For measuring success rates and intertrial variability we divided the trials into blocks of 25 trials (each experimental set of 50 trials was divided into two blocks to increase the resolution in time). To improve robustness and account for outliers, we fitted the errors in each block with a t-distribution and used the location and scale parameters (µ and σ) as the blocks’ centre and variability measures. The learning curve over blocks (Fig. 1E) emphasised the reduction in the inter-subject variability during learning (decreasing error bars). The success rate over blocks (percentage of successful trials in each block; Fig. 1F) showed similar learning to the directional error.

Learning was also evident in the intertrial variability in the shooting direction which decayed over learning (Fig. 1G). Since learning also occurred within a block (especially during the first block) and the variability might be driven by the learning gradient, we corrected for it by calculating intertrial variability over the residuals from a regression line fitted to the ball direction in each block (while the learning curve is exponential, within the small blocks of 25 trials it is almost linear). This corrected intertrial variability showed only minor reduction in the initial blocks, relative to the uncorrected variability, and showed the same decay pattern over the learning (Fig. 1H). The scatter in the trial-by-trial data (Fig. 1D) is due to both inter-trial variability (evident in Fig. 1G,H) and inter-subject variability (evident in the error bars in Fig. 1E).

Overall, the task performance data suggested that subjects reached peak performance by the fifth experimental set (blocks 9–10, trials 200–250) and were doing the same (or even slightly worse) on the last experimental set (blocks 11–12, trials 250–300). Thus, we refer to the last two experimental sets (blocks 9–12, trials 201–300) as the ‘learning plateau’, while being mindful that professional pool players train over months and years to improve or maintain their skills.

Kinematic data were recorded using a wearable motion tracking ‘suit’ of wireless IMUs, where individual wireless sensors (matchbox-sized) were attached via Velcro to elastic straps fixed around the subjects’ body without constraining movement. The full-body kinematics were analysed in terms of joint angles using 3 degrees of freedom for each joint following the International Society of Biomechanics (ISB) recommendations for Euler angle extractions of Z (flexion/extension), X (abduction/adduction), and Y (internal/external rotation). Note, this standard approach includes hinge joints of the body which have only 1 degree of freedom being recorded as 3 Euler angles. The full-body movements were analysed over the angular joint velocity profiles of all joints. The data allowed us to reconstruct the full-body pose at any given moment, which we checked for visual correctness on a subject-by-subject basis against video ground truth. However, we chose not to look at joint angle’s probability distributions, as those are more sensitive to potential drifts in the IMUs (and contain small changes not spottable by the human eye). We previously showed that joint angular velocity probability distributions are more subject invariant than joint angle distributions suggesting these are the reproducible features across subjects in natural behavior29. In the current study, this robustness is quite intuitive: all subjects stood in front of the same pool table and used the same cue stick, thus the subjects’ body size influenced their joint angles distributions (taller subjects with longer arms had to bend more towards the table and flex their elbow less than shorter subjects with shorter limbs) but not joints angular velocity probability distributions (Fig. 2).

In the first step of our data-driven analysis, we wanted to identify the key joints for the task: we analysed the angular velocity profiles of all joints, averaged across the initial block trials of all subjects, and found that most of the movement is done by the right arm, and specifically in the right shoulder (Fig. 2 inner circle). This is expected as all subjects were right-handed and used their right hand to hold the cue stick and make the shot. Taking domain understanding of the task into account, we can explain the shoulder movement by the naivety of the subjects, as pool billiards guidebooks30,31,32,33 emphasize that the shooting movement should be from the elbow down while the shoulder should be kept still. Correspondingly, the angular velocity profiles averaged across the initial block of the learning plateau (trials 201–225) showed similar distributions with an overall decrease in peak velocities relative to the initial trials but an increase in the peak angular velocity of ‘right elbow rotation’, which is the rotation between the upper arm and the forearm sensor and is equivalent to forearm supination (Fig. 2 outer circle).

The angular velocities of the shooting (right) arm were much larger than those of all other joints. For visibility, in Fig. 2 we used two different ranges on the y-axis for the joints of the shooting right arm (6 rad/s) and for all other body joints (2 rad/s). The high variance (relative to the mean) in some of the non-shooting arm joints that do move (such as the left elbow) could be caused primarily by variability across 1. trials and 2. subjects and specifically in its timing relative to the shot (e.g. While some subjects in some shots had a small left elbow movement just before the peak of the shot, others had it shortly after). Thus, aligning the traces with respect to the shot results in variability. In previous studies, the sensor noise was shown to be much smaller than both the inter- and intra-subject variability34, and to decay even more for slower movements35,36.

To quantify the overall change in the within-trial variability structure of the body over trials, we use the generalised variance, which is the determinant of the covariance matrix37 and is intuitively related to the multidimensional scatter of data points around their mean. We measured the generalised variance over the angular velocity profiles of all joints and found that it increased rapidly over the first ~ 40 trials and later decreased slowly (Fig. 3A). To understand what drives the generalised variance peak we plotted the variance–covariance matrixes of the first block, the second block (over the peak generalised variance), and ninth block (after learning plateaus) (Fig. 3B). It shows that the changes in the generalised variance were driven by an increase in the variance of all DoFs of the right shoulder and the negative covariance between the abduction/adduction and internal/external rotation of the right shoulder to the flexion/extension of the right shoulder and wrist. The internal/external rotation of the right elbow showed a continuous increase in its variance, which did not follow the trend of the generalised variance.

Next we set to study the complexity of the movement—as defined by the number of degrees of freedom used by the subject—since the use of multiple degrees of freedom in the movement is a hallmark of skill learning38. For that purpose, we applied Principal component analysis (PCA) across joints for the angular velocity profiles per trial for each subject and used the number of PCs that explain more than 1% of the variance to quantify the degrees of freedom in each trial movement. While in all trials of all subjects most of the variance can be explained by the first PC (Supplementary Fig. S2), there is a slow but consistent rise in the number of PCs that explain more than 1% of the variance in the joint angular velocity profiles (Fig. 3C). The manipulative complexity, suggested by Belić and Faisal39 as a way to quantify complexity for a given number of PCs on a fixed scale (C = 1 implies that all PCs contribute equally, and C = 0 if one PC explains all data variability), showed cleaner trajectory with the same trend (Fig. 3D). This suggests that over trials subjects use more degrees of freedom in their movement.

In the next step of our data-driven analysis, we wanted to identify signatures of learning joint-by-joint. For that, we defined a measure of task performance in single-joint space, which we named the Velocity Profile Error (VPE). VPE is the minimal correlation distances between the angular velocity profile of each joint in each trial to the angular velocity profiles of that joint in all successful trials (for more see “Methods” section). For all joints, VPE showed a clear pattern of decay over trials in an exponential learning curve (Fig. 4A). We fitted it with a single exponential learning curve (see fits time constants and goodness of fit in Supplementary Table S1). A proximal-to-distal gradient in the time constant of these learning curves was observed across the right arm, from the shoulder to the elbow and the wrist rotation (Supplementary Fig. S3). Intertrial variability in joint movement was measured over the VPEs in each block. Learning was also evident in the decay of the VPE intertrial variability during the learning over most joints across the body (Fig. 4B).

### Inter-subject differences in variability and learning

In the final step of our data-driven analysis, we addressed the individuality of the subjects and looked across subjects for correlations between their task performance, learning rate, and joint movements. Since this is an exploratory study, all statistical tests are reported with caution and thus are not presented in the text but only in the figure (Fig. 5)—where the readers can see the data points and make their judgment as for the true significance. The statistics are presented in Spearman rank correlation, to deal account for outliers and non-linear trends, and p-values are FDR corrected for multiple comparisons. The regression lines are presented only for visual account and include their 95% confidence intervals to address outlier biases. We found substantial differences between subjects in their initial errors, final errors, intertrial variability, and learning, which are overlooked in the group average results. One subject, who had low initial errors, showed no learning, i.e. did not reduce her error over trials from the first block (trials 1–25) to the learning plateau (trials 201–300). For all other subjects, the final errors were smaller than the initial errors (Fig. 5A). There was a significant correlation between the initial and the final errors, meaning subjects with higher initial errors tended to have higher final errors as well.

While over the learning most subjects decreased their intertrial variability in the outcome (ball direction; Figs. 1H, 5B) there was some tendency (though non-significant) for subjects who were initially more variable to be also more variable after learning (Fig. 5B). The intertrial variability of the joint angular velocity profiles, which also decreased over learning (Fig. 4B), showed a clearer and stronger correlation between the initial and the final intertrial variability (Fig. 5E, Supplementary Fig. S4). While this phenomenon was observed in various joints across the body, and dominant in the abduction across the spine joints, it was most dominant in the right shoulder abduction and rotation, the two joint angles that do most of the movement and carry most of its variance (Fig. 2).

Learning was defined as the difference between the initial error (over the first block: trials 1–25) and the final error (over the learning plateau: trials 201–300) normalised by the initial error. For the one subject who showed no learning (had bigger errors during the learning plateau than during the first block), we set learning to zero to avoid negative learning value. While there is a negative relation between learning and final error by definition, due to the normalization by the initial error (which was highly variant), there was no significant correlation between the learning and the final error (as subjects who started worse could have learned more but still not perform better after learning), but there was only a trend that more learning leads to smaller final errors (Fig. 5C). We speculated that the manipulative complexity (the degrees of freedom in the movement) might explain part of the inter-subject variability in learning rates. Presumably, subjects that learn more also show a higher increase in their manipulative complexity. Yet, we found no such relation. Both the initial (over the first block: trials 1–25) and the final (over the learning plateau: trials 201–300) manipulative complexity levels showed only a weak, non-significant, correlation to learning, and the increase in manipulative complexity showed no correlation to learning (Supplementary Fig. S5).

We then tested if higher levels of initial task-relevant motor variability (variability in the directional error of the target ball) in this complex real-world task could predict faster learning across individuals, as found in simple lab experiments40. We indeed found that individuals with higher intertrial variability in the directional error of the target ball over the first block showed more learning (Spearman rank correlation r = 0.64, p < 0.001; Fig. 5D). Importantly, this is the corrected intertrial variability (as in Fig. 1H) which is calculated over the residuals from a regression line fitted to the ball direction to correct for the learning that is happening within the block. As a control, we also tested for correlation with the initial variability in the target ball velocity—which is a task-irrelevant motor variability—and found no correlation (Spearman rank correlation r = 0.06, p = 0.77).

Since much of the learning is happening during the first 25 trials, calculating learning over blocks can lead to a ceiling effect. Therefore, to test the robustness of the correlation between learning and variability, we re-calculated the learning rate based on the 5, 7, 10, 15, and 20 initial trials (Supplementary Fig. S6). All choices of more than 5 trials showed a significant correlation between the initial variability and the learning. In the case of 5 trials, there were 2 outlier subjects with high variability and little learning who damaged the correlation. These 2 subjects had an initial ‘lucky shot’ which biased the learning calculation. Once including more trials, this effect was washed out.

Next, we tested the link between learning and initial variability over the joint angular velocity profiles of the right arm (Fig. 5F). We found that the only joint angle where the intertrial variability showed a significant correlation to learning was the right elbow rotation (Spearman rank correlation r = 0.47, p = 0.0086), which is the forearm supination. We further tested the link over the full-body kinematics (Supplementary Fig. S7) and found no other joint that showed this correlation. Thus, while learning leads to an overall reduction in movement variability, only initial variability in specific, task-relevant, dimensions can facilitate/predict learning.

## Discussion

In this paper, we introduce a new paradigm for studying naturalistic motor learning during whole-body movement in a complex real-world motor skill task. Our results present new insights into motor learning in the real-world. While the learning curves in this in-the-wild paradigm are within the same range of those reported in reductionistic motor adaptation tasks2,41 we find that this learning is taking place not only in the task-relevant joints but across the entire body. Also, we found that task-relevant initial variability in the ball direction (movement outcome) can predict learning, like in laboratory-tasks40, and so can the initial variability in the right forearm supination which is the task-relevant joint angle variability.

While pushing towards real-world neuroscience, we started here with a relatively constrained version of the real-world task, asking subjects to perform repeated trials of the same pool shot. This was to enable analysis using well-developed methods of laboratory-tasks. Nonetheless, it is a major step in the direction of a naturalistic study. First, we allow full-body unconstrained movement. Second, we do not use any artificial go cue and allow self-paced movement and as many preparatory movements as the subject needs for each shoot. Third, subjects receive natural somatosensory feedback. And last, we do not perturb the feedback to induce learning.

### Fundamentals of real-world motor learning

Across all subjects, we found that motor learning is a holistic process—the body is affected as a whole by learning the task. This was evident in the decrease in the VPE and the intertrial variability over learning (Fig. 4A,B). This result should not come as a surprise considering decades of research in sport science showing this relationship. For example, baseball pitcher's torso, pelvis, and leg movements are directly associated with ball velocity42,43,44. Recently it was also demonstrated with full-body motion capture in a ball throwing task45. And yet, unlike baseball pitches, basketball throws, or any unconstrained overarm throw, where the whole body is moving, in a pool shot the shooting arm is doing most of the movement and there is very little body movement. Thus, the whole-body learning is not trivial and suggestive that even in arm movement laboratory-tasks there is probably a whole-body learning aspect that is overlooked.

We also found a proximal-to-distal gradient in the learning rates over the right arm joints (Fig. 4A, Supplementary Fig. S3). This is especially interesting in light of the well-known phenomenon of proximal-to-distal sequence in limb movements in sports science46 and rehabilitation47. While there are records of proximal-to-distal sequence at multiple time scales48, our results are the first to suggest that this gradient also occur over repetitions as part of the learning process.

### Variability and learning

Intertrial variability is a fundamental characteristic of human movements and its underling neural activity49. It was recently reported that individuals exhibit distinct magnitudes of movement variability, which are consistent across movements and effectors, suggesting individual traits in movement variability50. Our results show that subjects who were initially more variable tended to be also more variable after learning in many joints across the body (Fig. 5E, Supplementary Fig. S4) and specifically in those of right shoulder that carry most of the variance in the movement. This result is in-line with the notion that there is an individual trait in movement variability.

A decay in the intertrial variability is considered a feature of skill learning20,22,51,52,53,54,55,56, but not of error-based adaptation. While lower variability associated with lower means is a well-known statistical phenomenon, it is not evident in all learning tasks. In current billiards paradigm for example, in our VR version of the task, we report no decay in intertrial variability over learning57, which may suggest that the learning there is more through error-based adaptation. Here, across subjects, we see a decay in intertrial variability which suggests skill learning. Importantly, the novel finding here is that we see this decay in intertrial variability not only in the trial outcome (the direction of the target ball) but also in the body movement across the different joints (Fig. 4B). Therefore, our data suggest that the decay in the intertrial variability during skill learning is also a holistic process evident across the whole body.

While there is a clear group effect of intertrial variability decay, we also highlight inter-subject differences in the variability decay (Fig. 5B). Namely, while for some subjects the inter-trial variability after learning is much lower than in early learning, for others there is no real difference between the two and for some even a slight increase. This is discussed in depth in the follow-up paper where we analyse the neural activity in this task and discuss the learning mechanisms involved28. There we found two subgroups of subjects based on their neural activity while performing the task. We also found significant differences in the variability decay patterns of the two subgroups but not in their absolute error decay. The neural and behavioural differences led us to speculate that two different learning mechanisms are contributing to the task: error-based adaptation and reward-based reinforcement learning, where the predominant learning mechanism is different between subjects28. This might be transformative for rehabilitation.

It is important to note that this refers specifically to the forearm supination around the elbow and not around the wrist. This is due to the nature of the data collected with the sensors suit where the joint angles are recorded with 3 degrees of freedom based on the angles between the sensors from both sides of each joint. Thus, hinge joints of the body which have only one anatomical degree of freedom have been recorded as 3 Euler angles. Specifically, the elbow rotation is the rotation between the upper arm sensor and the forearm sensor and is equivalent to forearm supination around the elbow. The wrist rotation is the rotation between the forearm sensor and the hand sensor and is equivalent to hand supination. Similarly, the elbow abduction/adduction should be negligible as it is a hinge joint66. Yet there is an abduction/adduction movement around our elbow (especially when the elbow flexion is around 90 degrees as in a pool shot) which is captured by the sensors and recorded as elbow abduction/adduction.

We note that the above results are correlational and cannot address the question of causality: e.g. can higher initial variability cause faster learning? While the study of real-world tasks takes us closer to understanding real-world motor learning, it is lacking the key advantage of laboratory tasks of highly controlled manipulations of known variables, to isolate specific movement/learning components. We therefore developed an embodied virtual reality (VR) version of our pool task57, As the VR-based approach enables visual feedback manipulations while subjects are executing the same real-world task. However, establishing a real-world task as done here is an essential precondition for understanding motor learning and the motor system and the way it has evolved and developed. For example, we have shown that measuring and incorporating unconstrained real-world kinematics can boost the decoding performance of smart neuroprosthetics67.

### Conclusions

In this study, we demonstrate the feasibility and importance of studying human neuroscience in-the-wild, and specifically in naturalistic real-world skill tasks. While finding similarities in learning structure between our real-world paradigm and lab-based motor learning studies, we highlight crucial differences, namely, real-world motor learning is a holistic full-body process. Looking at the motor behaviour over learning across the entire body enabled us to explore the relationship between variability and learning and define task-relevant variability that can facilitate learning.

## Methods

### Ethics statement

All experimental procedures were approved by Imperial College Research Ethics Committee and performed in accordance with the declaration of Helsinki. All subjects gave informed consent prior to participating in the study. Written informed consent was obtained from the individual in Fig. 1A for the publication of any potentially identifiable images included in this article.

### Experimental setup and design

30 right-handed healthy human volunteers with normal or corrected-to-normal visual acuity (12 women and 18 men, aged 24 ± 3) participated in the study. The recruitment criteria were that they played pool/billiards/snooker for leisure fewer than 5 times in their life, never in the recent 6 months, and had never received any pool game instructions. All volunteers gave informed consent before participating in the study, and all experimental procedures were approved by the Imperial College Research Ethics Committee and performed in accordance with the declaration of Helsinki. The volunteers stood in front of a 5 ft pool table (Riley Leisure, Bristol, UK) with 1 7/8″ (48 mm diameter) pool balls. Volunteers performed 300 repeated trials where the cue ball (white) and the target ball (red) were placed in the same locations. We asked volunteers to shoot the target ball towards the pocket of the far-left corner (Fig. 1A). Trials were split into 6 sets of 50 trials with a short break in-between to allow the subjects to rest a bit and reduce potential fatigue. Each experimental set (of 50 trials) took 8 to 12 min. For the data analysis, we further split each set into two blocks of 25 trials each, resulting in 12 blocks. During the entire learning process, we recorded the subjects’ full-body movements with a motion-tracking ‘suit’ of 17 wireless inertial measurement units (IMUs; Fig. 1B). The balls on the pool table were tracked with a high-speed camera (Dalsa Genie Nano, Teledyne DALSA, Waterloo, Ontario) to assess the subjects’ success in the game and to analyze the changes throughout learning, not only in the body movement but also in its outcome—the ball movement (Fig. 1C).

### Balls tracking

The balls movement on the pool table were tracked with a computer vision system mounted from the ceiling. The computer vision camera was a Genie Nano C1280 Color Camera (Teledyne Dalsa, Waterloo, Canada), colour images were recorded with a resolution of 752 × 444 pixels and a frequency of 200 Hz. This Ethernet-based camera was controlled via the Common Vision Blox Management Console (Stemmer Imaging, Puchheim, Germany) and image videos recorded with our custom software written in C++ based on a template provided by Stemmer Imaging. Our software captured the high-performance event timer, the camera frames and converted the images from the camera’s proprietary CVB format to the open-source OpenCV (https://opencv.org/) image format for further processing in OpenCV. The video frames were stored as an uncompressed AVI file to preserve the mapping between pixel changes and timings and the computer’s real-time clock time-stamps were recorded to a text file. Each trial was subject-paced, so the experimenter observed the subject and hit the spacebar key as an additional trigger event to the time-stamps text file. This timing data was later used to assist segmentation of the continuous data stream into trials. The positions of the two pool balls (white cue ball and red target ball) were calculated from the video recordings offline using custom software written in C++ using OpenCV. Then, with custom software written in MATLAB (R2017a, The MathWorks, Inc., MA, USA), we segmented the ball tracking data and extracted the trajectory of the balls in each trial. For each trial, a 20 × 20 pixels (approx 40 × 40 mm) bounding box was set around the centre of the 48 mm diameter cue ball. The time the centre of the ball left the bounding box was recorded as the beginning of the cue ball movement. The pixel resolution and frame rate were thus sufficient to detect movement onset, acceleration and deceleration of the pool balls. The target (red) ball initial position and its position in the point of its peak velocity were used to calculate the ball movement angle (relative to a perfectly straight line between the white cue ball and the red target ball). We subtracted this angle from the centre of the pocket angle (the angle the target ball initial position and the centre of the pocket relative to the same straight line between the balls) to calculate the directional error for each shot.

### Full-body motion tracking

Kinematic data were recorded at 60 Hz using a wearable motion tracking ‘suit’ of 17 wireless IMUs (Xsens MVN Awinda, Xsens Technologies BV, Enschede, The Netherlands). Data acquisition was controlled via a graphical interface (MVN Analyze, Xsens Technologies BV, Enschede, The Netherlands). Xsens MVN uses a biomechanical model and proprietary algorithms to estimate 3D joint kinematics68,69. The Xsens sensors shows high accuracy35,70, and the Xsens MVN system was used and validated in tracking real-world behaviour in many sports including football34, horse riding71, ski72 and snowboarding73. The Xsens 3D joint kinematics were exported as XML files and analysed using custom software written in MATLAB (R2017a, The MathWorks, Inc., MA, USA). The Xsens full-body kinematics were extracted in joint angles in 3 degrees of freedom for each joint that followed the International Society of Biomechanics (ISB) recommendations for Euler angle extractions of Z (flexion/extension), X (abduction/adduction) Y (internal/external rotation). This standard approach includes hinge joints of the body which have only 1 degree of freedom being recorded as 3 Euler angles.

### Angular velocity profile analysis

From the Xsens 3D joint angles we extracted the angular velocity profiles of all joints in all trials. We defined the peak of the trial as the peak of the average absolute angular velocity across the DoFs of the right shoulder and the right elbow. We aligned all trials around the peak angular velocity of the trial and cropped a window of 1 s around the peak for the analysis of joint angular velocity profiles during the shot and its follow-through. This time window covered the entire movement of the pool shoot while eliminating the preparatory movement and the mock shoots (Fig. 2).

### Task performance and learning measures

The task performance was measured by the trial error which was defined as an absolute angular difference between the target ball movement vector direction and the desired direction to land the target ball in the centre of the pocket. The decay of error over trials is the clearest signature of learning in the task. For measuring success rates and intertrial variability we divided the trials into blocks of 25 trials by dividing each experimental set of 50 trials to two blocks. This was done to increase the resolution in time from calculating those on the full sets. Success rate in each block was defined by the ratio of successful trial (in which the ball fell into the pocket). To improve robustness and account for outliers, we fitted the errors in each block with a t-distribution and used the location and scale parameters (µ and σ) as the blocks’ centre and variability measures. To correct for learning within a block, we also calculated a corrected intertrial variability, which was the intertrial variability over the residuals from a regression line fitted to the ball direction in each block. This correction for the learning trend within a block does not change the variability measure by much (Fig. 1G,H). This is since our variability measure is not the standard deviation, but the scale parameter of a t-distribution fitted to the errors. When correcting the change in the distribution fitted was mostly an increase in the degrees of freedom and a not decrease in the scale (i.e. the early trials which were much higher than the mean and the late trials which were much lower become closer to the mean and therefore the distribution is more normal as it loses the heavy tails). This is highlighting the robustness of the scale measure for variability.

To quantify the within-trial variability structure of the body movement, we use the generalised variance, which is the determinant of the covariance matrix37 and is intuitively related to the multidimensional scatter of data points around their mean. We measured the generalised variance over the velocity profiles of all joints in each trial to see how it changes with learning. To study the complexity of the body movement which was defined by the number of degrees of freedom used by the subject we applied principal component analysis (PCA) across joints for the velocity profiles per trial for each subject and used the number of PCs that explain more than 1% of the variance to quantify the degrees of freedom in each trial movement. We also calculated the manipulative complexity which was suggested by Belić and Faisal39 as a way to quantify complexity for a given number of PCs on a fixed scale (C = 1 implies that all PCs contribute equally, and C = 0 if one PC explains all data variability).

### Statistical analysis

Trial by trial learning curves of single-trial performance measure (directional error of the target ball relative to the centre of the pocket) were fitted with a single, double, and triple exponential learning curve using Matlab fit function. As in most motor learning datasets, the double exponential curve showed the best fit (Supplementary Fig. S1).

As a measure of task performance in body space, correlation distances (one minus Pearson correlation coefficient) were calculated between the angular velocity profile of each joint in each trial to the angular velocity profiles of that joint in all successful trials. The minimum over these correlation distances produced a single measure of Velocity Profile Error (VPE) for each joint in each trial.

$${VPE}_{i}=\underset{s}{\mathrm{min}}\left(corrDist\left({velProf}_{i},{velProf}_{s}\right)\right)$$

Thus, VPE in trial $$i$$ was the minimal correlation distances between the angular velocity profile in trial $$i$$ ($${velProf}_{i}$$) and the angular velocity profiles in successful trials $$s$$ ($${velProf}_{s}$$). While there are multiple combinations of body variables that can all lead to successful task performance, this measure looks for the distance from the nearest successful solution used by the subjects and thus provides a metric that accounts for the redundancy in the body.

All correlations between error, variability, and learning are Spearman's rank correlation coefficients to be robust to outliers and non-linear trends, and their p-values are FDR corrected for multiple comparisons. Regression lines are based on linear regression fits (in logarithmic scale for VPE variability) and are presented with 95% confidence intervals.

## References

1. Shadmehr, R. & Mussa-Ivaldi, F. A. Adaptive representation of dynamics during learning of a motor task. J. Neurosci. 14, 3208–3224 (1994).

2. Smith, M. A., Ghazizadeh, A. & Shadmehr, R. Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol. 4, e179 (2006).

3. Diedrichsen, J., Hashambhoy, Y., Rane, T. & Shadmehr, R. Neural correlates of reach errors. J. Neurosci. 25, 9919–9931 (2005).

4. Howard, I. S., Wolpert, D. M. & Franklin, D. W. The value of the follow-through derives from motor learning depending on future actions. Curr. Biol. 25, 397–401 (2015).

5. Krakauer, J. W., Pine, Z., Ghilardi, M. & Ghez, C. Learning of visuomotor transformations for vectorial planning of reaching trajectories. J. Neurosci. 20, 8916–8924 (2000).

6. Mazzoni, P. & Krakauer, J. An implicit plan overrides an explicit strategy during visuomotor adaptation. J. Neurosci. 26, 3642–3645 (2006).

7. Taylor, J. A., Krakauer, J. W. & Ivry, R. B. Explicit and implicit contributions to learning in a sensorimotor adaptation task. J. Neurosci. 34, 3023–3032 (2014).

8. Haar, S., Donchin, O. & Dinstein, I. Dissociating visual and motor directional selectivity using visuomotor adaptation. J. Neurosci. 35, 6813–6821 (2015).

9. Bromberg, Z., Donchin, O. & Haar, S. Eye movements during visuomotor adaptation represent only part of the explicit learning. eNeuro 6, 1–12 (2019).

10. Reis, J. et al. Noninvasive cortical stimulation enhances motor skill acquisition over multiple days through an effect on consolidation. Proc. Natl. Acad. Sci. U.S.A. 106, 1590–1595 (2009).

11. Ma, L., Narayana, S., Robin, D. A., Fox, P. T. & Xiong, J. Changes occur in resting state network of motor system during 4weeks of motor skill learning. Neuroimage 58, 226–233 (2011).

12. Clerget, E., Poncin, W., Fadiga, L. & Olivier, E. Role of Broca’s area in implicit motor skill learning: Evidence from continuous theta-burst magnetic stimulation. J. Cogn. Neurosci. 24, 80–92 (2012).

13. Yokoi, A., Arbuckle, S. A. & Diedrichsen, J. The role of human primary motor cortex in the production of skilled finger sequences. J. Neurosci. 38, 1430–1442 (2018).

14. Ingram, J. N. & Wolpert, D. M. Naturalistic approaches to sensorimotor control. Prog. Brain Res. 191, 3–29 (2011).

15. Wolpert, D. M., Diedrichsen, J. & Flanagan, J. R. Principles of sensorimotor learning. Nat. Rev. Neurosci. 12, 739–751 (2011).

16. Faisal, A., Stout, D., Apel, J. & Bradley, B. The manipulative complexity of lower paleolithic stone toolmaking. PLoS ONE 5, e13718 (2010).

17. Hecht, E. E. et al. Acquisition of Paleolithic toolmaking abilities involves structural remodeling to inferior frontoparietal regions. Brain Struct. Funct. 220, 2315–2331 (2014).

18. Laughlin, S. A simple coding procedure enhances a neuron’s information capacity. Z. fur Nat. Sect. C J. Biosci. 36, 910–912 (1981).

19. Simoncelli, E. P. & Olshausen, B. A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001).

20. Cohen, R. G. & Sternad, D. Variability in motor learning: Relocating, channeling and reducing noise. Exp. Brain Res. 193, 69–83 (2009).

21. Abe, M. O. & Sternad, D. Directionality in distribution and temporal structure of variability in skill acquisition. Front. Hum. Neurosci. 7, 225 (2013).

22. Shmuelof, L., Krakauer, J. W. & Mazzoni, P. How is a motor skill learned? Change and invariance at the levels of task success and trajectory control. J. Neurophysiol. 108, 578–594 (2012).

23. Yang, J. F. & Scholz, J. P. Learning a throwing task is associated with differential changes in the use of motor abundance. Exp. Brain Res. 163, 137–158 (2005).

24. Scholz, J., Klein, M. C., Behrens, T. E. & Johansen-Berg, H. Training induces changes in white-matter architecture. Nat. Neurosci. 12, 1370–1371 (2009).

25. Sampaio-Baptista, C. et al. Gray matter volume is associated with rate of subsequent skill learning after a long term training intervention. Neuroimage 96, 158–166 (2014).

26. Sampaio-Baptista, C. et al. Changes in functional connectivity and GABA levels with long-term motor learning. Neuroimage 106, 15–20 (2015).

27. Ono, Y. et al. Motor learning and modulation of prefrontal cortex: an fNIRS assessment. J. Neural Eng. 12, 066004 (2015).

28. Haar, S. & Faisal, A. A. Brain activity reveals multiple motor-learning mechanisms in a real-world task. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2020.00354 (2020).

29. Thomik, A. A. C. On the Structure of Natural Human Movement (Imperial College London, London, 2016).

30. Phelan, M. The Game of Billiards (D. Appleton and Company, New York, 1859).

31. De Vere, A. Billiards Made Easy, by ‘Winning Hazard’ (Houlston and Sons, London, 1873).

32. Mizerak, S. Pocket Billiards Tips and Trick Shots (McGraw-Hill, New York, 1982).

33. Leider, N. Pool & Billiards for Dummies (Wiley, Hoboken, 2010).

34. Blair, S., Duthie, G., Robertson, S., Hopkins, W. & Ball, K. Concurrent validation of an inertial measurement system to quantify kicking biomechanics in four football codes. J. Biomech. 73, 24–32 (2018).

35. Lebel, K., Boissy, P., Hamel, M. & Duval, C. Inertial measures of motion for clinical biomechanics: comparative assessment of accuracy under controlled conditions—Effect of velocity. PLoS ONE 8, e79945 (2013).

36. Lebel, K., Boissy, P., Nguyen, H. & Duval, C. Inertial measurement systems for segments and joints kinematics assessment: Towards an understanding of the variations in sensors accuracy. Biomed. Eng. 16, 56 (2017).

37. Wilks, S. S. Certain generalizations in the analysis of variance. Biometrika 24, 471 (1932).

38. Bernstein, N. The Co-ordination and Regulation of Movements (Pergamon Press, Oxford, 1967).

39. Belić, J. J. & Faisal, A. A. Decoding of human hand actions to handle missing limbs in neuroprosthetics. Front. Comput. Neurosci. 9, 27 (2015).

40. Wu, H. G., Miyamoto, Y. R., Gonzales-Castro, L. N., Ölveczky, B. C. & Smith, M. A. Temporal structure of motor variability is dynamically regulated and predicts motor learning ability. Nat. Neurosci. 17, 312–321 (2014).

41. McDougle, S. D., Bond, K. M. & Taylor, J. A. Explicit and implicit processes constitute the fast and slow processes of sensorimotor learning. J. Neurosci. 35, 9568–9579 (2015).

42. Kageyama, M., Sugiyama, T., Takai, Y., Kanehisa, H. & Maeda, A. Kinematic and kinetic profiles of trunk and lower limbs during baseball pitching in collegiate pitchers. J. Sports Sci. Med. 13, 742–750 (2014).

43. Oliver, G. D. & Keeley, D. W. Pelvis and torso kinematics and their relationship to shoulder kinematics in high-school baseball pitchers. J. Strength Cond. Res. 24, 3241–3246 (2010).

44. Stodden, D. F., Langendorfer, S. J., Fleisig, G. S. & Andrews, J. R. Kinematic constraints associated with the acquisition of overarm throwing part I. Res. Q. Exerc. Sport 77, 417–427 (2006).

45. Maselli, A. et al. Where are you throwing the ball? I better watch your body, not just your arm!. Front. Hum. Neurosci. 11, 505 (2017).

46. Herring, R. M. & Chapman, A. E. Effects of changes in segmental values and timing of both torque and torque reversal in simulated throws. J. Biomech. 25, 1173–1184 (1992).

47. Twitchell, T. E. The restoration of motor function following hemiplegia in man. Brain 74, 443–480 (1951).

48. Serrien, B. & Baeyens, J. P. The proximal-to-distal sequence in upper-limb motions on multiple levels and time scales. Hum. Mov. Sci. https://doi.org/10.1016/j.humov.2017.08.009 (2017).

49. Faisal, A., Selen, L. P. J. & Wolpert, D. M. Noise in the nervous system. Nat. Rev. Neurosci. 9, 292–303 (2008).

50. Haar, S., Donchin, O. & Dinstein, I. Individual movement variability magnitudes are explained by cortical neural variability. J. Neurosci. 37, 9076–9085 (2017).

51. Deutsch, K. M. & Newell, K. M. Changes in the structure of children’s isometric force variability with practice. J. Exp. Child Psychol. 88, 319–333 (2004).

52. Müller, H. & Sternad, D. Decomposition of variability in the execution of goal-oriented tasks: Three components of skill improvement. J. Exp. Psychol. Hum. Percept. Perform. 30, 212–233 (2004).

53. Guo, C. C. & Raymond, J. L. Motor learning reduces eye movement variability through reweighting of sensory inputs. J. Neurosci. 30, 16241–16248 (2010).

54. Huber, M. E., Brown, A. J. & Sternad, D. Girls can play ball: Stereotype threat reduces variability in a motor skill. Acta Psychol. (Amst) 169, 79–87 (2016).

55. Sternad, D. It’s not (only) the mean that matters: Variability, noise and exploration in skill learning. Curr. Opin. Behav. Sci. 20, 183–195 (2018).

56. Krakauer, J. W., Hadjiosif, A. M., Xu, J., Wong, A. L. & Haith, A. M. Motor learning. Compr. Physiol. 9, 613–663 (2019).

57. Haar, S., Sundar, G. & Faisal, A. A. Embodied virtual reality for the study of real-world motor learning. BioRxiv https://doi.org/10.1101/2020.03.19.998476 (2020).

58. Herzfeld, D. J. & Shadmehr, R. Motor variability is not noise, but grist for the learning mill. Nat. Neurosci. 17, 149–150 (2014).

59. Teo, J. T. H., Swayne, O. B. C., Cheeran, B., Greenwood, R. J. & Rothwell, J. C. Human theta burst stimulation enhances subsequent motor learning and increases performance variability. Cereb. Cortex 21, 1627–1638 (2011).

60. Braun, D. A., Aertsen, A., Wolpert, D. M. & Mehring, C. Motor task variation induces structural learning. Curr. Biol. 19, 352–357 (2009).

61. Wilson, C., Simpson, S. E., van Emmerik, R. E. A. & Hamill, J. Coordination variability and skill development in expert triple jumpers. Sports Biomech. 7, 2–9 (2008).

62. Dhawale, A. K., Smith, M. A. & Ölveczky, B. P. The role of variability in motor learning. Annu. Rev. Neurosci. 40, 479–498 (2017).

63. Singh, P., Jana, S., Ghosal, A. & Murthy, A. Exploration of joint redundancy but not task space variability facilitates supervised motor learning. Proc. Natl. Acad. Sci. 113, 14414–14419 (2016).

64. He, K. et al. The statistical determinants of the speed of motor learning. PLoS Comput. Biol. 12, e1005023 (2016).

65. van der Vliet, R. et al. Individual differences in motor noise and adaptation rate are optimally related. eneuro https://doi.org/10.1523/ENEURO.0170-18.2018 (2018).

66. Wu, G. et al. ISB recommendation on definitions of joint coordinate systems of various joints for the reporting of human joint motion—Part II: Shoulder, elbow, wrist and hand. J. Biomech. 38, 981–992 (2005).

67. Xiloyannis, M., Gavriel, C., Thomik, A. A. C. & Faisal, A. A. Gaussian process autoregression for simultaneous proportional multi-modal prosthetic control with natural hand kinematics. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 1785–1801 (2017).

68. Roetenberg, D., Luinge, H. & Slycke, P. Xsens MVN: Full 6DOF human motion tracking using miniature inertial sensors. Xsens. Motion Technol. BV 8, 1–7 (2009).

69. Schepers, M., Giuberti, M. & Bellusci, G. Xsens MVN: Consistent tracking of human motion using inertial sensing. Xsens. Technol. https://doi.org/10.13140/RG.2.2.22099.07205 (2018).

70. Auepanwiriyakul, C., Waibel, S., Songa, J., Bentley, P. & Faisal, A. A. Accuracy and acceptability of wearable motion tracking smartwatches for inpatient monitoring. medRxiv https://doi.org/10.1101/2020.07.24.20160663 (2020).

71. Gandy, E. A., Bondi, A., Hogg, R. & Pigott, T. M. C. A preliminary investigation of the use of inertial sensing technology for the measurement of hip rotation asymmetry in horse riders. Sport Technol. 7, 79–88 (2014).

72. Lee, S. K., Kim, K., Kim, Y. H. & Lee, S. S. Motion anlaysis in lower extremity joints during ski carving turns using wearble inertial sensors and plantar pressure sensors. in 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, 2017-January, 695–698 (Institute of Electrical and Electronics Engineers Inc., 2017).

73. Krüger, A. & Edelmann-Nusser, J. Biomechanical analysis in freestyle snowboarding: Application of a full-body inertial measurement system and a bilateral insole measurement system. Leis. Loisir 2, 17–23 (2009).

## Acknowledgements

We thank Marlene Gonzalez for her contribution to the data collection and our participants for taking part in the study. We acknowledge the technical and computational support of Alex Harston and Chaiyawan Auepanwiriyakul. We thank Alex Harston for helpful comments on the manuscript. The study was enabled by financial support to a Royal Society-Kohn International Fellowship (NF170650; SH & AAF) and by eNHANCE (https://www.enhance-motion.eu) under the European Union’s Horizon 2020 research and innovation program grant agreement No. 644000 (SH & AAF).

## Author information

Authors

### Contributions

S.H. and A.A.F. conceived and designed the study; S.H., C.V.A. and A.A.F. developed the experimental setup; S.H. and C.V.A. acquired the data; S.H. and A.A.F. analysed and interpreted the data; S.H. drafted the paper; SH and A.A.F. revised the paper.

### Corresponding authors

Correspondence to Shlomi Haar or A. Aldo Faisal.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

### Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Haar, S., van Assel, C.M. & Faisal, A.A. Motor learning in real-world pool billiards. Sci Rep 10, 20046 (2020). https://doi.org/10.1038/s41598-020-76805-9

• Accepted:

• Published:

• DOI: https://doi.org/10.1038/s41598-020-76805-9

• ### Development of new 9-ball test protocols for assessing expertise in cue sports

• Jing Wen Pan
• John Komar
• Pui Wah Kong

BMC Sports Science, Medicine and Rehabilitation (2021)

• ### Playing the piano with a robotic third thumb: assessing constraints of human augmentation

• Ali Shafti
• Shlomi Haar
• A. Aldo Faisal

Scientific Reports (2021)