Introduction

Motor learning is a key feature of our development and daily lives, from a baby learning to crawl, an adult learning a new sport, or a patient undergoing rehabilitation after a stroke. The process of learning a real-world motor skill is usually long and complex, and difficult to quantify as tasks are naturally unconstrained and highly variable. Most of the motor learning literature focuses on relatively simple tasks, performed in a laboratory setup or even within an MRI scanner, such as force-field adaptations1,2,3,4, visuomotor perturbations5,6,7,8,9, and sequence-learning of finger tapping or pinching tasks10,11,12,13. These laboratory-based learning tasks enable us to isolate specific features of motor learning and dissect them individually, thus provide elegant experiment designs to verify the experimenters’ hypothesis. As a result, motor learning in the real world is rarely studied. While laboratory-tasks play an important role in our understanding of sensorimotor control and learning, they address a very restricted range of behaviours that do not capture the full complexity of real-world motor control and may overlook fundamental principles of motor control and learning in real-life14,15.

Neurobehavioral mechanisms are subject to evolutionary selection pressures and survive only if they are relevant in natural tasks. Thus, studying operation in natural contexts allows us to evaluate mechanisms the nervous system has been designed for16,17. Example in sensory neuroscience the use of natural sensory stimuli has led to a revolution of our mechanistic understanding of perception18,19. Over the past decade, there were a few notable efforts to study motor learning in unconstrained tasks. One line of research devised more complex tasks for skill learning20,21,22 (e.g. skittles) which were implemented as computer-based gamified tasks that emulate real-world tasks. Others moved away from the computer screen but were still highly constrained; e.g. throwing a frisbee while the subject’s trunk is strapped to the chair to prevent trunk movement23. Another line of inquiry used free-behaviour in real-world tasks such as tool-making or juggling17,24,25,26,27. In these studies, we and others focused on anatomical and functional MRI measurable changes following learning. In a 3-year long complex tool-making apprenticeship experiment17 we were able to quantify changes in motor control precision and improvements of task outcomes, but given the 100 s of hours of training involved and complexity of the task itself, we were not able to record trial-by-trial learning effects. Therefore, the findings and insights of learning studied in the computational motor control literature—at the level of actual changes in motor coordination and control policies—has received little attention in previous real-world learning studies.

One of the main challenges of understanding real-world behaviour and specifically motor learning is to identify underlying simplicities in a highly variable stream of movements that are not well constrained. We take a data-driven approach to analyse a real-world task and thereby illustrate a process by which one can investigate a real-world motor learning task in a principled manner. We aim to inject as few assumptions a priori as possible about task-relevant joints or mechanisms of learning, instead we aim to use plausible methods to reveal these to us. The paradigm we choose is the game of pool table billiards. Pool is a real-world task that involves many different sub-tasks (precision, alignment, ballistic movements, high-level sequential planning and sequencing of shots and ball positions) which requires advanced skills—hence it is a highly competitive sport. While our ultimate goal is to move to a data analysis framework for real-world motor learning in arbitrary tasks, we find that the game of pool offers a useful intermediate goal that frames the behavioural data in a way that makes our approach amenable to be understood in the currently dominant framework of laboratory-tasks and measures. Pool billiards has natural spatial constraints (the area in and around the pool table), divisibility of behaviour into trials (shots of the ball) allowing us to visualize results in the same framework of lab-based motor learning tasks and a clear outcome (sink the ball into the pocket). Subjects had to do a pool shot to put the ball in the pocket using unconstrained full-body, self-paced movement, with as many preparatory movements as the subject needs for each shoot, the only constraints arose from the placement of the white cue ball which the subjects shoot with the cue stick and the red target ball (that needs to go into the pocket). We implemented this as a real-world experiment, effectively only adding sensors to the subject and the pool table, i.e. subjects use the normal pool cue, balls, and pool table they would in a leisure setting and thus carry out natural motor commands, receive the natural somatosensory feedback and experience the same satisfaction (rewards) when they put the ball in the pocket as this is a real-world task. Crucially, the skill of subjects in putting the ball into the pocket is learnable in the time course of 1–2 h, allowing us to record and analyse the experiments as one session.

To tackle the complexity of the high dimensional full-body motor control and task-space (game objects) movement, we recorded continuously the full-body movement during the entire learning period (about an hour and a half) and measured balls movements automatically on the table. EEG activity was also recorded during the task via mobile brain imaging, but to focus here on the motor kinematics learning we chose to report the neural activity results elsewhere28. We quantify the trends in full-body movement and task performance separately during the entire learning process, and look for correlations between the changes in the body movement and the performance in the task.

We structured the results as follows: We ground our results in previous work on laboratory-tasks, to show that our unconstrained task and its task goal (directional error of the target ball relative to the pocket it is meant to go in) displays the well-known features of human motor learning, namely learning curves with characteristic double exponential shape. We then characterize full-body movement structure during the task, and how learning changes the kinematics of every of the measured 18 joints. In our analysis, we will alternate between taking a data-driven view that attempts to be task-ignorant to identify underlying simplicities indicative of biological mechanisms in the data, and a task-based view to interpret the data-driven findings by using task-domain knowledge. Finally, we compare across subjects to characterize how their performance, motor variability, and learning rates are linked.

Results

30 right-handed volunteers, with little to no previous experience playing billiards, performed 300 repeated trials (6 sets of 50 trials each with short breaks in-between) where the cue ball and target ball were placed in the same locations, and subjects were asked to shoot the target ball towards the far-left corner pocket (Fig. 1A). During the entire learning process, we recorded the subjects' full-body movements with a ‘suit’ of inertial measurement units (IMUs; Fig. 1B), and the balls on the pool table were tracked with a high-speed camera to assess the outcome of each trial (Fig. 1C).

Figure 1
figure 1

Experimental setup and task performance. (A) 30 right-handed healthy subjects performed 300 repeated trials of billiards shoots of the target (red) ball towards the far-left corner. (B) Full body movement was recorded with a ‘suit’ of 17 wireless IMUs (Xsens MVN Awinda). (C) The pool balls were tracked with a high-speed camera. Dashed lines show the trajectories of the cue (white) and target (red) balls over 50 trials of an example subject. (D) The trial-by-trial directional error of the target-ball (relative to the direction from its origin to the centre of the target pocket), averaged across all subjects, with a double-exponential fit (red curve). The time constant of the fast and slow components were 6 and 129 trials, respectively. Grey lines mark the range of successful trials (less than 3 degrees form the centre of the pocket). (E) The mean absolute directional error of the target-ball. (F) The success rates. (G) Directional variability and (H) directional variability corrected for learning (see text). (E–H) presented over blocks of 25 trials, averaged across all subjects, error bars represent SEM.

Full size image

Movement and learning in a real-world pool task

The ball tracking data showed learning curve for the decay in the directional error of the target ball (relative to the direction from its origin to the centre of the target pocket) over trials (Fig. 1D). This learning curve was best fit with a double exponential curve (Supplementary Fig. S1). The direction of the error in the initial trials was consistent across subjects as they tended to hit the centre of the target ball and shot it forward towards the centre of the table. For measuring success rates and intertrial variability we divided the trials into blocks of 25 trials (each experimental set of 50 trials was divided into two blocks to increase the resolution in time). To improve robustness and account for outliers, we fitted the errors in each block with a t-distribution and used the location and scale parameters (µ and σ) as the blocks’ centre and variability measures. The learning curve over blocks (Fig. 1E) emphasised the reduction in the inter-subject variability during learning (decreasing error bars). The success rate over blocks (percentage of successful trials in each block; Fig. 1F) showed similar learning to the directional error.

Learning was also evident in the intertrial variability in the shooting direction which decayed over learning (Fig. 1G). Since learning also occurred within a block (especially during the first block) and the variability might be driven by the learning gradient, we corrected for it by calculating intertrial variability over the residuals from a regression line fitted to the ball direction in each block (while the learning curve is exponential, within the small blocks of 25 trials it is almost linear). This corrected intertrial variability showed only minor reduction in the initial blocks, relative to the uncorrected variability, and showed the same decay pattern over the learning (Fig. 1H). The scatter in the trial-by-trial data (Fig. 1D) is due to both inter-trial variability (evident in Fig. 1G,H) and inter-subject variability (evident in the error bars in Fig. 1E).

Overall, the task performance data suggested that subjects reached peak performance by the fifth experimental set (blocks 9–10, trials 200–250) and were doing the same (or even slightly worse) on the last experimental set (blocks 11–12, trials 250–300). Thus, we refer to the last two experimental sets (blocks 9–12, trials 201–300) as the ‘learning plateau’, while being mindful that professional pool players train over months and years to improve or maintain their skills.

Kinematic data were recorded using a wearable motion tracking ‘suit’ of wireless IMUs, where individual wireless sensors (matchbox-sized) were attached via Velcro to elastic straps fixed around the subjects’ body without constraining movement. The full-body kinematics were analysed in terms of joint angles using 3 degrees of freedom for each joint following the International Society of Biomechanics (ISB) recommendations for Euler angle extractions of Z (flexion/extension), X (abduction/adduction), and Y (internal/external rotation). Note, this standard approach includes hinge joints of the body which have only 1 degree of freedom being recorded as 3 Euler angles. The full-body movements were analysed over the angular joint velocity profiles of all joints. The data allowed us to reconstruct the full-body pose at any given moment, which we checked for visual correctness on a subject-by-subject basis against video ground truth. However, we chose not to look at joint angle’s probability distributions, as those are more sensitive to potential drifts in the IMUs (and contain small changes not spottable by the human eye). We previously showed that joint angular velocity probability distributions are more subject invariant than joint angle distributions suggesting these are the reproducible features across subjects in natural behavior29. In the current study, this robustness is quite intuitive: all subjects stood in front of the same pool table and used the same cue stick, thus the subjects’ body size influenced their joint angles distributions (taller subjects with longer arms had to bend more towards the table and flex their elbow less than shorter subjects with shorter limbs) but not joints angular velocity probability distributions (Fig. 2).

Figure 2
figure 2

Angular velocity profiles. Angular velocity profiles in 3 degrees of freedom (DoF) for each joint (blue: flexion/extension, red: abduction/adduction; green: internal/external rotation) averaged across subjects and trials over the first block of trials (1–25) in the inner circle (grey background) and the first block after learning plateau (201–225) in the outer circle (white background). Shaded areas represent the standard error of the mean. Data was recorded in 60 Hz, X axis is in seconds, covering a 1 s window around the timepoint the cue hit the ball. Y axis is in radians per second. The joints of the right arm which do most of movement in the task are highlighted in orange box and have a different scale on the y axis and grey line to indicate the y axis limits of all other joints. The avatar image in the centre is the full body reconstruction of one of the subjects during a trial, taken through the Xsens MVN analyze.

Full size image

In the first step of our data-driven analysis, we wanted to identify the key joints for the task: we analysed the angular velocity profiles of all joints, averaged across the initial block trials of all subjects, and found that most of the movement is done by the right arm, and specifically in the right shoulder (Fig. 2 inner circle). This is expected as all subjects were right-handed and used their right hand to hold the cue stick and make the shot. Taking domain understanding of the task into account, we can explain the shoulder movement by the naivety of the subjects, as pool billiards guidebooks30,31,32,33 emphasize that the shooting movement should be from the elbow down while the shoulder should be kept still. Correspondingly, the angular velocity profiles averaged across the initial block of the learning plateau (trials 201–225) showed similar distributions with an overall decrease in peak velocities relative to the initial trials but an increase in the peak angular velocity of ‘right elbow rotation’, which is the rotation between the upper arm and the forearm sensor and is equivalent to forearm supination (Fig. 2 outer circle).

The angular velocities of the shooting (right) arm were much larger than those of all other joints. For visibility, in Fig. 2 we used two different ranges on the y-axis for the joints of the shooting right arm (6 rad/s) and for all other body joints (2 rad/s). The high variance (relative to the mean) in some of the non-shooting arm joints that do move (such as the left elbow) could be caused primarily by variability across 1. trials and 2. subjects and specifically in its timing relative to the shot (e.g. While some subjects in some shots had a small left elbow movement just before the peak of the shot, others had it shortly after). Thus, aligning the traces with respect to the shot results in variability. In previous studies, the sensor noise was shown to be much smaller than both the inter- and intra-subject variability34, and to decay even more for slower movements35,36.

To quantify the overall change in the within-trial variability structure of the body over trials, we use the generalised variance, which is the determinant of the covariance matrix37 and is intuitively related to the multidimensional scatter of data points around their mean. We measured the generalised variance over the angular velocity profiles of all joints and found that it increased rapidly over the first ~ 40 trials and later decreased slowly (Fig. 3A). To understand what drives the generalised variance peak we plotted the variance–covariance matrixes of the first block, the second block (over the peak generalised variance), and ninth block (after learning plateaus) (Fig. 3B). It shows that the changes in the generalised variance were driven by an increase in the variance of all DoFs of the right shoulder and the negative covariance between the abduction/adduction and internal/external rotation of the right shoulder to the flexion/extension of the right shoulder and wrist. The internal/external rotation of the right elbow showed a continuous increase in its variance, which did not follow the trend of the generalised variance.

Figure 3
figure 3

Variance and complexity. (A) The trial-by-trial generalised variance, with a double-exponential fit (red curve). (B) The variance covariance matrix of the right arm joints angular velocity profiles averaged across subjects and trials over the initial block (trials 1–25), the second block (trials 26–50), in which the generalised variance peaks, and first block after learning plateau (block 9, trials 201–225). The order of the DoF for each joint is: flexion/extension, abduction/adduction, internal/external rotation. (C) The number of principal components (PCs) that explain more than 1% of the variance in the angular velocity profiles of all joints in a single trial, with an exponential fit (red curve). (D) The manipulative complexity (Belić and Faisal, 2015), with an exponential fit (red curve). (A,C,D) Averaged across all subjects over all trials.

Full size image

Next we set to study the complexity of the movement—as defined by the number of degrees of freedom used by the subject—since the use of multiple degrees of freedom in the movement is a hallmark of skill learning38. For that purpose, we applied Principal component analysis (PCA) across joints for the angular velocity profiles per trial for each subject and used the number of PCs that explain more than 1% of the variance to quantify the degrees of freedom in each trial movement. While in all trials of all subjects most of the variance can be explained by the first PC (Supplementary Fig. S2), there is a slow but consistent rise in the number of PCs that explain more than 1% of the variance in the joint angular velocity profiles (Fig. 3C). The manipulative complexity, suggested by Belić and Faisal39 as a way to quantify complexity for a given number of PCs on a fixed scale (C = 1 implies that all PCs contribute equally, and C = 0 if one PC explains all data variability), showed cleaner trajectory with the same trend (Fig. 3D). This suggests that over trials subjects use more degrees of freedom in their movement.

In the next step of our data-driven analysis, we wanted to identify signatures of learning joint-by-joint. For that, we defined a measure of task performance in single-joint space, which we named the Velocity Profile Error (VPE). VPE is the minimal correlation distances between the angular velocity profile of each joint in each trial to the angular velocity profiles of that joint in all successful trials (for more see “Methods” section). For all joints, VPE showed a clear pattern of decay over trials in an exponential learning curve (Fig. 4A). We fitted it with a single exponential learning curve (see fits time constants and goodness of fit in Supplementary Table S1). A proximal-to-distal gradient in the time constant of these learning curves was observed across the right arm, from the shoulder to the elbow and the wrist rotation (Supplementary Fig. S3). Intertrial variability in joint movement was measured over the VPEs in each block. Learning was also evident in the decay of the VPE intertrial variability during the learning over most joints across the body (Fig. 4B).

Figure 4
figure 4

Learning over joints. (A) Trial-by-trial Velocity Profile Error (VPE) for all 3 DoF of all joints, averaged across all subjects, with an exponential fit. The time constants of the fits are reported under the title and the goodness of fit is reported in Supplementary Table S1. (B) VPE intertrial variability (ITV) over blocks of 25 trials, averaged across all subjects. The color code of the DoF is the same as in Fig. 2 (blue: flexion/extension; red: abduction/adduction; green: internal/external rotation). The avatar image in the centre is the full body reconstruction of one of the subjects during a trial, taken through the Xsens MVN analyze.

Full size image

Inter-subject differences in variability and learning

In the final step of our data-driven analysis, we addressed the individuality of the subjects and looked across subjects for correlations between their task performance, learning rate, and joint movements. Since this is an exploratory study, all statistical tests are reported with caution and thus are not presented in the text but only in the figure (Fig. 5)—where the readers can see the data points and make their judgment as for the true significance. The statistics are presented in Spearman rank correlation, to deal account for outliers and non-linear trends, and p-values are FDR corrected for multiple comparisons. The regression lines are presented only for visual account and include their 95% confidence intervals to address outlier biases. We found substantial differences between subjects in their initial errors, final errors, intertrial variability, and learning, which are overlooked in the group average results. One subject, who had low initial errors, showed no learning, i.e. did not reduce her error over trials from the first block (trials 1–25) to the learning plateau (trials 201–300). For all other subjects, the final errors were smaller than the initial errors (Fig. 5A). There was a significant correlation between the initial and the final errors, meaning subjects with higher initial errors tended to have higher final errors as well.

Figure 5
figure 5

Variability and learning across subjects. (A) Correlation between subjects’ mean absolute directional error (in degrees) over the first block (trials 1–25) and the learning plateau (trials 201–300). (B) Correlation between subjects’ directional variability (in degrees) over first block (corrected for learning trend, see text) and over the learning plateau. (C) Correlation between subjects’ mean absolute directional error over the learning plateau and their learning. (D) Correlation between subjects’ directional variability over the first block (corrected for learning trend, see text) and their learning. (E) Correlation between subjects’ VPE variability (in logarithmic scale) over the first block and the learning plateau for the right arm joints. (F) Correlation between subjects’ VPE variability (in logarithmic scale) over the first block and their learning for the right arm joints. (A–F) Correlation values are Spearman rank correlation, p-values are FDR corrected for multiple comparisons, regression lines (black) are linear fits with 95% confidence intervals (doted lines). (A,B) unity lines are in grey.

Full size image

While over the learning most subjects decreased their intertrial variability in the outcome (ball direction; Figs. 1H, 5B) there was some tendency (though non-significant) for subjects who were initially more variable to be also more variable after learning (Fig. 5B). The intertrial variability of the joint angular velocity profiles, which also decreased over learning (Fig. 4B), showed a clearer and stronger correlation between the initial and the final intertrial variability (Fig. 5E, Supplementary Fig. S4). While this phenomenon was observed in various joints across the body, and dominant in the abduction across the spine joints, it was most dominant in the right shoulder abduction and rotation, the two joint angles that do most of the movement and carry most of its variance (Fig. 2).

Learning was defined as the difference between the initial error (over the first block: trials 1–25) and the final error (over the learning plateau: trials 201–300) normalised by the initial error. For the one subject who showed no learning (had bigger errors during the learning plateau than during the first block), we set learning to zero to avoid negative learning value. While there is a negative relation between learning and final error by definition, due to the normalization by the initial error (which was highly variant), there was no significant correlation between the learning and the final error (as subjects who started worse could have learned more but still not perform better after learning), but there was only a trend that more learning leads to smaller final errors (Fig. 5C). We speculated that the manipulative complexity (the degrees of freedom in the movement) might explain part of the inter-subject variability in learning rates. Presumably, subjects that learn more also show a higher increase in their manipulative complexity. Yet, we found no such relation. Both the initial (over the first block: trials 1–25) and the final (over the learning plateau: trials 201–300) manipulative complexity levels showed only a weak, non-significant, correlation to learning, and the increase in manipulative complexity showed no correlation to learning (Supplementary Fig. S5).

We then tested if higher levels of initial task-relevant motor variability (variability in the directional error of the target ball) in this complex real-world task could predict faster learning across individuals, as found in simple lab experiments40. We indeed found that individuals with higher intertrial variability in the directional error of the target ball over the first block showed more learning (Spearman rank correlation r = 0.64, p < 0.001; Fig. 5D). Importantly, this is the corrected intertrial variability (as in Fig. 1H) which is calculated over the residuals from a regression line fitted to the ball direction to correct for the learning that is happening within the block. As a control, we also tested for correlation with the initial variability in the target ball velocity—which is a task-irrelevant motor variability—and found no correlation (Spearman rank correlation r = 0.06, p = 0.77).

Since much of the learning is happening during the first 25 trials, calculating learning over blocks can lead to a ceiling effect. Therefore, to test the robustness of the correlation between learning and variability, we re-calculated the learning rate based on the 5, 7, 10, 15, and 20 initial trials (Supplementary Fig. S6). All choices of more than 5 trials showed a significant correlation between the initial variability and the learning. In the case of 5 trials, there were 2 outlier subjects with high variability and little learning who damaged the correlation. These 2 subjects had an initial ‘lucky shot’ which biased the learning calculation. Once including more trials, this effect was washed out.

Next, we tested the link between learning and initial variability over the joint angular velocity profiles of the right arm (Fig. 5F). We found that the only joint angle where the intertrial variability showed a significant correlation to learning was the right elbow rotation (Spearman rank correlation r = 0.47, p = 0.0086), which is the forearm supination. We further tested the link over the full-body kinematics (Supplementary Fig. S7) and found no other joint that showed this correlation. Thus, while learning leads to an overall reduction in movement variability, only initial variability in specific, task-relevant, dimensions can facilitate/predict learning.

Discussion

In this paper, we introduce a new paradigm for studying naturalistic motor learning during whole-body movement in a complex real-world motor skill task. Our results present new insights into motor learning in the real-world. While the learning curves in this in-the-wild paradigm are within the same range of those reported in reductionistic motor adaptation tasks2,41 we find that this learning is taking place not only in the task-relevant joints but across the entire body. Also, we found that task-relevant initial variability in the ball direction (movement outcome) can predict learning, like in laboratory-tasks40, and so can the initial variability in the right forearm supination which is the task-relevant joint angle variability.

While pushing towards real-world neuroscience, we started here with a relatively constrained version of the real-world task, asking subjects to perform repeated trials of the same pool shot. This was to enable analysis using well-developed methods of laboratory-tasks. Nonetheless, it is a major step in the direction of a naturalistic study. First, we allow full-body unconstrained movement. Second, we do not use any artificial go cue and allow self-paced movement and as many preparatory movements as the subject needs for each shoot. Third, subjects receive natural somatosensory feedback. And last, we do not perturb the feedback to induce learning.

Fundamentals of real-world motor learning

Across all subjects, we found that motor learning is a holistic process—the body is affected as a whole by learning the task. This was evident in the decrease in the VPE and the intertrial variability over learning (Fig. 4A,B). This result should not come as a surprise considering decades of research in sport science showing this relationship. For example, baseball pitcher's torso, pelvis, and leg movements are directly associated with ball velocity42,43,44. Recently it was also demonstrated with full-body motion capture in a ball throwing task45. And yet, unlike baseball pitches, basketball throws, or any unconstrained overarm throw, where the whole body is moving, in a pool shot the shooting arm is doing most of the movement and there is very little body movement. Thus, the whole-body learning is not trivial and suggestive that even in arm movement laboratory-tasks there is probably a whole-body learning aspect that is overlooked.

We also found a proximal-to-distal gradient in the learning rates over the right arm joints (Fig. 4A, Supplementary Fig. S3). This is especially interesting in light of the well-known phenomenon of proximal-to-distal sequence in limb movements in sports science46 and rehabilitation47. While there are records of proximal-to-distal sequence at multiple time scales48, our results are the first to suggest that this gradient also occur over repetitions as part of the learning process.

Variability and learning

Intertrial variability is a fundamental characteristic of human movements and its underling neural activity49. It was recently reported that individuals exhibit distinct magnitudes of movement variability, which are consistent across movements and effectors, suggesting individual traits in movement variability50. Our results show that subjects who were initially more variable tended to be also more variable after learning in many joints across the body (Fig. 5E, Supplementary Fig. S4) and specifically in those of right shoulder that carry most of the variance in the movement. This result is in-line with the notion that there is an individual trait in movement variability.

A decay in the intertrial variability is considered a feature of skill learning20,22,51,52,53,54,55,56, but not of error-based adaptation. While lower variability associated with lower means is a well-known statistical phenomenon, it is not evident in all learning tasks. In current billiards paradigm for example, in our VR version of the task, we report no decay in intertrial variability over learning57, which may suggest that the learning there is more through error-based adaptation. Here, across subjects, we see a decay in intertrial variability which suggests skill learning. Importantly, the novel finding here is that we see this decay in intertrial variability not only in the trial outcome (the direction of the target ball) but also in the body movement across the different joints (Fig. 4B). Therefore, our data suggest that the decay in the intertrial variability during skill learning is also a holistic process evident across the whole body.

While there is a clear group effect of intertrial variability decay, we also highlight inter-subject differences in the variability decay (Fig. 5B). Namely, while for some subjects the inter-trial variability after learning is much lower than in early learning, for others there is no real difference between the two and for some even a slight increase. This is discussed in depth in the follow-up paper where we analyse the neural activity in this task and discuss the learning mechanisms involved28. There we found two subgroups of subjects based on their neural activity while performing the task. We also found significant differences in the variability decay patterns of the two subgroups but not in their absolute error decay. The neural and behavioural differences led us to speculate that two different learning mechanisms are contributing to the task: error-based adaptation and reward-based reinforcement learning, where the predominant learning mechanism is different between subjects28. This might be transformative for rehabilitation.

Initial intertrial kinematic variability is also thought to be critical for motor learning58,59,60,61,62. It was suggested that individuals with higher levels of task-relevant movement variability exhibit faster motor learning in both skill learning and error-based motor adaptation paradigms40. The failures to reproduce this result in visuomotor adaptation studies63,64, led to the idea that experiments with task-relevant feedback (which is common in visuomotor studies) emphasize execution noise over planning noise, whereas measurements made without feedback (as in40) may primarily reflect planning noise62. This is in-line with a recent modelling work in a visuomotor adaptation study (with task-relevant feedback) in which subjects with higher planning noise showed faster learning, but the overall movement variability was dominated by execution noise that was negatively correlated with learning65. In our task there were no manipulations or perturbations, thus, task-relevant feedback was fully available to the participants. On the other hand, in real-world, there is no baseline variability, and the variability was measured during early learning and therefore is probably dominated by planning noise, as subjects explore, regardless of the visual feedback. Indeed, subjects with higher variability in the target ball direction over the first block showed higher learning rates (Fig. 5D). Our results straighten the link between variability and learning and are the first to show that it applies to real-world tasks. Moreover, the only joint angle that showed a significant correlation between initial variability and learning was the right forearm supination (measured by the right elbow rotation in our IMUs setup, Fig. 5F, Supplementary Fig. S7). Following the idea that task-relevant variability predicts learning, it would suggest that the right elbow rotation is the task-relevant joint angle to adjust during the initial learning of a simple pool shoot. Indeed, guidebooks for pool and billiards emphasize that while shooting one should keep one’s body still and move only the back (right) arm from the elbow down. While the elbow flexion movement gives the power to the shoot, the forearm supination (also known as ‘screwing’ in billiards) maintains the direction of the cue.

It is important to note that this refers specifically to the forearm supination around the elbow and not around the wrist. This is due to the nature of the data collected with the sensors suit where the joint angles are recorded with 3 degrees of freedom based on the angles between the sensors from both sides of each joint. Thus, hinge joints of the body which have only one anatomical degree of freedom have been recorded as 3 Euler angles. Specifically, the elbow rotation is the rotation between the upper arm sensor and the forearm sensor and is equivalent to forearm supination around the elbow. The wrist rotation is the rotation between the forearm sensor and the hand sensor and is equivalent to hand supination. Similarly, the elbow abduction/adduction should be negligible as it is a hinge joint66. Yet there is an abduction/adduction movement around our elbow (especially when the elbow flexion is around 90 degrees as in a pool shot) which is captured by the sensors and recorded as elbow abduction/adduction.

We note that the above results are correlational and cannot address the question of causality: e.g. can higher initial variability cause faster learning? While the study of real-world tasks takes us closer to understanding real-world motor learning, it is lacking the key advantage of laboratory tasks of highly controlled manipulations of known variables, to isolate specific movement/learning components. We therefore developed an embodied virtual reality (VR) version of our pool task57, As the VR-based approach enables visual feedback manipulations while subjects are executing the same real-world task. However, establishing a real-world task as done here is an essential precondition for understanding motor learning and the motor system and the way it has evolved and developed. For example, we have shown that measuring and incorporating unconstrained real-world kinematics can boost the decoding performance of smart neuroprosthetics67.

Conclusions

In this study, we demonstrate the feasibility and importance of studying human neuroscience in-the-wild, and specifically in naturalistic real-world skill tasks. While finding similarities in learning structure between our real-world paradigm and lab-based motor learning studies, we highlight crucial differences, namely, real-world motor learning is a holistic full-body process. Looking at the motor behaviour over learning across the entire body enabled us to explore the relationship between variability and learning and define task-relevant variability that can facilitate learning.

Methods

Ethics statement

All experimental procedures were approved by Imperial College Research Ethics Committee and performed in accordance with the declaration of Helsinki. All subjects gave informed consent prior to participating in the study. Written informed consent was obtained from the individual in Fig. 1A for the publication of any potentially identifiable images included in this article.

Experimental setup and design

30 right-handed healthy human volunteers with normal or corrected-to-normal visual acuity (12 women and 18 men, aged 24 ± 3) participated in the study. The recruitment criteria were that they played pool/billiards/snooker for leisure fewer than 5 times in their life, never in the recent 6 months, and had never received any pool game instructions. All volunteers gave informed consent before participating in the study, and all experimental procedures were approved by the Imperial College Research Ethics Committee and performed in accordance with the declaration of Helsinki. The volunteers stood in front of a 5 ft pool table (Riley Leisure, Bristol, UK) with 1 7/8″ (48 mm diameter) pool balls. Volunteers performed 300 repeated trials where the cue ball (white) and the target ball (red) were placed in the same locations. We asked volunteers to shoot the target ball towards the pocket of the far-left corner (Fig. 1A). Trials were split into 6 sets of 50 trials with a short break in-between to allow the subjects to rest a bit and reduce potential fatigue. Each experimental set (of 50 trials) took 8 to 12 min. For the data analysis, we further split each set into two blocks of 25 trials each, resulting in 12 blocks. During the entire learning process, we recorded the subjects’ full-body movements with a motion-tracking ‘suit’ of 17 wireless inertial measurement units (IMUs; Fig. 1B). The balls on the pool table were tracked with a high-speed camera (Dalsa Genie Nano, Teledyne DALSA, Waterloo, Ontario) to assess the subjects’ success in the game and to analyze the changes throughout learning, not only in the body movement but also in its outcome—the ball movement (Fig. 1C).

Balls tracking

The balls movement on the pool table were tracked with a computer vision system mounted from the ceiling. The computer vision camera was a Genie Nano C1280 Color Camera (Teledyne Dalsa, Waterloo, Canada), colour images were recorded with a resolution of 752 × 444 pixels and a frequency of 200 Hz. This Ethernet-based camera was controlled via the Common Vision Blox Management Console (Stemmer Imaging, Puchheim, Germany) and image videos recorded with our custom software written in C++ based on a template provided by Stemmer Imaging. Our software captured the high-performance event timer, the camera frames and converted the images from the camera’s proprietary CVB format to the open-source OpenCV (https://opencv.org/) image format for further processing in OpenCV. The video frames were stored as an uncompressed AVI file to preserve the mapping between pixel changes and timings and the computer’s real-time clock time-stamps were recorded to a text file. Each trial was subject-paced, so the experimenter observed the subject and hit the spacebar key as an additional trigger event to the time-stamps text file. This timing data was later used to assist segmentation of the continuous data stream into trials. The positions of the two pool balls (white cue ball and red target ball) were calculated from the video recordings offline using custom software written in C++ using OpenCV. Then, with custom software written in MATLAB (R2017a, The MathWorks, Inc., MA, USA), we segmented the ball tracking data and extracted the trajectory of the balls in each trial. For each trial, a 20 × 20 pixels (approx 40 × 40 mm) bounding box was set around the centre of the 48 mm diameter cue ball. The time the centre of the ball left the bounding box was recorded as the beginning of the cue ball movement. The pixel resolution and frame rate were thus sufficient to detect movement onset, acceleration and deceleration of the pool balls. The target (red) ball initial position and its position in the point of its peak velocity were used to calculate the ball movement angle (relative to a perfectly straight line between the white cue ball and the red target ball). We subtracted this angle from the centre of the pocket angle (the angle the target ball initial position and the centre of the pocket relative to the same straight line between the balls) to calculate the directional error for each shot.

Full-body motion tracking

Kinematic data were recorded at 60 Hz using a wearable motion tracking ‘suit’ of 17 wireless IMUs (Xsens MVN Awinda, Xsens Technologies BV, Enschede, The Netherlands). Data acquisition was controlled via a graphical interface (MVN Analyze, Xsens Technologies BV, Enschede, The Netherlands). Xsens MVN uses a biomechanical model and proprietary algorithms to estimate 3D joint kinematics68,69. The Xsens sensors shows high accuracy35,70, and the Xsens MVN system was used and validated in tracking real-world behaviour in many sports including football34, horse riding71, ski72 and snowboarding73. The Xsens 3D joint kinematics were exported as XML files and analysed using custom software written in MATLAB (R2017a, The MathWorks, Inc., MA, USA). The Xsens full-body kinematics were extracted in joint angles in 3 degrees of freedom for each joint that followed the International Society of Biomechanics (ISB) recommendations for Euler angle extractions of Z (flexion/extension), X (abduction/adduction) Y (internal/external rotation). This standard approach includes hinge joints of the body which have only 1 degree of freedom being recorded as 3 Euler angles.

Angular velocity profile analysis

From the Xsens 3D joint angles we extracted the angular velocity profiles of all joints in all trials. We defined the peak of the trial as the peak of the average absolute angular velocity across the DoFs of the right shoulder and the right elbow. We aligned all trials around the peak angular velocity of the trial and cropped a window of 1 s around the peak for the analysis of joint angular velocity profiles during the shot and its follow-through. This time window covered the entire movement of the pool shoot while eliminating the preparatory movement and the mock shoots (Fig. 2).

Task performance and learning measures

The task performance was measured by the trial error which was defined as an absolute angular difference between the target ball movement vector direction and the desired direction to land the target ball in the centre of the pocket. The decay of error over trials is the clearest signature of learning in the task. For measuring success rates and intertrial variability we divided the trials into blocks of 25 trials by dividing each experimental set of 50 trials to two blocks. This was done to increase the resolution in time from calculating those on the full sets. Success rate in each block was defined by the ratio of successful trial (in which the ball fell into the pocket). To improve robustness and account for outliers, we fitted the errors in each block with a t-distribution and used the location and scale parameters (µ and σ) as the blocks’ centre and variability measures. To correct for learning within a block, we also calculated a corrected intertrial variability, which was the intertrial variability over the residuals from a regression line fitted to the ball direction in each block. This correction for the learning trend within a block does not change the variability measure by much (Fig. 1G,H). This is since our variability measure is not the standard deviation, but the scale parameter of a t-distribution fitted to the errors. When correcting the change in the distribution fitted was mostly an increase in the degrees of freedom and a not decrease in the scale (i.e. the early trials which were much higher than the mean and the late trials which were much lower become closer to the mean and therefore the distribution is more normal as it loses the heavy tails). This is highlighting the robustness of the scale measure for variability.

To quantify the within-trial variability structure of the body movement, we use the generalised variance, which is the determinant of the covariance matrix37 and is intuitively related to the multidimensional scatter of data points around their mean. We measured the generalised variance over the velocity profiles of all joints in each trial to see how it changes with learning. To study the complexity of the body movement which was defined by the number of degrees of freedom used by the subject we applied principal component analysis (PCA) across joints for the velocity profiles per trial for each subject and used the number of PCs that explain more than 1% of the variance to quantify the degrees of freedom in each trial movement. We also calculated the manipulative complexity which was suggested by Belić and Faisal39 as a way to quantify complexity for a given number of PCs on a fixed scale (C = 1 implies that all PCs contribute equally, and C = 0 if one PC explains all data variability).

Statistical analysis

Trial by trial learning curves of single-trial performance measure (directional error of the target ball relative to the centre of the pocket) were fitted with a single, double, and triple exponential learning curve using Matlab fit function. As in most motor learning datasets, the double exponential curve showed the best fit (Supplementary Fig. S1).

As a measure of task performance in body space, correlation distances (one minus Pearson correlation coefficient) were calculated between the angular velocity profile of each joint in each trial to the angular velocity profiles of that joint in all successful trials. The minimum over these correlation distances produced a single measure of Velocity Profile Error (VPE) for each joint in each trial.

$${VPE}_{i}=\underset{s}{\mathrm{min}}\left(corrDist\left({velProf}_{i},{velProf}_{s}\right)\right)$$

Thus, VPE in trial \(i\) was the minimal correlation distances between the angular velocity profile in trial \(i\) (\({velProf}_{i}\)) and the angular velocity profiles in successful trials \(s\) (\({velProf}_{s}\)). While there are multiple combinations of body variables that can all lead to successful task performance, this measure looks for the distance from the nearest successful solution used by the subjects and thus provides a metric that accounts for the redundancy in the body.

All correlations between error, variability, and learning are Spearman's rank correlation coefficients to be robust to outliers and non-linear trends, and their p-values are FDR corrected for multiple comparisons. Regression lines are based on linear regression fits (in logarithmic scale for VPE variability) and are presented with 95% confidence intervals.