Identifying important sensory feedback for learning locomotion skills

Robot motor skills can be acquired by deep reinforcement learning as neural networks to reflect state–action mapping. The selection of states has been demonstrated to be crucial for successful robot motor learning. However, because of the complexity of neural networks, human insights and engineering efforts are often required to select appropriate states through qualitative approaches, such as ablation studies, without a quantitative analysis of the state importance. Here we present a systematic saliency analysis that quantitatively evaluates the relative importance of different feedback states for motor skills learned through deep reinforcement learning. Our approach provides a guideline to identify the most essential feedback states for robot motor learning. By using only the important states including joint positions, gravity vector and base linear and angular velocities, we demonstrate that a simulated quadruped robot can learn various robust locomotion skills. We find that locomotion skills learned only with important states can achieve task performance comparable to the performance of those with more states. This work provides quantitative insights into the impacts of state observations on specific types of motor skills, enabling the learning of a wide range of motor skills with minimal sensing dependencies. Traditional feedback-state selection in robot learning is empirical and requires substantial engineering efforts. Yu et al. develop a quantitative and systematic state-importance analysis, revealing crucial feedback signals for learning locomotion skills.


Introduction
The notion of learning machines predates the origins of cybernetics, control theories and apparatus in 1940s [62], with a longstanding interest in creating functioning replicas of living organisms. Robots with morphologies similar to their biological counterparts provide unique opportunities to develop machines with motion capabilities comparable to that of animals. As easyto-control platforms, robots allow scientists to study sensorimotor learning, providing opportunities to conduct control experiments and generate quantitative data analysis [26,31,39,11].
In the field of robotics, a large part of physical motor skills can be formulated as feedback control, i.e., control policies represented as state-action mapping. For conventional model-based approaches, such as trajectory optimization, task skills are represented by motion trajectories which are optimized based on the prior models from the domain knowledge [38,32], and the resulted motions are the natural outcomes of the optimized physical interactions rather than being the underlying motor control policies. For learning-based approaches, motor policies in the form of Deep Neural Networks (DNNs) can perform closed-loop control and generate actions constantly based on the sensing of feedback states. The selection of feedback states is essential for learning robot skills effectively, because inappropriate feedback states can impede the learning process, especially if key feedback is missing, then the robot will not be able to achieve the desired behaviors or will perform very poorly. Physics simulation allows access to as many ideal feedback states as possible, potentially leading to better results [30]. However, while transferring learned policies to real systems, often the easily-available privileged information in the simulation are no longer accessible or not reliable in real-world applications. Also, some states are not directly measurable in real robots compared to that in the simulation, and therefore need to be estimated by state estimation which is subject to computational errors and uncertainties [63,25]. In conclusion, it is desirable to employ realistic sensory information and avoid using unreliable or inaccessible sources of feedback during the training of DRL-based policies.
Deep reinforcement learning (DRL) is one of the promising approaches to learn a wide range of whole-body motor skills in robotic applications. For example, DRL methods can learn different legged locomotion skills, such as trotting [24,33,42,65], pacing, spinning [42], walking [19], galloping [56], balance recovery [24], and multi-skill locomotion [65]. Although deep neural networks (DNNs) show success in acquiring complex motor skills, due to the lack of interpretability [66,28], it is yet to answer how to determine the relative importance of different feedback states, or what state observations are more effective than others. Therefore, it often requires a lot of human insights and engineering efforts to select appropriate feedback states empirically in robot learning [57,30,44,33,35,59,34].
Different recent work selects a combination of representative feedback signals for learning quadruped locomotion. For example, joint position, joint velocity, angular velocity, and body orientation are used to learn free gait transitions between walking, trotting, pacing and bound-ing [49]. History of feedback signals has also been used in existing work, where a quadruped robot can adjust locomotion policies in the real world using history of root orientation, joint position and previous actions as input [52]. High speed locomotion on natural terrains has been achieved by using joint position and velocity, body orientation (gravity vector), and previous actions [36]. Overall, these feedback states are selected empirically and tested by trials without a systematic approach to determine the qualitative or quantitative importance of various feedback states among all the input signals.

Biological insights and motivation
Biological studies find that animals use multimodal sensory information collected from various sensory organs to achieve different locomotion tasks, including visual, mechanical, chemical, and thermal sensation, all in unison to render feedback from their own movements and the surroundings [15,45,58,10]. As shown in Fig. 1a, the vestibular system of vertebrates senses angular and linear accelerations, muscle spindles measure the stretch and stretch speed of skeletal muscles, the Golgi tendon organs measure exerted muscular forces, and the skin feels pressures among others. All this sensory information inherits redundancy which ensures robustness for the movement control in animal locomotion [46]. The Central Nervous System (CNS) fuses all information from these sensory organs and generates appropriate motor actions to complete tasks. To study the importance of different sensing information from various sensory organs, biological studies use lesion studies or ablation studies [14,53]. However, it is challenging to conduct such experiments on live animals due to ethical limitations and the difficulty of selectively stimulating and/or removing different receptors [41]. To date, robotic systems can have onboard sensors similar to their biological counterparts (see Fig. 1), and therefore offer a range of comparable proprioceptive and exteroceptive information to learn animal-like motor skills. The biological findings indicate that to recreate similar sensorimotor control in mechatronic robots, the understanding of the importance of different sensory feedback is crucial in order to produce desired behaviors, which is still missing in the robotics field.

Related work
Ablation experiments are commonly adopted to study the qualitative importance of individual feedback states, which focus on the impact of removing a single feedback state on performance. Experiments on a lamprey-like robot demonstrated that the distributed hydrodynamic force feedback contributes to the generation and coordination of rhythmic undulatory swimming motion [59]. Ablation studies showed that including the Cartesian joint position or contact information has different influences on learning robot locomotion behaviors [44]. Three sets of proprioceptive state feedback are used to learn CPG-based quadruped locomotion separately and foot contact booleans and CPG states are finally selected through ablation studies [5]. In the above research using ablation studies, a feedback state is considered to be important if the performance drops after it has been removed. In other words, the ablation study examines Fig. 1 | Comparison of various functional sensory feedback signals between the quadrupedal animals and their robotic counterparts. a, Sensory organs and feedback of a dog. The central nervous system fuses sensory information from various organs, such as the inner ear, muscle and skin, and then produces motor commands. The vestibular system of vertebrates senses angular and linear accelerations, muscle spindles measure the stretch and stretch speed of skeletal muscles, the Golgi tendon organs measure exerted muscular forces, and the skin feels pressures among others. b, Sensors and feedback on a legged robot. An onboard computer processes measurements of signals from different sensors, such as an inertial measurement unit, motor encoders and force sensors, and then generates actions for electric motors.
the difference of performance with and without the signals of interest. Therefore, the above ablation methods are not able to provide a quantitative ranking of the relative importance of each feedback signal, or the percentage of contribution of one sensing signal in comparison with others among the whole set of sensory feedback.
There have also been limited attempts to compare the importance of a certain type of feedback at different time steps quantitatively. For example, the influence of proprioceptive states on foot height commands was quantified and compared at different time steps [33]. However, this quantitative approach is used for analyzing the foot-trapping behavior during trotting and has not been extended to studying other motor skills.
To the best of our knowledge, the literature suggests a gap in formalizing systematic and quantitative approaches to compare the relative importance of different sensory feedback, which are the enabling tools to select the essential feedback and to learn various robot skills more effectively. Therefore, our research aims to answer the following open questions in the context of robot learning: What is the relative importance of different sensory feedback signals for a given motor task and various quadrupedal gaits? Which sensory feedback signals are essential, necessary and sufficient for learning quadrupedal locomotion? Which redundant feedback is beneficial to have but not entirely necessary?

Contribution
Here we propose a systematic approach to study the quantitative influence of different sensory feedback among distinct quadrupedal tasks, i.e., balance recovery, trotting and bounding, which are learned as feedback control policies represented by neural networks. We rank the level of importance of sensory feedback and identified a common set of essential feedback states for general quadruped locomotion. Further, we apply the proposed approach to learn new locomotion skills such as pacing and galloping using only the most essential feedback states.
In summary, our main contributions are: (i) Development of a systematic saliency analysis method to specifically quantify and rank the importance of each sensory feedback for a specific motor task; (ii) Identification of a common set of essential feedback states for general quadruped locomotion based on the saliency analysis of representative locomotion skills; (iii) Successful robot learning of new motor skills using only essential feedback states, demonstrating the efficacy of a minimal set of sensors.
Our study contributes to identifying the most essential feedback, i.e., key states, in a taskspecific manner, enabling robust motor learning using only the key states. The results provide new insights into the quantitative relative importance of different feedback states in locomotion behaviors. The identification of essential sensory feedback guides the selection of a minimum and necessary set of sensors, allowing robots to learn and perform robust motor skills with minimal sensing dependencies.

Results
This section elaborates on the investigation of the quantitative relative importance of common feedback states for three representative and distinct locomotion skills (balance recovery, trotting and bounding), and identification of a set of key feedback states that are consistently more important than others: joint positions, gravity vector (i.e., body orientation), base linear and angular velocities. Our results show that learning locomotion skills with only the key feedback states achieves comparable performance to using all available states. Further, we demonstrate the effectiveness of these key feedback states for learning new locomotion skills, such as pacing and galloping. Finally, we validate the robustness of the key-state policies in scenarios that were not encountered during training, and against robot uncertainties including sensing noises, and variations of robot mass and control gains. The concepts and ideas of our approaches are presented in Fig. 2 and more results can be seen in Supplementary Video 1-4. approach for identifying and employing key feedback states to learn effective locomotion skills. a, The proposed saliency analysis can rank the importance among different feedback states, given a set of feedback states and a targeted task-level skill. The doughnut chart shows our findings on the essential feedback states for quadruped locomotion, including joint position, gravity vector, base linear velocity and base angular velocity. b, Key-pose taxonomy used in robot pose initialization for learning effective balance recovery, trotting and bounding, respectively. Each key pose is represented by an array of body height, body orientation (roll and pitch angles) and joint positions, which determines the pose of a floating-base robot and is categorized by distinct contact patterns that are unique to the targeted type of gait. c, The DRL framework utilizing key feedback states and key-pose taxonomy that are sufficient for successful and effective learning of robot motor skills, where the phase vector (sin2πφ, cos2πφ) inputs to the policy network in parallel with key feedback states for periodic locomotion skills (φ is temporal information representing 0-100% phase over a gait period).
Identifying key feedback states for quadruped locomotion Quantifying the relative importance of feedback states We formulate a systematic saliency analysis for quantifying the relative importance of various feedback states for a desired motor skill. Inspired by the commonly available sensory information in animals (see Fig. 1a) and those widely used in deep reinforcement learning of robot locomotion [65, 33], we design a common set of feedback states, referred as full feedback states, including nine types of feedback states with 64 dimensions in total: base position, gravity vector, base angular velocity, base linear velocity, joint position/angle, joint velocity, joint torque, foot position, and foot contact (contact status or forces) (see Methods for definitions). Using this full set of states, we obtain the neural network policies for locomotion skills on the A1 quadruped robot [1] in PyBullet simulation via the DRL framework detailed in Fig. 2c and Supplementary Note 1. At each time step, we compute the saliency values of each dimension of the feedback states with respect to the associated actions using the integrated gradients method [55] (see Methods). The saliency value measures the influence of the input signal on generated actions. For each feedback dimension, it quantifies the amount of changes in output actions as the input signal varies at a certain time step. The higher the saliency value, the more associated actions change as the input signal varies, indicating a higher level of importance and task-relevance of a particular signal. Finally, we formulate the relative importance of a given feedback state as the percentage of its quantified importance over the entire period of motions with respect to the sum of importance of all the feedback states (see Methods).  3 | Ranking of feedback states and saliency analysis that identified top four key states for balance recovery, trotting and bounding skills. a, Saliency maps showing the importance of the 64dimensional feedback for three learned skills, respectively. At each time instant in the saliency map, a darker pixel indicates that the corresponding feedback signal has more importance and influence on the generated actions, compared to others. b, Doughnut charts showing the relative importance of nine feedback states for three learned skills, summarizing the overall averaged importance based on Fig.  3a. c, A box plot showing statistics of relative importance of each feedback state for balance recovery, trotting, and bounding, respectively. Each box reflects the slight variation of the relative importance of the corresponding feedback state among multiple learned policies for each locomotion skill (see details in Supplementary Figure 1-7). The full set of nine feedback states are ranked in an order of relative importance from high to low, suggesting the key feedback states for quadruped locomotion include joint positions, gravity vector (i.e., body orientation), base linear velocity, and base angular velocity. Blue, green and yellow shaded areas enclose key feedback states for three locomotion skills, respectively.

Key feedback states for quadruped locomotion
Here we identify key feedback states from the collection of nine feedback states according to the ranking of relative importance for three representative and distinct locomotion tasks: balance recovery, trotting and bounding, respectively.
To assess the significance of each feedback state in relation to their associated actions, we visualize saliency values as saliency maps (see Fig. 3a) for highlighting their relative importance, as well as the summary using doughnut charts (Fig. 3b) to delineate the contribution of each state over time and their overall impacts to the above skills -which reveals key feedback states with around 80% relative importance in total for the three skills summarized in Fig. 6c, compared to 20% relative importance for task-irrelevant states.
From the time plots of relative importance for nine feedback states in Extended Data Figure  1, we found that relative importance varies depending on the robot posture or phase over time. During balance recovery, gravity vector and joint positions are found to be the most important feedback state during body flipping and standing, respectively. During periodic trotting and bounding, within each gait cycle, the most crucial feedback state alternates between joint positions and base linear velocity.
Our study reveals that each joint has a different level of contribution to distinct locomotion tasks (Fig. 4). For example, sagittal movement states and joints that have a large range of motions usually have a higher contribution to locomotion than others. During trotting, as the rear legs deliver more power to propel the robot forward and overcome energy loss caused by friction and impacts, the rear hip pitch joints move in larger ranges, resulting in higher importance. During bounding, the front knee joints are more important than the rear knee joint positions, as bounding requires the front knee joints to buffer landing impacts more and provide stable body weight support during the pre-landing and stance phase. Overall, through our proposed systematic approach, a unified set of key states can be identified across three representative and distinct locomotion skills for quadruped robots, which are joint position, gravity vector, base linear velocity and base angular velocity.

Key feedback states under various circumstances
Rough terrain In previous results, we train the trotting and bounding on a flat ground. Here we aim to investigate if training on rough terrain will cause any changes in key feedback states. We train new trotting and bounding policies on rough terrain in a 6.4m × 6.4m area, which consists of 4096 cubes each with 0.1 m length and width, and heights sampled from (0 ∼ 3cm). Figure 2 show that the key states for trotting and bounding remain the same.

Results in Extended Data
Gait frequencies Trotting and bounding policies were trained with 1 Hz, 2Hz and 5 Hz, respectively. Saliency maps and doughnut charts in Supplementary Figure 8 indicate that key feedback states for trotting and bounding are consistent across low, medium and high gait frequencies. Thus, we conclude that the identified key feedback states are not affected by the gait frequency within such a range.
Foot contact status versus foot contact forces The foot contact can be represented in various formulations for learning locomotion skills. The previous analysis uses sigmoid contact which is a continuous signal indicating the contact status (see Methods). Here we replace the sigmoid foot contact with normalized foot contact forces for learning trotting and bounding, where the foot contact forces are normalized by the body weight and capped between zero and one. We conclude that using either of the foot contact formulations as feedback states renders the same conclusions regarding the importance of foot contact (see Supplementary Figure 9).

Importance of history states
To investigate the impact of historical information on sensory feedback selection, we trained a new policy for balance recovery with an expanded set of state input, including states at the current time step and two history steps. The results of the saliency map and the bar plot in Extended Data Figure 3a-b show that: (i) for each type of feedback state, current information is more important than history information; (ii) important states remain important within each set of history states; and (iii) the overall importance of all states at the current time step is much higher than that at both history steps.

Correlation between feedback states
Heatmaps were generated to reflect the absolute value of Pearson correlation coefficient [12] across feedback states ( Fig. 6e and Extended Data Figure 3c). The average correlation coefficients between any two types of current feedback states were visualized using chord diagrams in Fig. 6f and Extended Data Figure 3d. These findings reveal that the key states identified for balance recovery are correlated with all other task-irrelevant states to varying degrees. Moreover, the measurements of these key states are usually less noisy, making them more suitable to be used. Therefore, these results suggest that selecting key states with underlying correlations with other signals can effectively reduce the number of sensors required for feedback in the closed-loop control.
Quantifying the relative importance of the feedforward phase vector Importance of the feedforward phase vector For trotting and bounding, the phase vector is used to enforce the cyclic pattern as a feedforward input to the neural network besides the feedback states ( Fig. 2c and Methods). We shall note that the phase vector is included in the training of all feedback control policies for trotting and bounding but excluded from the ranking of the important states except this section. Without the phase vector as input, the robot would fail to learn cyclic motion. The analysis in Extended Data Figure 4 shows that the feedforward phase vector counts for the relative importance of 28% and 38% for trotting and bounding, respectively, which is more important than any type of feedback states.
Importance of the phase vector during swing and stance From the time plot of saliency value for phase vector in Extended Data Figure 5, we found the importance of the phase vector follows a cyclic pattern, reaching the highest value twice within each gait around the transitions between swing and stance during trotting and bounding. The cyclic pattern of the phase vector is synchronized with foot-ground contact. Such high importance during the contact transitions indicate that the phase vector regulates the timing of establishing and breaking foot-ground contact, resulting in a synchronized cyclic pattern with the transitions of swing and stance.

Benchmarking of learned motor skills
To validate the effectiveness of our proposed approach to identifying important states, we formulate task-related performance metrics for the quantitative evaluation of three locomotion tasks (see Methods), and benchmark the following five settings in ten scenarios with the same DRL framework: (i) full-state policies with nine feedback states (see For balance recovery, the metrics include recovery speed, final foot placement, final body height, final body orientation and joint torque. For trotting and bounding, the same physical quantities are used, including forward velocity, heading accuracy, body height, body orientation and joint torque. By averaging the five performance metrics (see Methods), we found the performance of key-state policies is comparable to full-state policies for three skills (Fig.  5d), and if key feedback states are missing, there would be a significant drop in task performance (forward velocity and heading accuracy for trotting and bounding) or learning success rate (balance recovery). More details can be found in Fig  task performance for full-state policies and key-state policies for balance recovery, trotting and bounding. a, Balance recovery from lying down on the back by a full-state policy (top) and a key-state policy (bottom). b, Learned trotting gait by a full-state policy (top) and a key-state policy (bottom). c, Learned bounding gait by a full-state policy (top) and a key-state policy (bottom). d, Comparison of task performance using metrics for full-state, key-state, and irrelevant-state policies over balance recovery, trotting, and bounding. Higher values of metrics indicate better performance. The key-state policies achieved 94.7%, 99.1% and 93.7% on average, respectively, compared to that of the full-state policies. Irrelevant-state trotting and bounding policies achieve 11.7%, 5.9% in forward velocity and 57.3%, 57.3% in heading accuracy with respect to full-state policies. Although the selected irrelevant-state balance recovery policy achieves similar performance to full-state and keystate policies, the success rate of learning such policy drops significantly. Details of robustness tests in Supplementary Note 3.

Applicability to learning new skills
This section aims to investigate whether the identified key feedback states can be directly used for learning new locomotion skills. Using the key feedback states identified from three rep-resentative locomotion skills and the newly designed task-specific key poses (Supplementary Figure 17), the A1 quadruped robot successfully learns robust pacing and galloping gaits, (   6 | The use of key feedback states for learning novel locomotion skills. a, Successful pacing gait learned by A1 robot with an average forward velocity of 0.46m/s. b, Successful galloping gait learned by A1 robot with an average forward velocity of 1.57m/s. c, Key feedback states used for balance recovery, trotting and bounding skills. Each color represents one type of feedback state: blue, green, yellow and orange dots represent joint position, gravity vector, base linear and angular velocities, respectively. d, The 2D projections of the 18-dimensional trajectories (body height, body linear velocity, roll and pitch angles and joint positions) sampled from key-state pacing and galloping policies, full-state and key-state balance recovery, trotting and bounding policies using t-SNE. e, Heatmap showing absolute Pearson correlation coefficient between any two dimensions of nine feedback states from full-state balance recovery policy, where a darker color indicates stronger correlation. f, Chord diagram summarizing the correlation between any two feedback states for full-state balance recovery (self-correlation and percentage < 25% are removed for clarity), where the wider the link between the two states, the stronger they correlate with each other. Fig. 6d, we found that the trajectories of key-state pacing and galloping are located within the circle formed by the trajectories of balance recovery, trotting and bounding policies. This suggests that the studied skills encapsulate the common patterns of leg movements with sufficient diversity and the newly learned skills are closely-related neighborhood skills to these skills. Therefore, using the same set of key states can effectively learn both new gait patterns. However, learning more distinct new locomotion skills may require identifying new key feedback states, which can be achieved by reapplying our approach.

Discussion
This work has developed a quantitative analysis method to investigate the quantitative influence of various common feedback states for learning locomotion skills. In contrast to qualitative ablation experiments for state selection, the quantitative importance ranking obtained by our proposed systematic and efficient approach offers a useful guideline for selecting the most essential and task-relevant feedback states to learn a wide range of robot behaviors, such as balance recovery, trotting, bounding, pacing, and galloping. As a result of motor learning through neural networks, the importance of states is indirectly encapsulated by a large amount of learned neural network weights.
Our method contributes to the interpretation, comparison and validation of the state importance by a direct ranking of quantitative relative importance over common sensory feedback for quadruped locomotion. The study suggests that joint positions, gravity vector, base linear and angular velocities are the essential states, comprising 80% total relative importance for motor learning. In comparison, foot positions, foot contact, base position, joint torque and velocities play a less critical role and are the types of feedback signals that are better to have but not entirely indispensable, and therefore they can be excluded from motor learning without significantly affecting robot performance.

Benefits for robot control design
Our method provides a quantitative ranking of feedback states and hence identifies the level of their importance, guiding the selection of a minimum and suitable set of sensors to learn robust quadruped locomotion. Different combinations of sensors can be chosen based on the hardware availability of particular applications. For example, we found motor encoders, Inertial Measurement Unit (IMU), and estimation of body linear velocities suffice to achieve common quadruped locomotion skills.
To summarize, this research benefits the design of robot control in several aspects: (i) improves design efficiency of the learning framework by selecting important feedback states through one single training session, which is more efficient than empirical trial-and-error or ablation studies that require multiple iterative processes; (ii) promotes lightweight and cost-effective robot design by equipping only task-relevant sensors; (iii) reduces the need for developing state estimation for task-irrelevant or unimportant states, thus reducing the dependencies of task success on sensing and state estimation uncertainties that helps to mitigate sim-to-real mismatch.
Our quantitative analysis approach requires a successful sensorimotor policy, i.e., a differentiable state-action mapping of the motor skills, to be used for identifying the importance level of states. In cases where motion learning is initially infeasible via reinforcement learning, all commonly available sensing shall be included to facilitate motor learning, or other approaches like supervised learning or imitation learning can be used if demonstrations are available.

Feedforward pathways
In this work, to simplify the implementation, the periodic feedforward signals are implemented as a 2-dimensional phase vector (sin2πφ, cos2πφ) generated directly using the temporal information φ (0-100% phase over a gait period), representing continuous periodicity (see Fig. 2c). The use of such phase input serves as a representation of priors or prior knowledge as in other previous works [29,64], and the phase signals are not modulated by sensory feedback signals in our study.
The use of the feedforward signals, i.e., the 2-dimensional phase vector, may result in a small contribution of foot contact information as revealed in this study. The types of locomotion skills investigated in this study do not involve complex terrain properties (e.g., soft, muddy, granular), and the limited uncertainties of the ground contact further reduce the necessity or reliance on foot contact forces during the motor learning process of state-action mapping. Additionally, our supplementary analysis has revealed a correlation between foot contact and the phase signals, as detailed in Supplementary Figure 19. This correlation also implies that once the locomotion gait reaches a limit cycle, a stable gait can be sustained without the explicit use of foot-ground contact information if there are no large external perturbations.
In principle, these feedforward signals could be implemented in CPG networks and be modulated in various ways, e.g. with phase resetting mechanisms or with feedback terms that continuously modulate the phase and amplitude of CPG signals, as in [40,4,17,5]. In future work, it would be interesting to include such feedback mechanisms and investigate whether the relative importance of different sensory modalities would change. For instance, it might lead to a higher importance of load feedback, which has been suggested to be important for cat locomotion [16] and shown to be a sufficient source of information for interlimb coordination [40].

Relation to biology
On top of our findings contributing to robotics and learning, the importance of key feedback states provides meaningful implications on sensorimotor control in animal locomotion. In general, our findings are in agreement with biological findings and hypotheses. The key sensory feedback found by our studies on quadrupedal robots maps to the vestibular system and muscle spindles which have been proved to be critical for postural control and goal-directed vertebrate locomotion on the biological counterparts [45,18]. Our analysis also reveals that important states vary with tasks and certain sensory signals are more critical than others at different moments during a gait cycle, consistent with existing biological findings and hypotheses [45,9].
It is important to note that our conclusions on important states stem from common locomotion skills on a mechanical robot and may differ from the general findings in animals, such as the importance of contact forces and limb loading as discussed in the previous section. Our contribution to biology is to provide biologists with the computational approach to identify important states on simulated animals, e.g., neuromechanical simulations [22], when ethical or technical limitations prevent certain biological studies on live animals.

Limitation and future work
To obtain general conclusions on state importance with minimal human bias and good statistical characteristics, we designed basic reward functions that encourage the exploration of feasible but slightly different motor policies across training sessions. Compared to the state-of-the-art locomotion [42,33, 65], we do not rely on reference trajectories or heavily fine-tuned reward terms to achieve natural-looking gaits. This allows the feedback control policy to fully exploit the solution space, drawing general conclusions about the importance of different states.
The identified key important states enabled successful learning of motor skills, demonstrating robustness to uncertainties in sensing, environment, and robot dynamics (Supplementary Video 1-4), which suggests that our saliency analysis based on the neural network, i.e., the mapping from filtered state input to unfiltered action output, has captured essential features of the overall policy for the identification of key feedback states. As future work, further investigation of other components of the framework and the properties of the overall policy would be an interesting direction. While considering the accuracy of sensory feedback, the importance ranking of feedback state can be further refined by composing the saliency map and sensitivity matrix of sensor noise levels, see details in Supplementary Note 4.
In future applications, in case certain states are identified as important but unreliable due to hardware limitations, one potential solution is to train a neural network to infer the estimation of such error-prone states from more reliable feedback. Prior work [27] has demonstrated the feasibility of such an approach, e.g., estimating body velocity and foot contact from joint positions measured by motor encoders, and the gravity vector plus base angular velocities obtained from IMU measurements.

Robot platform
We chose the A1 quadruped robot [1] for our study, which is close to a real small-size dog and commonly used for locomotion research (see robot specifications in Supplementary Table 4). The extensive simulation validations were conducted in a physics-based simulation -PyBullet [13]. All the robot locomotion tasks were simulated in PyBullet with high-fidelity physics, and resulted robot motions and data were rendered in Unity [21] for high resolutions snapshots and videos, which provide better visualization quality of the physical interactions and movements.

Saliency analysis Integrated gradients
Our use of the saliency analysis is inspired by feature attribution methods in image classification and explainable artificial intelligence [51,66,55,28]. There are several saliency methods or attribution methods, such as integrated gradients [55], guided backpropagation [54], DeepLift [50] and Grad-CAM [48]. Here, we use integrated gradients method to define the saliency values of feedback states for a learned policy. Integrated gradients method satisfies two axioms which are desirable for attribution methods [55]: (i) sensitivity, i.e., the attribution should be non-zero if each input and baseline lead to different outputs; and (ii) implementation invariance, i.e., the attributions should be the same for two networks if the outputs of both networks are identical for all the inputs, regardless of the detailed implementation. Some other common saliency methods are not able to satisfy both.
For example, vanilla gradients of the output with respect to the input and guided backpropagation break the axiom sensitivity [50], which will result in the gradients focusing on irrelevant features. Another common technique DeepLift breaks the axiom implementation invariance, where results may differ for the networks with same functionalities but different implementation. In summary, the use of integrated gradients allows us to identify feedback states that are truly relevant. The same conclusion holds for each type of locomotion skill regardless of the implementation of deep neural networks, as long as the outputs of two implementations are the same for the identical inputs.
It shall be noted that the integrated gradients method is applicable to the state-action mapping that is differentiable, meaning that it can analyze the influence of feedback states of motor skills which can be represented by differentiable machine learning models. However, it cannot be applied to underlying state-action policies that are nondifferentiable. At time step t, for the i-th dimension of feedback states x ∈ R n , integrated gradients G(x i,t ) are defined as follows: where F (x t ) ∈ R m are the generated actions at time step t,x i,t is the baseline input zero, p = 25 is the number of steps in the Riemann approximation of the integral. The partial derivative is computed through backpropagation by calling tf.gradients() in TensorFlow [2]. For a better visualization to reveal the relative importance among feedback states via saliency maps, we define the raw saliency value S d (x i,t ) as follows instead of directly using the computed integrated gradients: where, N is the number of total time steps during the entire motion. The raw saliency value S d (x i,t ) is further normalized to the range of [0, 1] as follows:

Relative importance of feedback states
For the i-th dimension of feedback states x ∈ R n , overall importance during the entire motion I i is computed as follows: where S(x i,t ) is the saliency value for x i at time step t, and N is the number of total time steps during the entire motion.
For feedback state o ∈ R h , (h ≤ n), overall importance I o is computed as follows: where i(o, q) is the index of the dimension of feedback states x which maps to the q-th dimension of feedback state o (see Supplementary Table 1).
This work considers nine feedback states in total. For feedback state o, relative importance r o is defined as follows: Feedback states for learning locomotion skills Here we introduce a longlist of nine candidate states used in legged locomotion and the way to measure or estimate them on a real robot: (i) base position in the world frame which can be estimated using visual inertial odometry [37], the three-dimensional base position was used rather than the base height alone for a fair comparison with other states; (ii) normalized gravity vector in the robot's local frame which reflects the body orientation of the robot and can be computed using roll and pitch angle measurements from IMU; (iii) base angular velocity measured by IMU; (iv) base linear velocity in the robot heading frame estimated by fusing leg kinematics and the acceleration from IMU; (v) joint position measured by motor encoders; (vi) joint velocity which is measured by motor encoders and further normalized by maximum joint velocity; (vii) joint torque which is measured by torque sensors and further normalized by maximum joint torque; (viii) foot position relative to base in the robot heading frame which can be computed through forward kinematics; and (ix) foot contact with the ground which is computed by applying sigmoid function to the L2 norm of contact force F i measured by the force sensor at the end of the i-th foot as follows: where c 1 = c 2 = 2.0. Thus, foot contact is continuous within the range of [0, 1] (an example of continuous foot contact is in Extended Data Figure 5) without a discontinuous switch between zero and one as in a threshold function which may affect the differentiability of the neural network for applying our analysis. For learning periodic locomotion tasks, such as trotting and bounding, we included a two-dimensional feedforward phase vector [sin(2πφ), cos(2πφ)] on top of the above set of feedback states to represent continuous temporal information that encodes phase φ from 0% to 100% of a gait period. At each time step, the phase increases by a constant increment without any phase resetting mechanisms and is computed as: where, k is the control step counter, T is desired gait period, f c is the control frequency. The phase vector is the feedforward term and thus was excluded for the quantification and comparison of the state importance, since the focus of this study is on the feedback terms.
For full-state policies, this complete set of feedback states were used for balance recovery (without the feedforward phase vector), trotting and bounding. For key-state policies, the states used were different for the three locomotion skills (see Fig. 6c). Specifically, states (ii) and (v) were used for balance recovery, states (ii), (iv), (v) and phase vector were used for trotting, states (ii), (iii), (iv), (v) and phase vector were used for bounding. Learning pacing and galloping skills used the same set of states as for bounding.

Key-pose taxonomy for effective exploration and learning
During the training of control policies using full feedback states, the robot pose is randomly initialized at each training episode to encourage the exploration of diverse states, which is a technique commonly used in robot learning [24,44]. However, random initialization is not data efficient in exploring the state space for a type of locomotion task, as most robot configurations are a priori invalid, due to the physical feasibility of the balance criteria. In other words, most of the robot configurations are not balanced or very far away from the desired locomotion, and therefore the collected samples are skewed by invalid exploration and less efficient for learning.
To this end, on top of the key feedback states, we propose key-pose taxonomy to initialize the robot configuration at each training episode, as seeding conditions to enable more effective exploration and learning.
Inspired by animal locomotion [3,6] and whole-body support pose taxonomy from humanoid robotics [7,8], given a specific locomotion task, we can design key-pose taxonomy which consists of representative robot-ground contact configurations and distinct robot poses. The robot-ground contact configurations are straightforward to obtain, since quadrupedal locomotion is well studied in biology and we can easily obtain representative contact phases, e.g., trotting of dogs and horses. Compared to the existing pose taxonomy [7,8] which is for the classification and inter-transitions of loco-manipulation, our proposed key-pose taxonomy here aims at task-specific effective learning.
Specifically, we use the configuration space of a floating-based robot to define each key pose, which is composed of body height, body orientation (roll and pitch angles), and joint angles. The base linear velocity and base angular velocity were set as zero at the start of each episode. Given the same contact configuration, we shall note that a quadruped robot may have multiple poses. For example, crouching and standing share the same robot-ground contact by four feet. Therefore, to balance the aspect of diversity, based on each ground contact or gait phase, we can use the robot configuration space to define multiple distinct key poses, so as to increase the number of initial poses which can sparsely cover the feasible motions related to a task.
Following this principle, we designed five key poses for balance recovery, six for trotting, four for bounding, four for pacing, and five for galloping as shown in Fig. 2b and Supplementary  Figure 17. The key poses within the designed taxonomy were sampled to initialize the robot pose at each training episode for each task, and the detailed transitions between the key poses will be explored during the learning process and thus obtained as a natural outcome. Using key-pose taxonomy as initial posture setting makes learning more efficient by narrowing the solution space for learning compared to random exploration.
In this work, we assigned the key poses within the taxonomy with equal probability for the DRL agent to encounter and explore upon. It shall be noted that the probability of each key pose can be more flexible. For example, we can assign higher probabilities to the key poses which are task-relevant but less likely to be encountered in natural interactions with the environment.

Quantitative metrics of performance evaluation
To quantify the performance for comprehensive comparison, a set of performance metrics, S, is designed for each task. For balance recovery, the performance metric set S recovery = {s τ , s r , s f , s h N , s φ N }. For trotting and bounding, the performance metric sets S trotting and S bounding are the same, i.e., S = {s τ , s v , s ψ , s h , s φ }. Note that the performance metrics are used for post-learning performance evaluations, which are not the same as the reward terms designed for learning in terms of formulations and weights for each physical quantity.
The metric value for each physical quantity is in the range of [0, 1], and N is the number of total time steps of an episode. Joint torque is used for the performance evaluation across all the three locomotion tasks. The performance metrics for this physical quantity are evaluated as follows. A higher value of joint torque metric indicates a more energy-efficient motion.
where τ i,t is the joint torque of the i-th joint at time step t, andτ = 33.5 N m is the maximum joint torque.

Performance metrics for balance recovery
For balance recovery, the performance metrics for the other four physical quantities are evaluated as follows: (i) Recovery speed metric where T is the time duration of recovery to a standing posture andT is the time duration from the start of recovery to the end of an episode. A higher value indicates the recovery is completed within a shorter time period.
(ii) Final foot placement metric where where h N is the body height at the end of recovery andĥ = 0.25 m is the nominal standing height of the robot. A higher value means that the final body height is closer to the nominal standing height of the robot.
(iv) Final body orientation metric where g N is the gravity vector of the robot at the end of recovery, andĝ = [0, 0, −1] is the nominal gravity vector of the robot. A higher value indicates that the final body orientation is closer to the nominal body orientation, i.e., zero roll angle and pitch angle.

Performance metrics for trotting and bounding
For trotting and bounding, the other four physical quantities are the same and the performance metrics are evaluated as follows: (i) Forward velocity metric where V t is the forward velocity in the horizontal plane at time step t,V is the nominal forward velocity, andV = 0.5 m/s for trotting andV = 1.0 m/s for bounding. A higher value indicates a faster average forward velocity during the entire episode. It shall be noted that a lower value for bounding does not indicate worse learning performance. The reason is that we did not set a desired forward velocity strictly in the reward for training bounding as for trotting. We set the desired velocity for training bounding as 1.0m/s. However, we do not penalize velocity higher than 1.0m/s to encourage higher velocity if possible. As a result, the robot may learn bounding policies with different average forward velocities with the same training settings.
(ii) Heading accuracy metric where (iii) Body height metric where h t is the body height at time step t, andĥ = 0.3 m is the nominal height of the robot. A higher value means that the body height is closer to the nominal height during the entire episode.
(iv) Body orientation metric where g t is the gravity vector of the robot at time step t, andĝ = [0, 0, −1] is the nominal gravity vector of the robot. A higher value indicates that the body orientation is closer to the nominal body orientation during the entire episode, i.e., zero roll angle and pitch angle.

Metrics for task-performance evaluation
Given the five individual performance metrics for each locomotion task, we can further evaluate overall performance of key-state policies with respect to full-state policies. Consider a taskrelated physical quantity i, metric for the key-state policy s i,key and metric for the full-state policy s i,f ull , overall performance of the key-state policy s key is defined as follows: Additionally, we can compute the mean of s key for multiple tasks to statistically evaluate the overall performance (balance recovery, trotting, bounding) in a statistical manner.

Summary of key concepts and ideas
In the following sub-section, present a set of questions and answers which capture the main concepts and ideas discussed in the paper. These concise summaries aim to help readers to form key takeaway messages and understand the core content of the research.
How to select feedback states through the proposed saliency analysis of motor skills?
We first define a comprehensive list of nine feedback states related to common physical quantities of a legged robot. Then, we train the desired motor skill using this list of feedback states. The obtained full-state policies are then used to perform the saliency analysis. See Section Saliency analysis in the main text for more details. Through the saliency analysis, we can obtain an importance ranking (Fig. 3c) among all the feedback states and then select feedback states for learning motor skills accordingly.
In which cases the saliency analysis can be used? Which cases are not applicable?
The proposed saliency analysis can be applied to any differentiable state-action mapping. A typical case is a differentiable neural network and is not restricted to a specific neural network architecture or implementation. However, it is not applicable to nondifferentiable models since gradients need to be computed.

Can saliency analysis be applied to traditional control methods?
The saliency analysis cannot be directly applied to traditional control methods since it can only be applied to differentiable models. However, after some processing of the trajectories generated by traditional control methods, saliency analysis can be applied. One feasible way is to collect the state-action pairs from traditional control methods and find a neural network to model the state-action mapping via supervised learning. Then, the saliency analysis can be applied to the learned model.
Is it possible to identify in advance sensor specifications (e.g., sensor type and location, etc.) needed to generate a specific robot motion?
Identifying sensor specifications, such as sensor type and location, for a specific robot motion is a challenging task without prior motor skills or application scenarios. While domain knowledge and human heuristics may be applied, it is hard to formulate appropriate quantitative analysis and the investigation of sensor specifications is out of the scope of this paper. Our approach requires an existing neural network policy in the first place and exploits the simulated motions to obtain a differentiable state-action mapping of the motor skills, in the form of a neural network, to determine appropriate types of sensory feedback information for a given motion task. Though this requires additional computational resources, it is necessary for effective identification and analysis of key feedback states in motor skill learning, similar to other computer-/simulation-aided design approaches.
How to quantify and benchmark the performance of each motor skill?
We define five performance metrics for each motor skill. See more details in Section Quantitative metrics of performance evaluation in the main text. The value of each metric is between zero and one, and a higher value indicates a better performance correspondingly. The average value of all the five performance metrics indicates the overall performance of the motor skill, which can be used for benchmarking.
What are the differences in terms of performance of the learned policies using full feedback states and key feedback states?
Using the performance metrics defined for each motor skill, we demonstrate that the learned policies using key feedback states can achieve comparable overall performance to those using full feedback states across different locomotion skills in various test scenarios. Here, the performance of the full-state policies are used as baselines to benchmark performance from other policies. As shown in Fig. 5d, the learned policies using only the key feedback states can achieve 94.7%, 99.1% and 93.7% of the task performance of those using full feedback states on a flat ground for balance recovery, trotting and bounding, respectively.
Moreover, the learned key-state policies can also guarantee similar level of performance in robustness tests as shown in Supplementary Figure 16. When the robot is perturbed by a flying box, the the overall performance (balance recovery, trotting, bounding) of the key-state policies achieves 101.1%, 100.6% and 97.4% respectively, compared to the baseline. The key-state policies can achieve 96.8%, 101.3% and 96.1% of the overall performance for these three locomotion skills over unseen rubble terrain.

Supplementary Information
Supplementary Figure  Supplementary Figure 16. Comparison of performance metrics for full-state policies and key-state policies for balance recovery, trotting, and bounding in two unseen scenarios. Supplementary Figure 17. Key-pose taxonomy for learning pacing and galloping gaits. Supplementary Figure 18. A box plot showing statistics of relative importance of nine feedback states reflected by the maximum saliency value for balance recovery, trotting and bounding. Supplementary Figure 19. Analysis of the correlation between the feedforward phase vector (sin2πφ, cos2πφ) and nine feedback states for trotting and bounding.  Supplementary Figure 9. Influence of different foot contact formulations on saliency analysis. a, Saliency maps and doughnut charts for trotting and bounding using sigmoid foot contact. b, Saliency maps and doughnut charts for trotting and bounding using normalised foot contact.
Supplementary Figure 10. Comparison of the learning curves for three cases: full feedback states with random-pose initialization, key feedback states with random-pose initialization, and key feedback states with key-pose initialization, for balance recovery, trotting and bounding, respectively. For each learning curve, we first compute the mean and standard deviation of episode return from five learning sessions, and further apply moving average with a sliding window size of five epochs for better visualization. a, Learning curves for balance recovery. b, Learning curves for trotting. c, Learning curves for bounding. Figure 11. Snapshots of A1 robot recovering from various initial fall postures. a, The robot recovers from lying on its left to a nominal standing posture using a full-state policy (top) and a key-state policy (bottom), respectively. b, The robot recovers from lying on its right to a nominal standing posture using a full-state policy (top) and a key-state policy (bottom), respectively. Supplementary Figure 16. Comparison of performance metrics for full-state policies and key-state policies for balance recovery, trotting, and bounding in two unseen scenarios (corresponding to snapshots in Extended Data Figure 6 and Supplementary Figure 15). a, Performance metrics of full-state policies and key-state policies under the box attack for balance recovery (left), trotting (middle) and bounding (right). The overall performance (balance recovery, trotting, bounding) of key-state policies with respect to full-state policies is 101.1%, 100.6% and 97.4%, respectively. b, Performance metrics of full-state policies and key-state policies over unseen rubble for balance recovery (left), trotting (middle) and bounding (right). The overall performance of key-state policies with respect to full-state policies is 96.8%, 101.3% and 96.1%, respectively.

Supplementary
Supplementary Figure 17. Key-pose taxonomy for learning pacing and galloping gaits. a, Key-pose taxonomy for pacing, including four key poses. b, Key-pose taxonomy for galloping, including five key poses.
Supplementary Figure 18. A box plot showing statistics of relative importance of nine feedback states reflected by the maximum saliency value for balance recovery, trotting and bounding, based on the saliency maps in Supplementary Figure 1 Supplementary Figure 19. Analysis of the correlation between the feedforward phase vector (sin2πφ, cos2πφ) and nine feedback states for trotting and bounding. a, Chord diagrams showing the correlation between any two states (self-correlation and percentage < 50% are removed for clarity), where the wider the link between the two states, the stronger they correlate with each other. b, Bar plot showing the absolute Pearson correlation coefficient between the phase vector and each of the nine feedback states.

Supplementary tables
Supplementary Steps per epoch 5e3

Supplementary notes
Deep reinforcement learning framework The objective of DRL is to learn a policy which can generate optimal actions given states observed from the environment. We used the Soft Actor Critic (SAC) algorithm [20] incorporated with smoothing loss to train the policy [65]. Here we provide details about essential components of our DRL framework for learning locomotion skills, including state observation, action space, neural network architecture, reward terms, control framework and training procedure.

State observation and action space
The design of state observation is detailed in Methods. Regarding the action space, we adopted the design from previous work [65], since it has been shown to improve performance for certain motor tasks [43]. Specifically, the actions are twelve desired joint positions for a quadruped robot.

Neural network architecture
Policy network. A stochastic policy π θ (s t ) is used as the actor in SAC for training, which is modeled as a Gaussian distribution N (µ θ (s t ), σ θ (s t ) 2 ) with mean µ θ (s t ) and standard deviation σ θ (s t ), where θ is the parameter set of a fully-connected neural network and s t is the state observed from the environment at time step t. The policy network has two hidden layers, each with 256 neurons and using a ReLU activation function. Suppose we have network input X ∈ R x , the network output Y ∈ R 24 is computed as follows: where, w 0 ∈ R 256×x , w 1 ∈ R 256×256 , w 2 ∈ R 24×256 are neural network weights, b 0 ∈ R 256 , b 1 ∈ R 256 , b 2 ∈ R 24 are neural network biases. The network output Y is in the range of [−1, 1], which is then scaled by the corresponding joint range (see Supplementary Table 4). When deploying the learned policy on the robot, we use the deterministic mean outputs of the stochastic policy µ θ (s t ) as the references of joint angles.
For full-state policies, the network input dimension is x = 64 for balance recovery, and x = 66 during trotting and bounding. For key-state policies, the network input dimension is x = 15, x = 20 for balance recovery and trotting, respectively, and x = 23 for bounding, pacing and galloping.
Critic network. The critic in SAC consists of two Q functions which are encoded as two separate fully-connected neural networks. Similar to policy network, each critic network has two hidden layers, each with 256 neurons and using a ReLU activation function. The network input includes state and action, and the output is a scalar value.

Reward terms
The reward function designed for learning quadruped locomotion is composed of eleven weighted reward terms related to the following physical quantities [65]: (i) base orientation φ, (ii) base height h, (iii) base linear velocity v, (iv) joint torque τ , (v) joint velocityq, (vi) body ground contact C bg , (vii) foot ground contact C f g , (viii) symmetric foot placement p f oot , (ix) swing and stance h f oot v f oot , (x) reference foot contactĈ f g , and (xi) yaw velocityψ. Balance recovery uses reward terms (i)-(viii), trotting, bounding, pacing and galloping use reward terms (i)-(xi). We use a radial basis function (RBF) to formulate each reward term related to continuous physical quantities: where x is the continuous physical quantity,x is the corresponding reference, and α is the shape parameter which controls the width of RBF. Reward terms (vi), (vii) and (x) which are related to discrete physical quantities are designed separately. The detailed mathematical formulation and weight for each reward term for the five locomotion tasks are in Supplementary Table 5 and 6, respectively.

Control framework
Our control framework for legged locomotion consists of a high-level behavior loop running the policy network at 25Hz and a low-level joint position tracking loop running joint-level PD controllers at 1kHz. In the high-level control loop, the measured or estimated states are first passed to a first-order low-pass butterworth filter with a cut-off frequency of 10Hz. The policy network receives the filtered states as input and generates desired joint positions at 25Hz. In the low-level control loop, the desired joint positions from the neural network output are filtered by a first order low-pass butterworth filter with a cut-off frequency of 4Hz. Joint-level PD controllers receive the filtered joint position references and the measured joint positions, and generate joint torques at 1kHz according to the following equation: where, τ is the computed joint torque, q d is the desired joint position, q m is the measured joint position, and K p , K d are PD gains (see Supplementary Table 3).

Training procedure
Soft Actor Critic with smoothing loss. Soft Actor Critic learns an optimal policy π * (s t ) by maximizing the expected sum of rewards and entropy: where s t and a t are the observed state and action at time step t, ρ π is the sample distribution, r(s t , a t ) is the reward received at time step t, H(π(·|s t )) is the entropy of policy π at time step t, and α is temperature parameter which is automatically tuned in SAC to balance exploitation and exploration. To minimize the applied joint torques and generate smooth robot motions, we introduce an additional smoothing loss term J smooth as in our previous work [65]: where µ(s t ) are the deterministic mean outputs of the stochastic policy used as the desired joint references, and q m are the measured joint positions. Thus the goal of the DRL agent in this work is to find a policy that maximizes the following objective: J(π(s t )) = J SAC (π(s t )) − J smooth (µ(s t )) Meanwhile, SAC also adopts Double Q-learning to learn two Q functions, details of which can be found in the literature [23,61]. The SAC hyperparameters are shown in Supplementary  Table 7.
Initialization. In this work, we used two methods to initialize the robot pose during training for comparison, i.e., random-pose initialization and key-pose initialization. The pose of the robot consists of: base height, base orientation (roll and pitch angles) and twelve joint positions. We note that base linear velocity and base angular velocity were both set as zero at the start of each episode. Key-pose initialization is introduced in Methods. For random-pose initialization, the robot was initially at a fixed height of 0.4m, yaw angle was set as zero, roll and pitch angles were both sampled from a uniform distribution [−180 • , 180 • ], and each of the twelve joint positions was sampled from a uniform distribution of its corresponding joint range [q min , q max ].
For balance recovery, the distribution of joint positions was the same as the joint range of the A1 quadruped robot, seen in Supplementary Early termination. During training, each episode can proceed either for a fixed time period or until certain termination criteria are met. For balance recovery, each episode terminates after a fixed time period of 10s. For trotting, bounding, pacing and galloping, we designed early termination criteria in order to augment the samples collected by the agent. Specifically, an episode will terminate if any body segment excluding feet is in contact with the ground or body orientation exceeds 90 • . Otherwise, the episode terminates after 10s.

Performance benchmark
As shown by the time-elapsed snapshots of balance recovery, trotting and bounding in Fig.  5a-c, the key-state policy can successfully acquire all the skills equally well as the full-state policy. Using only the key states, the robot was fully able to recover from various fallen poses to a standing posture, such as lying down on its back (Fig. 5a) and many other configurations (Supplementary Figure 11). Moreover, it can perform trotting (Fig. 5b) and bounding gaits (Fig. 5c) successfully and stably. However, irrelevant-state policies fail to move forward for trotting and bounding (see Supplementary Video 1).
The bar plots in Fig. 5d show the values of performance metrics of all the task-related physical quantities for the full-state policy and the key-state policy, which are nearly identical to each other. There are three nuances: balance recovery on the recovery speed and final foot placement; bounding on the forward velocity. Although these quantities slightly differ in terms of numerical values, both policies perform well and have little differences in terms of the actual task completion.
For the balance recovery speed, such as where the robot stands up from lying on its back in Fig. 5a, it takes 1.48s for the full-state policy and 1.84s for the key-state policy to complete. Regarding the final foot placement, as seen in the robot snapshots in Fig. 5a, both policies control the robot to stand with four feet placed symmetrically at the end of recovery. As for the subtle difference in the forward bounding velocity, we observed a slightly lower average steady-state velocity which comes from a naturally learned stable gait, as there was no penalty if the measured forward velocity is higher than the desired velocity specified during training (see Methods). More detailed data plots of task-related physical quantities can be found in Supplementary Figure 12, for the full-state and key-state policies of three locomotion tasks.

Robustness tests
We validated the robustness of key-state policies in two categories of uncertainties: (i) environmental uncertainties, i.e., scenarios that were not encountered during training; and (ii) robot uncertainties, i.e., sensing noises, and variations of robot mass and control gains. All the robust locomotion tests on the A1 robot in these unseen scenarios can be found in Supplementary Video 3.
We designed two unseen scenarios reflecting environmental uncertainties: (i) locomotion under box attack (Extended Data Figure 6a) and (ii) locomotion over uneven rubble (Extended Data Figure 6b), which were applied to balance recovery, trotting and bounding. In the box-attack scenario, a 10kg box was thrown at the robot's body from 1m away at an initial velocity of 11m/s, from the right side of the robot (for balance recovery) and from the top (for trotting and bounding). The robot was able to withstand all these force perturbations and continue the task when using the key-state policies (Extended Data Figure 6a). In the uneven rubble scenario, 200 cubes (3cm edge length) were randomly piled within an 0.8m × 0.8m area to test balance recovery policies; for trotting and bounding, 200 cubes were randomly distributed in an area of 0.8m × 0.5m. The robot was able to succeed in all the rubble scenarios across three locomotion tasks using key-state policies (Extended Data Figure 6b).
Besides, we also tested the robustness of full-state policies in the same scenarios as a baseline for comparison (Supplementary Figure 15). We concluded that key-state policies can achieve task performance equally well in all three locomotion tasks, compared to that of the full-state policies, with an average performance of 99.7% in the box-attack scenario and 98.1% in the uneven rubble scenario (see quantified metrics in Supplementary Figure 16).
Moreover, the key-state policies also demonstrated robustness against robot uncertainties in the following aspects: (i) noisy feedback states, where the noise was sampled from a Gaussian distribution with zero mean and corresponding standard deviation (see details in Supplementary  Table 2) and added to the corresponding feedback state; (ii) robot mass variation, where the original robot mass was changed by ±10%; and (iii) variation of the control gain, i.e., the proportional gain K p for joint-level PD control was altered by ±10% away from the original K p value (Supplementary Table 3). The A1 robot was able to successfully complete all the robustness tests, and see Supplementary Video 3 for the robust locomotion skills in action.

Evaluation of impact of sensing accuracy
We can study the impact of feedback accuracy by analyzing and composing two sensitivity matrices: (i) Sensitivity of action with respect to the state changes, i.e., saliency maps; (ii) Sensitivity matrix representing the sensor noise levels for each type of feedback states. We formulate the sensitivity matrix of the sensor noise level M sensor as follows: where σ sensor is the standard deviation of the sensor noise level, s max and s min is the upper and lower range of the state. Then, the sensitivity matrix of the action with respect to the measurement's quality can be evaluated as follows by composing two sensitivity matrices element-wise:

Comparison of relative importance using maximum saliency values
Due to the phase dependencies of saliency analysis discussed in Results, we conduct further investigation to validate whether it is appropriate to compare the relative importance of states by using the mean of saliency values over all time steps. We compare the results using both the mean and the maximum of saliency values to quantify the relative importance.
For a certain type of feedback state o ∈ R h , importance I o can be computed in two ways: (i) add up the saliency values over all time steps for each dimension of a type of feedback and average the importance over dimensions: (ii) find the maximum saliency value over all time steps across all the dimensions of a type of feedback: where S(o i,t ) is the saliency value for the i-th dimension of feedback state o at time step t, and N is the number of total time steps during the entire motion.
This work considers nine feedback states in total. For feedback state o, relative importance r o is defined as follows: Figure 3c and Supplementary Figure 18 show the rankings of relative importance among states according to mean and maximum saliency values, respectively, based on the same set of saliency maps in Supplementary Figure 1-3. While using maximum saliency value to rank the relative importance, we found that the key important states remain the same, although there are only slight variations within the key states, e.g., gravity vector becomes the highest ranking feedback state, replacing joint positions. Therefore, we confirm that using the mean of saliency values over all time is an effective way to identify the key states for locomotion skills.