Optimality principles in sensorimotor control

Abstract

The sensorimotor system is a product of evolution, development, learning and adaptation—which work on different time scales to improve behavioral performance. Consequently, many theories of motor function are based on 'optimal performance': they quantify task goals as cost functions, and apply the sophisticated tools of optimal control theory to obtain detailed behavioral predictions. The resulting models, although not without limitations, have explained more empirical phenomena than any other class. Traditional emphasis has been on optimizing desired movement trajectories while ignoring sensory feedback. Recent work has redefined optimality in terms of feedback control laws, and focused on the mechanisms that generate behavior online. This approach has allowed researchers to fit previously unrelated concepts and observations into what may become a unified theoretical framework for interpreting motor function. At the heart of the framework is the relationship between high-level goals, and the real-time sensorimotor control strategies most suitable for accomplishing those goals.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Schematic illustration of open- and closed-loop optimization.
Figure 2: Minimal intervention principle.
Figure 3: Application of optimal feedback control to a redundant stochastic system.

References

  1. 1

    Chow, C.K. & Jacobson, D.H. Studies of human locomotion via optimal programming. Math. Biosci. 10, 239–306 (1971).

  2. 2

    Hatze, H. & Buys, J.D. Energy-optimal controls in the mammalian neuromuscular system. Biol. Cybern. 27, 9–20 (1977).

  3. 3

    Davy, D.T. & Audu, M.L. A dynamic optimization technique for predicting muscle forces in the swing phase of gait. J. Biomech. 20, 187–201 (1987).

  4. 4

    Collins, J.J. The redundant nature of locomotor optimization laws. J. Biomech. 28, 251–267 (1995).

  5. 5

    Popovic, D., Stein, R.B., Oguztoreli, N., Lebiedowska, M. & Jonic, S. Optimal control of walking with functional electrical stimulation: a computer simulation study. IEEE Trans. Rehabil. Eng. 7, 69–79 (1999).

  6. 6

    Anderson, F.C. & Pandy, M.G. Dynamic optimization of human walking. J. Biomech. Eng. 123, 381–390 (2001).

  7. 7

    Rasmussen, J., Damsgaard, M. & Voigt, M. Muscle recruitment by the min/max criterion—a comparative numerical study. J. Biomech. 34, 409–415 (2001).

  8. 8

    Nelson, W.L. Physical principles for economies of skilled movements. Biol. Cybern. 46, 135–147 (1983).

  9. 9

    Hogan, N. An organizing principle for a class of voluntary movements. J. Neurosci. 4, 2745–2754 (1984).

  10. 10

    Flash, T. & Hogan, N. The coordination of arm movements: an experimentally confirmed mathematical model. J. Neurosci. 5, 1688–1703 (1985).

  11. 11

    Pandy, M.G., Garner, B.A. & Anderson, F.C. Optimal control of non-ballistic muscular movements: a constraint-based performance criterion for rising from a chair. J. Biomech. Eng. 117, 15–26 (1995).

  12. 12

    Todorov, E. & Jordan, M.I. Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J. Neurophysiol. 80, 696–714 (1998).

  13. 13

    Smeets, J.B. & Brenner, E. A new view on grasping. Motor Control 3, 237–271 (1999).

  14. 14

    Uno, Y., Kawato, M. & Suzuki, R. Formation and control of optimal trajectory in human multijoint arm movement: Minimum torque-change model. Biol. Cybern. 61, 89–101 (1989).

  15. 15

    Nakano, E. et al. Quantitative examinations of internal representations for arm trajectory planning: minimum commanded torque change model. J. Neurophysiol. 81, 2140–2155 (1999).

  16. 16

    Harris, C.M. & Wolpert, D.M. Signal-dependent noise determines motor planning. Nature 394, 780–784 (1998).

  17. 17

    Harwood, M.R., Mezey, L.E. & Harris, C.M. The spectral main sequence of human saccades. J. Neurosci. 19, 9098–9106 (1999).

  18. 18

    Hamilton, A.F. & Wolpert, D.M. Controlling the statistics of action: obstacle avoidance. J. Neurophysiol. 87, 2434–2440 (2002).

  19. 19

    Pandy, M.G., Zajac, F.E., Sim, E. & Levine, W.S. An optimal control model for maximum-height human jumping. J. Biomech. 23, 1185–1198 (1990).

  20. 20

    Valero-Cuevas, F.J., Zajac, F.E. & Burgar, C.G. Large index-fingertip forces are produced by subject-independent patterns of muscle excitation. J. Biomech. 31, 693–703 (1998).

  21. 21

    Todorov, E. Cosine tuning minimizes motor errors. Neural Comput. 14, 1233–1260 (2002).

  22. 22

    Fagg, A.H., Shah, A. & Barto, A.G. A computational model of muscle recruitment for wrist movements. J. Neurophysiol. 88, 3348–3358 (2002).

  23. 23

    Ivanchenko, V. & Jacobs, R.A. A developmental approach aids motor learning. Neural Comput. 15, 2051–2065 (2003).

  24. 24

    Loeb, G.E., Levine, W.S. & He, J. Understanding sensorimotor feedback through optimal control. Cold Spring Harb. Symp. Quant. Biol. 55, 791–803 (1990).

  25. 25

    Hoff, B. A Computational Description of the Organization of Human Reaching and Prehension. Ph.D. Thesis, Univ. Southern California (1992).

  26. 26

    Hoff, B. & Arbib, M.A. Models of trajectory formation and temporal interaction of reach and grasp. J. Mot. Behav. 25, 175–192 (1993).

  27. 27

    Kuo, A.D. An optimal control model for analyzing human postural balance. IEEE Trans. Biomed. Eng. 42, 87–101 (1995).

  28. 28

    Shimansky, Y.P. Spinal motor control system incorporates an internal model of limb dynamics. Biol. Cybern. 83, 379–389 (2000).

  29. 29

    Ijspeert, A.J. A connectionist central pattern generator for the aquatic and terrestrial gaits of a simulated salamander. Biol. Cybern. 84, 331–348 (2001).

  30. 30

    Todorov, E. & Jordan, M. Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, 1226–1235 (2002).

  31. 31

    Shimansky, Y.P., Kang, T. & He, J.P. A novel model of motor learning capable of developing an optimal movement control law online from scratch. Biol. Cybern. 90, 133–145 (2004).

  32. 32

    Li, W., Todorov, E. & Pan, X. Hierarchical optimal control of redundant biomechanical systems. in Proceedings of the 26th Annual International Conferences of the IEEE Engineering in Medicine and Biology Society (IEEE Press, ISSN 1094-687X, in the press).

  33. 33

    Marr, D. Vision (Freeman, San Francisco, 1982).

  34. 34

    Feldman, A.G. & Levin, M.F. The origin and use of positional frames of reference in motor control. Behav. Brain Sci. 18, 723–744 (1995).

  35. 35

    Bizzi, E., Hogan, N., Mussa-Ivaldi, F.A. & Giszter, S. Does the nervous system use equilibrium-point control to guide single and multiple joint movements? Behav. Brain Sci. 15, 603–613 (1992).

  36. 36

    Kelso, J.A.S. Dynamic Patterns: The Self-Organization of Brain and Behavior (MIT Press, Cambridge, Massachusetts, 1995).

  37. 37

    Sporns, O. & Edelman, G.M. Solving Bernstein's problem: a proposal for the development of coordinated movement by selection. Child Dev. 64, 960–981 (1993).

  38. 38

    Kalaska, J., Sergio, L.E. & Cisek, P. in Sensory Guidance of Movement: Novartis Foundation Symposium (ed. Glickstein, M.) 176–201 (Wiley, Chichester, UK, 1998).

  39. 39

    Bernstein, N.I. The Coordination and Regulation of Movements (Pergamon, Oxford, 1967).

  40. 40

    Loeb, G.E., Brown, I.E. & Cheng, E.J. A hierarchical foundation for models of sensorimotor control. Exp. Brain Res. 126, 1–18 (1999).

  41. 41

    Todorov, E. & Ghahramani, Z. Unsupervised learning of sensory-motor primitives. in Proceedings of the 25th Annual International Conference of the IEEE Engineering in Biology and Medicine Society 1750–1753 (IEEE Press, ISSN 1094-687X, 2003).

  42. 42

    Kalman, R. When is a linear control system optimal? Trans. AMSE J. Basic Eng. Ser. D 86, 51–60 (1964).

  43. 43

    Moylan, P. & Anderson, B. Nonlinear regulator theory and an inverse optimal control problem. IEEE Trans. Automatic Control AC-18, 460–465 (1973).

  44. 44

    Ng, A. & Russell, S. Algorithms for inverse reinforcement learning. in Proceedings of the 17th International Conference on Machine Learning (Morgan Kaufmann, San Francisco, 2000).

  45. 45

    Scott, S. Optimal feedback control and the neural basis of volitional motor control. Nat. Neurosci. Rev. 5, 534–546 (2004).

  46. 46

    Kording, K. & Wolpert, D. The loss function of sensorimotor learning. Proc. Natl. Acad. Sci. USA 101, 9839–9842 (2004).

  47. 47

    Lacquaniti, F., Terzuolo, C. & Viviani, P. The law relating the kinematic and figural aspects of drawing movements. Acta Psychol. 54, 115–130 (1983).

  48. 48

    Sutton, G.G. & Sykes, K. The variation of hand tremor with force in healthy subjects. J. Physiol. (Lond.) 191, 699–711 (1967).

  49. 49

    Schmidt, R.A., Zelaznik, H., Hawkins, B., Frank, J.S. & Quinn, J.T. Jr. Motor-output variability: a theory for the accuracy of rapid notor acts. Psychol. Rev. 86, 415–451 (1979).

  50. 50

    Sabes, P.N., Jordan, M.I. & Wolpert, D.M. The role of inertial sensitivity in motor planning. J. Neurosci. 18, 5948–5957 (1998).

  51. 51

    Rosenbaum, D.A. Human Motor Control (Academic, San Diego, 1991).

  52. 52

    Burdet, E., Osu, R., Franklin, D.W., Milner, T.E. & Kawato, M. The central nervous system stabilizes unstable dynamics by learning optimal impedance. Nature 414, 446–449 (2001).

  53. 53

    Hoffman, D.S. & Strick, P.L. Step-tracking movements of the wrist. IV. Muscle activity associated with movements in different directions. J. Neurophysiol. 81, 319–333 (1999).

  54. 54

    Zhu, G., Rotea, M.A. & Skelton, R. A convergent algorithm for the output covariance constraint control problem. Siam J. Control Optimization 35, 341–361 (1997).

  55. 55

    Bryson, A. & Ho, Y. Applied Optimal Control (Blaisdell, Waltham, Massachusetts, 1969).

  56. 56

    Kirk, D. Optimal Control Theory: An Introduction (Prentice Hall, Englewood Cliffs, New Jersey, 1970).

  57. 57

    Davis, M.H. & Vinter, R.B. Stochastic Modelling and Control (Chapman and Hall, London, 1985).

  58. 58

    Bertsekas, D. & Tsitsiklis, J. Neuro-Dynamic Programming (Athena Scientific, Belmont, Massachusetts, 1997).

  59. 59

    Sutton, R.S. & Barto, A.G. Reinforcement Learning: An Introduction (MIT Press, Cambridge, Massachusetts, 1998).

  60. 60

    Todorov, E. & Li, W. Optimal control methods suitable for biomechanical systems. in Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Press, ISSN 1094-687X, 2003).

  61. 61

    Li, W. & Todorov, E. Iterative linear quadratic regulator design for nonlinear biological movement systems. in Proceedings of the 1st International Conference on Informatics in Control, Automation, and Robotics (Kluwer, in the press).

  62. 62

    Montague, P.R., Dayan, P., Person, C. & Sejnowski, T.J. Bee foraging in uncertain environments using predictive hebbian learning. Nature 377, 725–728 (1995).

  63. 63

    Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

  64. 64

    O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

  65. 65

    Seymour, B. et al. Temporal difference models describe higher-order learning in humans. Nature 429, 664–667 (2004).

  66. 66

    Knill, D. & Richards, W. Perception as Bayesian Inference (Cambridge Univ. Press, 1996).

  67. 67

    Wolpert, D., Gharahmani, Z. & Jordan, M. An internal model for sensorimotor integration. Science 269, 1880–1882 (1995).

  68. 68

    Kording, K.P. & Wolpert, D.M. Bayesian integration in sensorimotor learning. Nature 427, 244–247 (2004).

  69. 69

    Saunders, J.A. & Knill, D.C. Visual feedback control of hand movements. J. Neurosci. 24, 3223–3234 (2004).

  70. 70

    Shadmehr, R. & Mussa-Ivaldi, F.A. Adaptive representation of dynamics during learning of a motor task. J. Neurosci. 14, 3208–3224 (1994).

  71. 71

    Flanagan, J.R. & Wing, A.M. The role of internal models in motion planning and control: Evidence from grip force adjustments during movements of hand-held loads. J. Neurosci. 17, 1519–1528 (1997).

  72. 72

    Kawato, M. Internal models for motor control and trajectory planning. Curr. Opin. Neurobiol. 9, 718–727 (1999).

  73. 73

    Flanagan, J.R. & Lolley, S. The inertial anisotropy of the arm is accurately predicted during movement planning. J. Neurosci. 21, 1361–1369 (2001).

  74. 74

    Li, C.S., Padoa-Schioppa, C. & Bizzi, E. Neuronal correlates of motor performance and motor learning in the primary motor cortex of monkeys adapting to an external force field. Neuron 30, 593–607 (2001).

  75. 75

    Gribble, P.L. & Scott, S.H. Overlap of internal models in motor cortex for mechanical loads during reaching. Nature 417, 938–941 (2002).

  76. 76

    Ostry, D.J. & Feldman, A.G. A critical evaluation of the force control hypothesis in motor control. Exp. Brain Res. 153, 275–288 (2003).

  77. 77

    Thoroughman, K.A. & Shadmehr, R. Learning of action through adaptive combination of motor primitives. Nature 407, 742–747 (2000).

  78. 78

    Wang, T., Dordevic, G.S. & Shadmehr, R. Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses. Biol. Cybern. 85, 437–448 (2001).

  79. 79

    Flanagan, J.R., Vetter, P., Johansson, R.S. & Wolpert, D.M. Prediction precedes control in motor learning. Curr. Biol. 13, 146–150 (2003).

  80. 80

    Korenberg, A. Computational and Psychophysical Studies of Motor Learning. PhD Thesis, University College London (2003).

  81. 81

    Desmurget, M. & Grafton, S. Forward modeling allows feedback control for fast reaching movements. Trends Cogn. Sci. 4, 423–431 (2000).

  82. 82

    Gribble, P.L. & Ostry, D.J. Compensation for interaction torques during single- and multijoint limb movement. J. Neurophysiol. 82, 2310–2326 (1999).

  83. 83

    Humphrey, D.R. & Reed, D.J. in Advances in Neurology: Motor Control Mechanisms in Health and Disease (ed. Desmedt, J.E) 347–372 (Raven, New York, 1983).

  84. 84

    Conditt, M.A. & Mussa-Ivaldi, F.A. Central representation of time during motor learning. Proc. Natl. Acad. Sci. USA 96, 11625–11630 (1999).

  85. 85

    Woodworth, R.S. The accuracy of voluntary movement. Psychol. Rev. Monogr. 3 (1899).

  86. 86

    Fitts, P. The information capacity of the human motor system in controlling the amplitude of movements. J. Exp. Psychol. 47, 381–391 (1954).

  87. 87

    Meyer, D.E., Abrams, R.A., Kornblum, S., Wright, C.E. & Smith, J.E.K. Optimality in human motor performance: Ideal control of rapid aimed movements. Psychol. Rev. 95, 340–370 (1988).

  88. 88

    Milner, T.E. & Ijaz, M.M. The effect of accuracy constraints on three-dimensional movement kinematics. Neuroscience 35, 365–374 (1990).

  89. 89

    Flash, T. & Henis, E. Arm trajectory modifications during reaching towards visual targets. J. Cogn. Neurosci. 3, 220–230 (1991).

  90. 90

    Komilis, E., Pelisson, D. & Prablanc, C. Error processing in pointing at randomly feedback-induced double-step stimuli. J. Motor Behav. 25, 299–308 (1993).

  91. 91

    Todorov, E. & Jordan, M. in Advances in Neural Information Processing Systems vol. 15 (eds. Becker, S., Thrun, S. & Obermayer, K.) 27–34 (MIT Press, Cambridge, Massachusetts, 2002).

  92. 92

    Cole, K.J. & Abbs, J.H. Kinematic and electromyographic responses to perturbation of a rapid grasp. J. Neurophysiol. 57, 1498–1510 (1987).

  93. 93

    Gracco, V.L. & Abbs, J.H. Dynamic control of the perioral system during speech: kinematic analyses of autogenic and nonautogenic sensorimotor processes. J. Neurophysiol. 54, 418–432 (1985).

  94. 94

    Robertson, E.M. & Miall, R.C. Multi-joint limbs permit a flexible response to unpredictable events. Exp. Brain Res. 117, 148–152 (1997).

  95. 95

    Scholz, J.P. & Schoner, G. The uncontrolled manifold concept: identifying control variables for a functional task. Exp. Brain Res. 126, 289–306 (1999).

  96. 96

    Scholz, J.P., Schoner, G. & Latash, M.L. Identifying the control structure of multijoint coordination during pistol shooting. Exp. Brain Res. 135, 382–404 (2000).

  97. 97

    Li, Z.M., Latash, M.L. & Zatsiorsky, V.M. Force sharing among fingers as a model of the redundancy problem. Exp. Brain Res. 119, 276–286 (1998).

  98. 98

    Gordon, J., Ghilardi, M.F., Cooper, S. & Ghez, C. Accuracy of planar reaching movements. II. Systematic extent errors resulting from inertial anisotropy. Exp. Brain Res. 99, 112–130 (1994).

  99. 99

    Messier, J. & Kalaska, J.F. Comparison of variability of initial kinematics and endpoints of reaching movements. Exp. Brain Res. 125, 139–152 (1999).

  100. 100

    D'Avella, A., Saltiel, P. & Bizzi, E. Combinations of muscle synergies in the construction of a natural motor behavior. Nat. Neurosci. 6, 300–308 (2003).

  101. 101

    Ivanenko, Y.P. et al. Temporal components of the motor patterns expressed by the human spinal cord reflect foot kinematics. J. Neurophysiol. 90, 3555–3565 (2003).

  102. 102

    Latash, M.L. in Motor Control, Today and Tomorrow (eds. Gantchev, G., Mori, S. & Massion, J.) 181–196 (Academic Publishing House “Prof. M. Drinov”, Sofia, 1999).

  103. 103

    Nicols, T.R. & Houk, J.C. Improvement in linearity and regulations of stiffness that result from actions of stretch reflex. J. Neurophysiol. 39, 119–142 (1976).

  104. 104

    Bernstein, N.I. On the Construction of Movements (Medgiz, Moscow, 1947).

  105. 105

    Bernstein, N.I. in Dexterity and its Development (eds. Latash, M.L. & Turvey, M.) 97–170 (Lawrence Erlbaum, Mahwah, New Jersey, 1996).

  106. 106

    Cisek, P., Grossberg, S. & Bullock, D. A cortico-spinal model of reaching and proprioception under multiple task constraints. J. Cogn. Neurosci. 10, 425–444 (1998).

  107. 107

    Barlow, H. in Sensory Communication (ed. Rosenblith, W.) 217–234 (MIT Press, Cambridge, Massachusetts, 1961).

  108. 108

    Olshausen, B.A. & Field, D.J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996).

  109. 109

    Lewicki, M.S. Efficient coding of natural sounds. Nat. Neurosci. 5, 356–363 (2002).

  110. 110

    Khatib, O. A unified approach to motion and force control of robotic manipulators: The operational space formulation. IEEE J. Robotics Automation RA-3, 43–53 (1987).

  111. 111

    Pratt, J., Chew, C.M., Torres, A., Dilworth, P. & Pratt, G. Virtual model control: An intuitive approach for bipedal locomotion. Intl. J. Robotics Res. 20, 129–143 (2001).

  112. 112

    Loeb, G.E., Peck, R.A., Moore, W.H. & Hook, K. BION system for distributed neural prosthetic interfaces. Med. Eng. Physics 23, 9–18 (2001).

  113. 113

    Hinton, G.E. Parallel computations for controlling an arm. J. Motor Behav. 16, 171–194 (1984).

  114. 114

    Pellionisz, A. Coordination: a vector-matrix description of transformations of overcomplete CNS coordinates and a tensorial solution using the Moore-Penrose generalized inverse. J. Theoret. Biol. 110, 353–375 (1984).

  115. 115

    Bullock, D. & Grossberg, S. Neural dynamics of planned arm movements: emergent invariants and speed-accuracy properties during trajectory formation. Psychol. Rev. 95, 49–90 (1988).

  116. 116

    Torres, E.B. & Zipser, D. Reaching to grasp with a multi-jointed arm. I. Computational model. J. Neurophysiol. 88, 2355–2367 (2002).

  117. 117

    Camacho, E.F. & Bordons, C. Model Predictive Control (Springer, London, 1999).

  118. 118

    Bizzi, E., Accornero, N., Chapple, W. & Hogan, N. Posture control and trajectory formation during arm movement. J. Neurosci. 4, 2738–2744 (1984).

  119. 119

    Todorov, E. Direct cortical control of muscle activation in voluntary arm movements: a model. Nat. Neurosci. 3, 391–398 (2000).

Download references

Acknowledgements

We thank G. Loeb and J. Triesch for their comments on the manuscript. This work was supported by US National Institutes of Health grant R01-NS045915.

Author information

Ethics declarations

Competing interests

The author declares no competing financial interests.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Todorov, E. Optimality principles in sensorimotor control. Nat Neurosci 7, 907–915 (2004). https://doi.org/10.1038/nn1309

Download citation

Further reading