This issue includes papers on three themes in AI with a foundation in neuroscience or cognitive science — reinforcement learning, perception and limb movements — while approaching each theme from a number of vantage points.

Credit: mrs/Getty Images

Reinforcement learning, which refers to learning to choose actions in a way that maximizes rewards and minimizes punishments or losses, is a much-studied process in neuroscience. In recent years, AI techniques based on reinforcement learning have led to exciting advances, such as a framework for vision-based grasping of objects that a robot has not seen before1. A Review Article by Emre Neftci and Bruno Averbeck describes reinforcement learning in both artificial and biological systems. They highlight, for instance, that reinforcement learning in artificial agents generally focuses on a single complex problem within a static environment. In biological agents, on the other hand, reinforcement learning is focused on simple learning problems embedded in flexible, dynamic environments, such as classical conditioning of animals in their habitats. Taking stock of different approaches and synergies, the authors promote the vision that an exchange of ideas between the fields would be mutually beneficial for both.

Perception is the study of how sensory information is used to form actionable impressions of reality. In machines, perception refers to experiences resulting from the stimulation of sensors, such as cameras for vision or microphones for audition. Applications of machine perception include some of the achievements of deep learning, such as facial recognition and object detection. In robotics, advances in deep learning for vision have enabled robots to perceive objects in their environments and identify targets of interest, which are necessary steps for goal-directed movements. For example, such approaches have been central to recent successes in the Amazon Robotics Challenge, in which research teams compete in building robots that can pick and stow items on shelves, a goal for robots in Amazon’s warehouses. In the Challenge Accepted in this issue, Juxi Leitner describes how one component of his team’s winning system in the 2017 competition was a perception system that segmented and identified the items in its environment, in conjunction with a vision-based grasp selection approach for manipulating items.

When a robot ‘sees’ an object — such as a book in a storage container in the Amazon Robotics Challenge — how is the object linked with the appropriate action? Does the robot’s software have to activate countless artificial neurons and transform the inputs via innumerable layers of a deep neural network, using data from an inestimable number of previous experiences of grasping a motley assortment of objects under diverse environmental conditions, just to pick up a book? A better understanding of how biological organisms seamlessly perform such vision-to-movement sequences would enable researchers to develop machines with such abilities. But how should we think about this problem? Does it require more sophisticated machine learning algorithms, better control systems for implementing robotic movements, or something else?

In a News Feature, Jeremy Hsu describes a concept developed by the psychologist James J. Gibson in the 1970s that is inspiring researchers to think about perception and action in new ways. Gibson argued that there is a reciprocity between an organism’s perception and its environment, such that an organism ‘directly perceives’ the action possibilities that are ‘afforded’ by objects. For example, imagine a cat approaching a table on which lies an item of interest, such as food. How does perception indicate to the cat that it can or cannot jump on the table? Must the cat’s nervous system perform physics calculations before each potential jump? Does the cat’s cognitive system perform a series of mental steps to make the jump? Or does the cat directly perceive that the table affords jumping, given the table in question, the particular cat’s body, and the context? Understanding the capacity to flexibly fuse perception and action in changing environments is a fundamental research topic in psychology and robotics.

Achieving complex limb movements is a goal in machine motion and manipulation, another main theme of AI. In an Article in this issue, Francisco Valero-Cuevas and colleagues develop a tendon-driven limb that could provide robots with the ability to adaptively move in everyday tasks. They take inspiration from biological examples of vertebrates learning movements by sparse trial and error. Their model-free approach for learning the generation of periodic movements in a tendon-driven robot suggests a mechanism by which motor habits may emerge, thereby establishing a relationship between the body and the environment. The authors present biologically inspired hardware and software, including reinforcement learning and an artificial neural network, to produce locomotion.

Taken together, the ideas and discoveries described in these papers integrate various themes in neuroscience, cognitive science, psychology and biomechanics, propelled by machine learning and other technology, to advance the field of machine intelligence.