Eight years ago, when Erik Ramsey was 16, a car accident triggered a brain stem stroke that left him paralyzed. Though fully conscious, Ramsey was completely paralyzed, essentially “locked in,” unable to move or talk. He could communicate only by moving his eyes up or down, thereby answering questions with a yes or a no.
Ramsey’s doctors recommended sending him to a nursing facility. Instead his parents brought him home. In 2004 they met neurologist Philip R. Kennedy, chief scientist at Neural Signals in Duluth, Ga. He offered Ramsey the chance to take part in an unusual experiment. Surgeons would implant a high-tech device called a neural prosthesis into Ramsey’s brain, enabling him to communicate his thoughts to a computer that would translate them into spoken words.
Today Ramsey sports a small metal electrode in his brain. Its thin wires penetrate a fraction of an inch into his motor cortex, the part of the brain that controls movement, including the motion of his vocal muscles. When Ramsey thinks of saying a sound, the implant captures the electrical firing of nearby neurons and transmits their impulses to a computer, which decodes them and produces the sounds. So far Ramsey can only say a few simple vowels, but Kennedy believes that he will recover his full range of speech by 2010.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Ramsey’s neural prosthesis ranks among the most sophisticated implanted devices that translate thoughts into actions. Such systems listen to the brain’s instructions for movement—even when actual movement is no longer possible—and decode the signals for use in operating a computer or moving a robot. The technology needed for such implants, including powerful microprocessors, improved filters and longer-lasting batteries, has advanced rapidly in the past few years. Funding for such projects has also grown. The U.S. Department of Defense, for example, sponsors research in prosthetics for wounded war veterans.
Only nine people, Ramsey included, have received brain-implanted prostheses. In the past, patients have used them to spell words on a computer, pilot a wheelchair or flex a mechanical hand. Monkeys have employed them to perform more complex tasks such as maneuvering mechanical arms to grab food or controlling a walking robot on a treadmill [see “Chips in Your Head,” by Frank W. Ohl and Henning Scheich; Scientific American Mind, April/May 2007]. Other experimental brain-computer interfaces read the brain’s output noninvasively, through electrodes attached to the human scalp [see “Thinking Out Loud,” by Nicola Neumann and Niels Birbaumer; Scientific American Mind, December 2004].
The technology promises to give thousands of victims of stroke, spinal cord injury and paralyzing illnesses the ability to, say, talk with a friend, flip through television channels or transport themselves by driving their own wheelchair. One day implants may enable paralyzed people to move robotic arms or even bypass damaged parts of the nervous system to reanimate unresponsive limbs. In the meantime, the quest to develop implanted neural prostheses is bringing with it revelations about how the brain manages motion and how it can remodel itself so that only a few neurons are needed to direct action through an implant.
Eavesdropping
Scientists have known for more than 220 years that electricity somehow controls muscle movement. In 1783 Italian physician Luigi Galvani, a contemporary of Benjamin Franklin, discovered that electric currents caused a severed pair of frog legs to twitch. By the 1860s German military doctors had discovered that small electric currents applied to the brain could cause certain muscles to contract. Over the following decades, dedicated researchers mapped which regions of the motor cortex control which groups of muscles in the body. But to discover how the brain actually orchestrates movement, scientists had to find a way to eavesdrop on the neural signals in the motor cortex while animals were awake and moving.
This task proved problematic until investigators figured out how to stably affix an electrode, a tiny sliver of conductive wire, to a neuron so they could register its weak, milliseconds-long pulses. When animals move, their brains shift slightly within their skulls, and the motions can rip an electrode from its anchor in the brain. In the late 1950s neurologists found that flooding the space between the skull and the brain with inert wax or neutral oil buffered the brain the way Styrofoam peanuts keep a box from moving inside a larger package. The buffer prevented a brain from shaking off its implant.
Despite this fix, no one could make sense at first of the chatter of individual neurons in the motor cortex. Researchers expected a one-to-one correspondence between the neurons that fired and the muscles that contracted during movements. But when they looked at individual neurons, they found the neurons would fire when a monkey moved its arm forward or backward or even when it kept the arm still.
In the late 1970s neurologist Apostolos Georgopoulos, now at the U.S. Department of Veterans Affairs and the University of Minnesota, had a brainstorm. The spinal cord exerts direct control over muscles, Georgopoulos realized. Thus, he supposed that the motor cortex might be directing movement at a somewhat higher level, specifying a trajectory rather than the muscles and joints needed to accomplish a movement.
To test his idea, Georgopoulos developed something called the center-out task, in which monkeys learn to move a joystick toward one of six targets arrayed in a semicircle. “Until then, all the research designs focused on very simple movements—forward, stop, back,” he explains. “In our experiment, the monkey was changing the position of its shoulder, elbow and wrist simultaneously.”
No one had looked at such complex motions before—or analyzed the data the way Georgopoulos and his colleagues did. Instead of trying to correlate the firing of particular neurons with the contractions of certain muscles, he averaged the responses of small groups of neurons over thousands of experiments. From that average, he saw through the noise that neurons produce when they direct motion, engage in other tasks or just fire spuriously. Although individual neurons fired with every movement, each neuron had a preferred direction: when the monkey moved the joystick that way, its firing frequency peaked. Neighboring neurons with similar preferred directions also became more excited. The closer a monkey’s arm moved to a neuron’s preferred direction, the more rapidly it fired; the farther away the arm moved, the more slowly it fired.
“It’s a sort of democracy,” Georgopoulos explains. “A given cell will keep voting on the direction of the movement, whether it’s in the majority or the minority, but the majority always rules. And the majority vote is an excellent predictor of direction.” In this way, the motor cortex sets a strategy for a movement. It calculates the direction (and, as Georgopoulos and others later found, the acceleration) needed for the hand to reach a target. It then sends the information to the spinal cord, which implements that strategy by operating muscles. Those more general commands from the brain, researchers believed, might indeed be useful for controlling external devices.
Making a Move
But progress on developing a neural prosthesis that could translate thoughts into action was slow. At first the electrodes were unreliable, and the electrical connections were sometimes finicky. The neurons themselves would also act unpredictably.
“Brain cells don’t behave the same way every time. Perhaps the cells are changing, or maybe the patient is tense or tired,” says Brown University neuroscientist John Donoghue, the second scientist after Kennedy to develop a neural prosthesis for human implantation.
Researchers also despaired at the problem of gleaning useful information from a relatively small number of neurons. “Usually the brain uses millions of neurons to perform a motor task. Now we’re asking people with prostheses attached to maybe 50 neurons to do the same thing,” Donoghue says. Yet those few neurons proved surprisingly capable.
Implant pioneer Eberhard Fetz, a biophysicist at the University of Washington, recalls experiments conducted in the late 1970s and early 1980s in which a monkey learned to use an implant to move the dial on an electrical meter to receive a drop of applesauce. Fetz and his team did not train the monkey, but it quickly learned to control the needle by trial and error, just by thinking. “He learned that there was something he could do to drive the meter to the right and trigger the feeder,” Fetz recalls. “Once he got the hang of it, he could do it every time.”
Neuroscientists believe that once the monkey chanced on a successful pattern of neural impulses, continued successes triggered the rewiring of its brain to create a faster and more efficient mechanism for repeating that pattern. This process also underpins other types of motor learning, such as that required to manipulate a fork or chopsticks. That is, the monkey learned to work the dial as if it were an extension of the monkey’s own body—which, in many ways, it was.
The ability of the brain to rewire itself on the fly is called plasticity. Investigators see examples of it all the time. In 2002 neurobiologist Andrew Schwartz of the University of Pittsburgh and his colleagues reported brain plasticity in a monkey that was trained to hit a target in a 3-D virtual-reality game using a ball that it controlled with its thoughts. Once the monkey learned to hit the target every time, Schwartz altered the settings so that the ball veered a few degrees to the right. Within about five minutes the monkey had adapted to the adjustment and began hitting the target again. “The only way the monkey could correct the error was by changing the firing of the neurons that we were recording,” Schwartz explains.
This past June, Schwartz’s team reported teaching a monkey to manipulate a gripper
on a hinged double-jointed robotic arm to lift food off a hook. Ordinarily the brain uses millions of neurons to control such a multipart, intricate movement. The monkey learned to retrieve the food, at least some of the time, with an implant that read the signals from only a few dozen neurons.
Connecting with People
With time, researchers parlayed their monkey studies into pilot trials with paralyzed people. Early implants generally enabled patients to translate their thoughts into simple actions, such as moving a computer cursor in one or two dimensions rather than using the complex, three-dimensional actions of a robotic arm.
In 1996, for example, a group of surgeons working under Kennedy inserted the first neural prosthesis into the brain of a paralyzed former teacher and artist in the terminal stages of amyotrophic lateral sclerosis, a progressive paralysis also known as Lou Gehrig’s disease. In the two months after the surgery, the woman learned to use it to turn on and off lights on a computer screen. A few years later a second patient, a locked-in 53-year-old former drywall contractor named Johnny Ray, learned to use the implant to move a cursor to pick out computer icons, spell words and generate musical tones.
Since then, seven more patients have received implants. With each one, the technology became more versatile and reliable. The surgical procedures, too, have come a long way since experimenters had to stabilize electrodes with wax. Kennedy, for example, has developed a cone-shaped electrode that contains chemicals to encourage neuron growth. Surgeons make a small hole in the skull above the ear and over the motor cortex and secure the electrode to the bone. When nearby neurons grow into the cone, they begin transmitting electrical signals to the electrode, which transmits them to a wireless receiver attached to the top of the head.
Researchers have also tried to improve the fidelity of the signals they receive by tapping more neurons. Donoghue and his colleagues developed an electrode array capable of receiving signals from 96 individual neurons. In 2004 neurosurgeons implanted it into the brain of 24-year-old Matthew Nagle of Weymouth, Mass., who was paralyzed when he intervened in a fight and was knifed through the spinal cord. Within only minutes of calibrating the prosthesis, Nagle could move a cursor on a computer. Over the next three years, before he died from an unrelated infection, he learned to control a television, check e-mail, and open and close an artificial hand. He made some rudimentary attempts to draw, which requires fine-motor control. His first attempt to sketch a circle wandered all over the screen, his second try led to more pronounced curves and his third produced an oval.
As investigators accumulate experience with human prostheses, they have raised their sights. Donoghue, for example, is teaming up with biomedical engineer Hunter Peckham of Case Western Reserve University, who has developed an electrical device that stimulates nerves or muscles to enable some movement after a partial or lower-level spinal cord injury. But Peckham’s system alone allows only simple, preprogrammed motions, such as boosting a person from a wheelchair to a walker. By linking a neural prosthesis to the device, however, Donoghue and Peckham hope to create a system that gives users greater flexibility. “Our goal is that within five years we will have a brain-controlled system that lets a tetraplegic take a glass of water, lift it and bring it to the mouth,” Donoghue says.
Fetz hopes to eventually connect a brain prosthesis directly to the spinal cord to flexibly reanimate nerves and muscles after spinal cord injuries. Such a device would tap the cord’s natural ability to coordinate groups of muscles.
Neurologist Richard A. Andersen of the California Institute of Technology is taking a different tack. Instead of decoding the motor cortex, he wants to capture the brain’s intentions before they become motor commands. Andersen believes those commands originate in the posterior parietal cortex (PPC), an area near the top of the back of the head that transforms sensory stimuli into a movement blueprint. Unlike the motor cortex, which estimates the trajectory an arm must take to reach a target, neurons in the PPC produce “goal” signals that specify the target itself. Recently Andersen and his colleagues at the Massachusetts Institute of Technology and McGill University showed that the PPC also predicts and adjusts for changes in a target’s motion.
The PPC’s focus on the goal makes tapping it potentially more efficient than reading a brain area that plots trajectories, Andersen says. A prosthesis implanted in the PPC might enable a patient to rapidly pick out letters on a screen to spell out words—just as fast-touch typists do on a keyboard. Because of its flexibility, such a prosthesis might let a user operate a wider range of devices than a motor cortex implant designed to control specific movements would. Andersen is hoping to embed the appropriate electronics into a person’s parietal cortex within a year or two.
Finding a Voice
Kennedy’s speech prosthesis arguably poses the greatest challenge yet because he had almost no experimental data on which to base its operation. After all, monkeys do not speak, and Ramsey is the first person to receive an implant to produce speech. This means that Kennedy must find a way to separate speech signals from neural noise without animal research to guide him.
Ramsey’s implant connects with about 50 neurons in the part of his motor cortex that translates how he thinks a syllable should sound into the muscle commands to make the syllable. The implant captures the signals that control the coordinated motion of his mouth, lips and tongue to form sounds.
The link between Ramsey’s neural implant and speech is a sophisticated computer program called Directions into Velocities of Articulators (DIVA), developed by Frank H. Guenther, a cognitive neuroscientist at Boston University. DIVA is a mathematical description of how the brain controls speech, parsing the process into eight parts that represent different speech functions in the brain. Mathematical formulas define neural firing rates in each area and neuronal connections among areas. DIVA made it possible to build a neural decoder that can decipher the speech signals amid the neural noise coming out of Ramsey’s implant. The decoder translates the speech signals into sound data that it sends to a speech synthesizer, which generates human sounds.
Guenther built DIVA by scouring the research literature on the brain’s speech centers. His group continually refines the program through additional experiments. “If we want to investigate how the brain corrects speech, we’ll perturb a volunteer’s speech. They may say ‘bet,’ but they hear ‘bit.’ Our model might predict that four parts of the brain should light up when they hear the perturbed sound, and we’ll see how that compares with what happens on a [brain] image. If the image lights up in five places, then we update the model to reflect this new information.”
DIVA learns to speak from experience. Initially DIVA babbles like a human infant. As it “listens” to the resulting sounds and “senses” the position of its virtual muscles, it uses the feedback to modify its mathematical relationships to speak more clearly. “Then comes the imitation stage,” Guenther says. “We have a human say something, and the model tries to reproduce it. It will be wrong at first, but DIVA will use feedback to keep getting it closer. It usually takes about five or six attempts to get it right.”
Similarly, the neural decoder based on DIVA does not accurately translate Ramsey’s initial attempts to speak, in part because the computer program receives input from just a tiny fraction of the millions of neurons that are involved in speech. The program and Ramsey, however, get better with practice. Guenther starts this learning process by playing a sequence of vowel sounds on a computer—vowels are easier to pronounce than consonants—and Ramsey sings along in his mind. Ramsey and the decoder botched their first five attempts at each of the first three vowels. But then Ramsey adjusted his brain signals based on the feedback from the synthetic sounds the computer produced, and on the next five, he got three or more right.
“Ramsey was able to quickly improve his performance by adjusting the brain signals that were sent to the synthesis system,” Guenther recalls. “Most of this learning is subconscious motor learning, like learning to shoot baskets or whistle or ride a bike, rather than requiring a conscious attempt to change the way one communicates.” It is slow, arduous work. Ramsey has only enough energy for two or three weekly sessions that usually last no more than an hour or two.
Eventually Kennedy hopes to implant more electrodes in different parts of the brain’s speech motor region to provide richer neural input for the speech program. “We’d like to have several electrodes spread out over areas that control the tongue, mouth, jaw and facial muscles. If we had more implants, that would give us even better resolution.”
From such endeavors, the neurologist hopes to change the lives of tens of thousands of people. Those who are now entombed within their own bodies will once again be able to communicate and connect with friends, caretakers and family. People who cannot move from room to room or change a television on their own will find a new freedom. Wounded warriors returning from battle may receive artificial limbs that respond to their unspoken commands.
Erik Ramsey is just the beginning.
Note: This article was originally printed with the title, "Putting Thoughts into Action".