Brain-reading implants turn thoughts into speech

Benjamin Thompson

Welcome back to the Nature Podcast this week how brain-computer interfaces are enabling people to communicate…

Nick Petrić Howe

…and how hot can it get before tropical leaves die. I'm Nick Petrić Howe…

Benjamin Thompson

…and I'm Benjamin Thompson.

<Music>

Benjamin Thompson

Brain-computer interfaces are devices which, as the name implies, link brains and computers. They're capable of detecting and deciphering brain activity, and translating that activity into things like text or movement. BCIs, as they're known, have been used by people with neurological disorders, perhaps using the systems to move a cursor on a screen say. This week, there are two papers out in Nature from two different teams showing how these BCIs could help people who have lost the ability to speak in a faster and more accurate way than has been previously possible. Here's Edward Chang from the University of California, San Francisco, who's part of one of the teams.

Edward Chang

So there are a series of neurological conditions like ALS or brainstem stroke, that can be very devastating, where people lose the ability to communicate not only the ability to speak, but also they lose ability community because they can't write as well. So, the paralysis itself is devastating because of the mobility. But in terms of people's ability to communicate with others, which is just really a fundamental part of being human, when that becomes compromised or impaired, it becomes a big issue.

Benjamin Thompson

In their paper, Edward and his colleagues designed a BCI that converted brain activity into different speech-related outputs. They tested their system in a pilot study with the help of a participant who had had a type of stroke 18 years ago, and who was unable to communicate either by speech or typing due to paralysis and muscle weakness in her face, arms and hands.

Edward Chang

She had a surgery here at UCSF, that went very well, where we implanted an electrode array with 250 sensors over the part of the brain that's called the motor cortex. And part of this motor cortex is the commands that control the vocal tract, the lips, the jaw of the tongue. And these sensors pick up her intended speech and vocalizations and transmits those signals to a computer.

Benjamin Thompson

Rather than attempting to read the participants mind and work out what she's thinking about saying, the interface actually works by picking up the intent of what she's trying to say. Because despite having limited movement of her facial muscles, when she silently tries to say the words, the parts of her brain associated with moving the jaw, or the tongue, for example, will still light up. And this activity can be recorded and deciphered by a computer.

Edward Chang

So basically, about three weeks after the surgery, we essentially plugged into the computer and started training. And what training involves is the participant looking at a screen and seeing a sentence. And then she's instructed to basically try to say that sentence that may or may not be associated with some small movements of her mouth. But the key thing is that there's some intention to try to say those words. And then we do this over hours, actually, and days, we collected about 10 to 14 days of data.

Benjamin Thompson

This training helped the computer learn what brain activity was associated with the sounds that made up each of the words. And this data was used in different ways when it was put to the test on a separate group of sentences.

Edward Chang

The speech signals are from one source. And then output is in three different forms. One is text. The second is speech synthesis, audible sounds that you can hear. And the third is basically a talking face like an avatar that's animated with actual facial movements.

Benjamin Thompson

For Edward, this approach was important.

Edward Chang

I think one of the key things to take away from this is that speech is multimodal in the sense that there's a sound associated with the words that are coming out of my mouth. But for us who are on Zoom, you can actually see the movements of my lips and that's why we value in person face-to-face interactions because there's so much that is there with our expression. And one of the things that I think that was really cool about what we're able to do with this is personalize that experience so our participant could choose from over 100 different avatar forms, choose the hair colour, you know, essentially personalize it to whatever she really preferred.

Benjamin Thompson

To enable the participants are really personalize the experience, in one part of the study, the team used audio from her wedding video to make a synthetic version of her own voice. And here's a little bit of what that sounds like.

Synthesised Voice

I think you are wonderful. Give me a few minutes. What do you think of my artificial voice?

Edward Chang

It was the first time in 18 years that she heard a voice that was generated from her own thoughts. And what she was trying to say, the device that she currently uses for typing things out that outputs sound, we call these assistive augmentative devices. Similar to Stephen Hawking’s. It’s pre-configured, for example, to a British voice. So for last, you know, 18 years, a lot of people actually thought that she had a British accent. But she's Canadian actually has, you know, more of a Canadian accent. So, it is important, it's part of who we are.

Benjamin Thompson

The assistive device that the participant normally uses, allows her to communicate at about 14 words per minute. But in this new paper, the team reported their system achieved an average of 78 words per minute for real-time brain-to-text translation. But it's not perfect. Despite using something called a language model to predict and correct the text, it had an error rate of 25%, when tested with a vocabulary of just over 1,000 words, the brain-to-voice system had an error rate of just over 28% When tested using a 372 word vocabulary. But as I alluded to at the start, this isn't the only approach to making a BCI to help people communicate. In the second BCI paper in this week's Nature, another team has shown a system that can work with a much larger vocabulary than Edward and his team tested. To achieve this, the second team used a quite different approach. Instead of measuring the activity across brain areas, this paper looks at the activity of single neurons, as Frank Willett, one of the authors of the second paper explains.

Frank Willett

So for this study, we placed four microelectrode arrays into speech-related areas of cortex. So these are tiny arrays 3.2 by 3.2 millimetres that are placed on the surface of the brain. And they penetrate about one-and-a-half millimetres into the cortex, so that it can record actual activity from single neurons. And then we take that activity and basically interpret the patterns of firing of these neurons. And from that, we're able to decipher what sounds the person is trying to make at each moment in time.

Benjamin Thompson

These tiny electrodes measure the activity of a handful of neurons out of the billions in the human brain. In a similar way to the other paper, this team trained their system to interpret the intent to make the specific sounds that make up words, in this case, with the help of a participant with amyotrophic lateral sclerosis, or ALS, and the team were quite impressed with the results.

Frank Willett

So the exciting result is that we were able to do a 23% word-error rate on a very large vocabulary. So 125,000 words, and these were tested on sentences of just general English, so we weren't constraining it to a small set of words and suggests that, okay, this is a real thing that maybe you know, as we start to refine this, we can really get it to a level where somebody could say anything they want to and they could be accurate enough to restore conversation. But you know, we're not there yet. You know, one out of every four words being wrong is still quite a lot of words being wrong.

Benjamin Thompson

At 62 words a minute, the system is a marked improvement on previous BCIs. Not quite as fast as Edward and his team's BCI, but working with a bigger vocabulary. So, we have two papers using different braincomputer interfaces to accomplish a similar goal. Nick Ramsey from University Medical Center Utrecht in the Netherlands has co-written a News and Views article about them in Nature. He says that these two methods differ in their approach, but were impressive in their outputs.

Nick Ramsey

They both show for the first time that they can get a very high level of performance by isolating attempted speech into a computer voice or into written language. And what distinguishes these from previous research is the number of words and the speed at which people can express words. And what I think is really, the most important part is that we're getting closer to a level of performance that is very interesting for people who are paralyzed, basically we’re aiming for is that people can speak again, but then through a computer.

Benjamin Thompson

Typical human speech is somewhere around the 150 words per minute mark, and the errors can hinder understandability, so there's still a way to go yet. Nick says that although these papers will hopefully spur academics and companies to develop improved BCIs he thinks there's a lot that needs to be considered. For example, any implants will need to be robust and long lasting and will need to be designed in a way that makes them useful and straightforward for users and caregivers to operate. Currently, there's a lot of equipment and academic expertise required to make these systems work. It also remains to be seen who might be able to use BCI is like these, the two people involved in these papers had limited facial movements. But that's not the case for everyone.

Nick Ramsey

When you're able to move your muscles a little bit, you actually generate information in your sensory part of the cortex, which contributes to the decoding that they show in these studies. For the people who are locked in there is no movement at all, so we don't know whether the lack of that kind of sensory feedback during speech while you’re controlling the system is going to degrade performance a lot. Those are unknown things.

Benjamin Thompson

Frank and Edward both acknowledge that this is something of an unknown, and that there's more work to be done to make BCIs that are accurate and reliable. Both say that individual work represents a proof of principle, and that, in time, brain-computer interface technology could make a huge difference to the people that need them. Here's Edward.

Edward Chang

As a physician, a neurosurgeon, who sees so many people suffering from the disconnection basically from society and from friends and family, the frustration, the inability to communicate. I think it's absolutely going to be transformative, and we need to work faster in order to make this a reality soon.

Benjamin Thompson

That was Edward Chang from the University of California, San Francisco in the US. You also heard from Frank Willett from Stanford University, also in the US, and Nick Ramsey from University Medical Center Utrecht in the Netherlands. For more on that story, check out the links in the show notes and be sure to look out for a video about it too, coming out later this week.

Nick Petrić Howe

Coming up, researchers have been working out what a warmer future may mean for tropical forests. Right now though, it's time for the Research Highlights, with Shamini Bundell.

Shamini Bundell

To achieve the fastest marathon ever, researchers have turned to wind tunnels. 2019 sort of the first ever sub two-hour marathon time, it was smashed by Kenyan runner Eliud Kipchoge, a feat made possible in part by drafting a strategy to reduce air resistance by positioning other athletes called pacers around the main runner. Now, to identify even faster drafting formations, researchers have positioned mannequins in a wind tunnel. They studied the aerodynamics of their mannequins while surrounded by pacers in a variety of formations. They identified three swordfish-shaped arrangements of paces that lowered the air resistance of the designated runners by 60%, compared to a solo runner. They estimate that this improvement could shave about four minutes off a marathon time. Try not to get blown away by that research in Proceedings of the Royal Society A.

Shamini Bundell

Some birds are famous for their flashy colouring, others for being so called ‘LBBs’ or ‘little brown birds’, a whole host of dully coloured and hard to distinguish species. But why do these differences exist? Researchers looked at bird illustrations from the Handbook of Birds of the World, analysing each pixel in the guidebooks 10,000 bird paintings to determine the proportion of each bird covered by one of 12 categories of colour. They then grouped the species by habitat, diet, and other factors that could influence coloration. They found that the most common colours were black, white, grey and brown. But birds that had evolved to impress mates with their plumage, as well as those that face less pressure from predators, were more likely to have flashier colours such as blue, purple, and red. While most of these results support existing theories, the researchers say they still don't fully understand the factors that influence bird coloration. Grab your binoculars and spot that research in the Proceedings of the National Academy of Sciences of the United States of America.

Nick Petrić Howe

As the climate warms, a key question to understand is how hot can it get before tropical forests run into trouble? Understanding this is important due to these forests’ role in carbon storage and as homes to masses of biodiversity. Now, it's already known that climate change will make droughts and fires more likely in these forests. But what about the direct impact of heat itself on the trees? Get hot enough, and the process of photosynthesis itself starts to break down, the leaves die, and if enough of them die, then likely the trees will follow suit. But how often are tropical trees likely to get to this temperature? And how much warming of the world could the trees take before they cross this threshold? Well, that's what a new paper in Nature is trying to figure out by combining several types of data. One of these was a recent bit of research on the maximum temperature that tropical leaves could withstand before the enzymes underlying photosynthesis stopped working. Here's Chris Doughty, one of the authors of the new paper to explain.

Chris Doughty

We basically looked at you know, what point to enzymes actually start to denature in these leaves, and it's actually surprisingly consistent across tropical forests around 46-47 °C. So it's a rigorous, solid way of understanding at what point these trees, specifically the leaves, but also the trees are in real trouble.

Nick Petrić Howe

Now, the fact that there is a temperature at which leaves will die has been known for a very long time, but only recently has enough data being collected from tropical forests to determine what this number is in this ecosystem. And as Chris says, it turns out, it's a remarkably consistent 46.7 °C across tropical species. Now, this number is a good starting point, if you want to model how bad things might get for tropical forests. But Chris also needed some on the ground measurements, or rather, in the tree measurements, which meant doing a bit of climbing.

Chris Doughty

I mean, in some sense, it sounds very simple and basic, you know, just get a thermometer out there. But actually, you know, when you think about actually the logistics of getting up to the top of the canopy, because those are the hot leaves, right, right at the very tippy-top of the canopy. And there's not many places in the tropics where you can actually have access to these. But we started doing the hard work of climbing up into these trees and you know, measuring individual leaf temperatures. And the reason we wanted to do this was if you look at the temperature any other way, you kind of get averages, and the problem with averages is it's not giving you the distribution.

Nick Petrić Howe

By getting to the tippy-top of the canopy and measuring the leaves directly Chris and his colleagues could accurately assess the temperatures the leaves were facing day to day. And whilst on average, the leaves weren't that close to that 46.7 ° threshold, across time, they found that some of the leaves were getting really very hot. In fact, a small proportion, were even getting over that 47 ° threshold already. But of course, these measurements were only in a few locations, it would be incredibly difficult to do such measures for every tree, in every tropical forests, across the whole world.

Chris Doughty

So we needed some way to scale this up. And the way we did that was with this new NASA satellite called ECOSTRESS.

Nick Petrić Howe

ECOSTRESS, which of course stands for ECOsystem Spaceborne Thermal Radiometer on Space Station is a device mounted on the International Space Station that measures the temperature of plants at incredible resolution. And so, Chris could use it to see how close tropical forests were getting to this upper temperature threshold worldwide. And it wasn't just isolated leaves that were feeling the heat.

Chris Doughty

What we found was individual canopies as well, we're getting up into the 40s. So much hotter than what was previously thought.

Nick Petrić Howe

So many leaves are already getting close to this upper threshold globally. But what might happen when the planet gets even hotter in the future. To figure that out, the team looked at some experiments where tropical forests have been artificially warmed.

Chris Doughty

We actually aggregated datasets where people have done warming experiments. They raise their temperatures by 2 °C, 3 °C, and 4 °C. So kind of the range of possible climate change scenarios. But even though we are only warming by 2, 3, 4 degrees, the actual upper temperature of the leaves raised by about 8 °C.

Nick Petrić Howe

So when air temperature is increased by 2, 3 or 4 degrees, Chris and the team found that the leaves were actually getting much hotter than that. This discrepancy in temperatures could spell disaster in a warmer future, with many more leaves reaching that upper temperature threshold. So using this data, the satellite data and the temperature data from the tree canopies, Chris and the team were able to build a model to answer the question ‘how hot would the world needs to get before leaf temperatures start to cross that upper temperature threshold and tropical forests start to get into real trouble?’. Their answer was 3.9 °C plus or minus half a degree which is within the worst-case scenarios for climate predictions. Now, there are some uncertainties in that, such as it being unclear how much forests can adapt as temperatures increase. But in general, the analysis suggests that this would be the point well be bad news for tropical forests.

Chris Doughty

If you take the ‘do-nothing’ scenario, so these are these upper temperatures like we basically keep pumping carbon in the air without thinking about it, tropical forests are in trouble.

Nick Petrić Howe

In such a scenario, leaf temperatures would often cross that critical threshold, meaning many more would die likely causing many trees to die. And this, combined with increasing droughts, fires and logging, could put enormous stresses on these vital ecosystems. Chris though is optimistic. From his perspective, whilst the climate is warming action is being taken. And so he thinks that if we continue to fight climate change will give these vital tropical forests a fighting chance.

Chris Doughty

I would say climate change is an issue in tropical forests. But you know, I also don't want people to get depressed over this that, you know, it's not like it's doomed. We, you know, we're definitely within the scenarios where, you know, if we have some mitigation for climate change, like you know, things that we're already doing, basic climate mitigation, reducing deforestation, it's not going to get to these upper temperature thresholds. So, the broad message is climate change will negatively affect these forests. But you know, it's within our abilities to keep it under this high-temperature threshold.

Nick Petrić Howe

That was Chris Doughty, from Northern Arizona University in the US. For more on that story, check out the show notes for a link to the paper.

Benjamin Thompson

Finally, on the show, it's time for the Briefing Chat, where we discuss a couple of articles that have been featured in the Nature Briefing. And Nick, why don't you go first this week? What have you got?

Nick Petrić Howe

So this week, I've been reading an article in Nature, all about Ötzi the Iceman and about some new analyses that have been done on him.

Benjamin Thompson

Yeah, I remember when he was found it was in some ice in the Alps, if I remember correctly.

Nick Petrić Howe

Yeah, that's right. So he was found on the border between Italy and Austria, like slightly on the Italian side. And he was remarkable for being very well preserved. And so this has given researchers a window into ancient peoples from Europe, as he died around 5,300 years ago. And now researchers have done like a very intense sort of genetic analysis of him and found out some of the things that we thought about him may not have actually been the case, especially with regards to his appearance,

Benjamin Thompson

Right, so I mean, I remember again, the pictures of what he looked like then. So, kind of long, straggly hair, quite pale, I suppose. Is that not necessarily the case.

Nick Petrić Howe

I mean, you've hit the nail on the head with those particular two things, it seems like from this latest genetic analysis, that Ötzi was actually a lot more dark skinned than we thought. And he had the markers for male pattern baldness, and being around 45 years old, they reckon that he was probably quite bald. And this actually matches more closely with his remains, his remains kind of look bald and darker than, you know, this sort of paler version of him that we had thought of before. But obviously, 5,000 years is quite a long time. So it's hard to know exactly what people look like. But now this genetic analysis has sort of filled in some of those gaps.

Benjamin Thompson

Right and did it take some sort of leaps of technology or something like that to fill in the gaps? So I'm guessing, as you say, he was quite intensively studied when discovered.

Nick Petrić Howe

Yeah, so the reason that we thought that he was pale, and with this sort of long, straggly hair was because of a previous analysis in 2012. And this was a genetic analysis again. It was known to be quite incomplete, so there was more that could be done. And over the past, like decade or so, as we've talked about many times on the podcast, like genetic analyses have come along leaps and bounds. And so we've been able to fill in these sorts of gaps and understand his genetics a lot better. And one of the key things seems to have been that actually there was some contamination in the previous 2012 study, which now researchers have been able to get around in the new study. And that may have led us down some sort of false alleys.

Benjamin Thompson

And so this new study, then what does it reveal, maybe about sort of populations in this area around, what 5,000 years ago?

Nick Petrić Howe

Well, the previous analysis has suggested that Ötzi had sort of steppe ancestry, which is a group of people that were thought to have emigrated to Europe like 1,000 years after Ötzi died. So that has always been a bit of a mystery, like, why did he have this ancestry. But the new analysis shows that he may not have actually had that. Instead, most of his ancestry seems to have been from Anatolia, who was one of the Anatolian farmer peoples, which is that sort of region between the Mediterranean and the Black Sea. And so what had been thought is these Anatolian people had come into southern Europe and then mixed with the hunter-gatherers who already lived there. But actually for Ötzi case, he seems to have very little hunter-gatherer ancestry seems to mostly be this Anatolian thing. So his group of people, his lineage seems to be quite distinct from other Europeans at the time.

Benjamin Thompson

Well, that's absolutely fascinating. And what research is saying about this kind of updated take on who he was.

Nick Petrić Howe

Researchers are not necessarily surprised. But this is like a real welcome addition to like the sort of portrait that we have about this person in Europe. And we don't have many examples of people like Ötzi to study. So this is really useful in that, but it's interesting as well, because, you know, it's been 32 years since he was discovered, and people have been intensively studying him since but still, there seems to be more to be revealed. And I'm sure as we go into the future and technologies get better, and science improves, we'll be able to find out even more about this person's life.

Benjamin Thompson

Well, I'm very excited to see what else researchers can find out with Ötzi’s help. But for the time being, let's move on to my story. And it couldn't be more different to be honest with you. It is something I read about in Nature. And it happened over the weekend. And it is the latest failed attempt from a country to land a craft on the Moon. And in this case, it's Russia.

Nick Petrić Howe

Ah, so there's been quite a few attempts to land on the Moon. And is it fair to say that not that many have been very successful?

Benjamin Thompson

Yeah, landing on the Moon is really, really hard. And maybe we'll talk about that in a bit. But this mission was, I suppose, Russia's Luna 25 mission, and it took off on an uncrewed rocket launched from eastern Russia on the 10th of August, okay. And it would have been Russia's first Moon mission since 1976. And the aim was to land in this kind of 100-kilometer-wide Boguslawsky crater near the south pole of the Moon, which would have made Russia the first country to land in his area, and it's an area that's getting some attention at the moment. But as I say, things didn't necessarily go to plan.

Nick Petrić Howe

And what exactly went wrong here? Was this a crash? Or did they just lose contact with it?

Benjamin Thompson

Well, I think that remains to be seen. But what we do know is that on the 19th of August that’s Saturday, just gone, things went awry. So Russian space agency, Roscosmos announced on social media, that communication with Luna 25 was interrupted. And apparently this was after sending a command for the spacecraft to lower its orbit in preparation for landing. And some reports suggest that the engines didn't necessarily shut down properly, and then attempts to contact the spacecraft the next day were unsuccessful. And Roscosmos determined that Luna 25 had, “ceased to exist as a result of a collision with the lunar surface”. And there's a commission coming up to figure out exactly why it failed.

Nick Petrić Howe

I mean, in either case, it sounds quite dramatic. So what does this mean, now then? What was the sort of mission that it was trying to do and what might be, I guess, lost the science without it?

Benjamin Thompson

Yeah, so if Luna 25 had been successful, then it would have put an 800 kilogram lander down on the Moon, which would have used various instruments, including a robotic arm apparently, to dig up the lunar surface and look for water ice, which is thought to be abundance in this south pole region, and potentially could be used as a resource for future space missions, you know, splitting it into rocket fuel, stuff like that.

Nick Petrić Howe

So what does this then mean more broadly for missions to understand the Moon, or for Russia's missions to understand the Moon?

Benjamin Thompson

Well, it is obviously a setback for Russia's space program, which has struggled as a result of things like sanctions imposed after Russia's invasion of Ukraine and the ongoing war there as well. Now, Russia has another two missions to the Moon plans, and they called Luna 26, and Luna 27, one and orbiter and one a lander. But what this failure actually means for that program is it's kind of unclear at the moment. But what it does show I think, is how hard it is to land on the Moon, as you alluded to earlier. In recent years, there have been many failed attempts from different space agencies and private companies, often in this final landing phase, but it's not putting people off to the south pole of the Moon is a hot destination at the moment. So later this year, NASA is hoping to land a craft there. And China has its sights set there too, apparently, but a lot sooner than that. Today. In fact, India's Chandrayaan-3 mission is due to send a lander to the surface now at the time of recording that hasn't happened yet. But by the time you're hearing this, listeners will know whether it managed to sort of get through that dangerous final stage. And we know their previous mission Chandrayaan-2 suffered the same fate as some of these other missions by crashing then as well.

Nick Petrić Howe

Well, maybe we should end the Briefing Chat there then so listeners and us can go and check out what happened to Chandrayaan-3. But thanks, Ben for telling me about that. And listeners. For more on the stories of where you can sign up to get more like them directly to your inbox, check out the show notes for some links.

Benjamin Thompson

And that's all we've got time for this week. As always, though, don't forget, you can keep in touch with this on X, the social media platform formerly known as Twitter, or you can send an email to podcast@nature.com. I'm Benjamin Thompson.

Nick Petrić Howe

And I'm Nick Petrić Howe, thanks for listening.