AI-enhanced night-vision lets users see in the dark

Nick Petrić Howe

Welcome back to the Nature Podcast, this time: how to see in total darkness...

Shamini Bundell

...and a mysterious link between COVID and diabetes. I'm Shamini Bundell.

Nick Petrić Howe

and I'm Nick Petrić Howe.

<Music>

Nick Petrić Howe

In spy films, high tech missions often involve sneaking around in the dark with fancy night vision goggles. These might rely on enhancing visible and near visible light, or could be thermal images, detecting heat rather than light so as to work in the pitch darkness. But these thermal images are plagued by an issue called ghosting. And there'll be more on that in a minute. So, to create a better way of seeing in the dark, researchers have taken a new approach combining a collection of algorithms and detectors to create a system they call Heat Assisted Detection and Ranging or HADAR. HADAR uses physical information about objects in the environment, as well as their heat signatures to allow a user to see in total darkness, as if it was daytime. I reached out to Zubin Jacob, one of the team behind HADAR and started by asking him how he got interested in night vision.

Zubin Jacob

What really made me pursue these ideas is just simple things from science-fiction to home discussions, if you've seen Predator and Arnold Schwarzenegger in there, you see that the Predator kind of sees in the night, and that's why it's able to hunt its prey really well.

Nick Petrić Howe

And so one thing I've read about in terms of trying to understand, like, heat signatures to make night vision is a challenge that's called the ghosting effect. Can you explain what this is, and why this is a challenge for you?

Zubin Jacob

So, when you look at an image that is just generated by the heat signatures from the body, the entire environment around you is also emitting heat radiation. So that actually bounces off your body and reaches the camera. So, suppose your friend is taking a picture of you, you will appear almost as a ghost to them, because there's so much signal, and there is a lot of noise, that is just always present in thermal signatures. So that's what we really mean by ghosting, just the lack of texture, lack of contrast, and the lack of information inside an image.

Nick Petrić Howe

And in your paper, you've developed something called a HADAR system, which seems to get around this problem. Can you tell me a bit about this system and how it works?

Zubin Jacob

So, when we started analyzing the problem, we started going back into the attributes of thermal physics, which are essentially our simple understanding of what is hot, what is called the other aspect is emissivity. It's the property of the body to emit heat. And then finally, the most interesting thing is the texture. So, what we came up with are a bunch of algorithms. They are machine learning driven, but we embed the our understanding of thermal physics inside the algorithms. And then we also use some advanced cameras to actually put all the hardware and software together and extract optimal information from the thermal radiation even in pitch darkness.

Nick Petrić Howe

Right, right. So, they are trained, they're taught what different objects, what properties they have. And then they can interpret that information from the sort of standard thermal camera image.

Zubin Jacob

That's very close to how we approach the problem. There's one or two kind of subtle details here. So, the one thing is that our standard thermal cameras do not have the capability to extract optimal information from the thermal radiation field, you actually need something more than the standard thermal camera, and that's essentially spectral resolution. Think of it as a camera that gives you red, green, blue colours, but these colours are invisible to the human eye, they are in the infrared range of the spectrum that the human eye cannot see. So those colours are very crucial in our algorithm. And along with that algorithms are trained to first exploit spectral resolution and also exploit the thermal physics of the problem. So, we are baking in some of the physical understanding of how heat emanates, and heat propagates in the environment that is involved in the machine learning training algorithms.

Nick Petrić Howe

Right, right. And they're then able to sort of take all that information and spit out an image that is more useful and more like daytime vision, I guess?

Zubin Jacob

Exactly, exactly.

Nick Petrić Howe

And how well would you say this system performs?

Zubin Jacob

So, the interesting and intriguing thing we found was that if you try to quantify aspects of depth, how far is an object? And you try to ask that question in the daytime, and then you ask that question in the nighttime. What we found is that the accuracy to range an object in the daytime is the same as the accuracy to range the object in pitch darkness if you are using our HADAR algorithms.

Nick Petrić Howe

And this may sound like a naive question. But what if it's really cold? Does that just throw everything off? Like, does the environment have an influence on how well this performs?

Zubin Jacob

So, in the cold, there's definitely a decrease in the signal so to speak. But as long as your camera can be actually at a temperature, which is colder than the surroundings, you can still collect a good thermal image. So, it makes it more difficult and challenging. But at the same time, it is possible to overcome the limitations of even like extremely cold environments.

Nick Petrić Howe

And how would one go about sort of implementing this? Let's say I have some sort of thermal imaging camera. Am I able to just implement your system out of the box? Or how much would it require of me to sort of start using what you've developed?

Zubin Jacob

This will take a little bit of time and effort in the hardware development. The thermal imaging cameras that you can buy in the market today, unfortunately, they will not work with the algorithms that we have, we need to add in some spectral resolution or add some filters, to be able to get this technique to work. So, we are very eager to actually bring this technology to the market and the hands of users around the world. So, I think in say, three years time, we should be able to have various kind of vendors who are bringing together kind of thermal cameras, spectral filters and algorithms, and it's just kind of a packaged camera that you can use every single night.

Nick Petrić Howe

And you know, if these hurdles are overcome, what do you hope that this will be used for?

Zubin Jacob

You know, one of our first goals is being able to provide driver assist systems that can help you see, or sense better, in the dark. I mean, one of the key statistics is most of the accidents, especially for pedestrians, happen in the night, it happens due to low visibility. Now, if you have advanced driver assist systems that actually can sensor a pedestrian really far away with very high accuracy. This could be a really game changing application. And as long as it is low cost, we can actually bring it to almost every single car that is on the road today.

Nick Petrić Howe

And to bring us back to the start. Have you tried this out yourself? And did you feel like the Predator when you were using it?

Zubin Jacob

That's a really good question. You know, as a professor, you're involved in a lot of things. But I have to say the fun stuff is done by the students. So, I know from a lot of the images that they sent me that my students did have a lot of fun in the lab working on this. We do make jokes about this student looking like a Predator or the other. So, this has been a lot of fun looking at it, but I have to say it was my students who had a lot of these images that they had, and they also went outside the lab in the fields, and also had a great time taking the data.

Nick Petrić Howe

That was Zubin Jacob from Purdue University, in the US. To find out more about Predator-like vision, check out the show notes for some links.

Shamini Bundell

Coming up, a new analysis of the puzzling link between COVID and type 1 diabetes. Right now though, it's time for the research highlights with Dan Fox.

<Music>

Dan Fox

A new analysis has shown that Benjamin Franklin more than earned his place on the $100 bill, by developing clever anti-counterfeiting techniques for paper money. Franklin, one of the founding fathers of the United States and advocate for paper money, created printing networks across the British colonies in North America in the middle of the 18th century and experimented with a variety of papers, inks and designs. To better understand these techniques, researchers imaged more than 600 paper notes printed between 1709 and 1790 and analyzed their chemistry. The results revealed that Franklin developed special graphite-based pigments, coloured fibers and intricate patterns inspired by nature to foil counterfeiters. The authors of this research hope that it can inform how best to preserve historical money in future. Read that research in full in the Proceedings of the National Academy of Sciences of the United States of America.

<Music>

Dan Fox

How much snow sits on top of Mount Everest? Well, according to a new expedition, several meters more than previously thought. Mountaineers and researchers have used everything from wooden stakes to radar in a decades-long effort to determine the precise depth of snow on the world's highest mountain summit. In May last year, researchers measured snow depth at intervals along the mountain's north slope using a system that sends radar pulses into the ground and measures the energy that bounces back from structures under the surface. Their results suggest a snow depth of around 9.5 meters at the summit, almost three times as much as a previous estimate that also used ground penetrating radar. The authors say that regular data will be needed in order to track seasonal and annual changes, as well as the impact of global warming. Read that research in full in The Cryosphere.

<Music>

Shamini Bundell

Finally on the show it's time for the Briefing Chat, where we discuss a couple of stories that have been highlighted in the Nature Briefing. So Nick, what story have you picked for us this time?

Nick Petrić Howe

So I've been reading a news story from Nature about how much longer it takes people whose native language is not English, to write papers and read papers in English and you know, sort of participate in science writ large.

Shamini Bundell

Has someone put some numbers on this?

Nick Petrić Howe

Yeah, that's exactly right. So, this news article is based on a paper in PLOS ONE, and they did a poll of 908 researchers from eight different countries, with varying levels of proficiency in English to try and figure out how long it takes different people to read and write articles. And they found that it took people twice as long reading an English language article if they were from a country, which didn't have great English proficiency. And it took them between 30 and 50% more time to write a paper as well. And that may not sound like much, but it can add up to quite a considerable amount. So they put into context by saying that if you're a PhD student, writing your thesis, for example, you would spend 19 additional days just reading papers. So, it could be quite a considerable difference between native and non-native English speakers.

Shamini Bundell

Science is this international community, people travel all over the world, there are a lot of people working in places that aren't their country of origin, working in not their language of origin. Is there a reason that these researchers looked at English in particular?

Nick Petrić Howe

Yeah, I mean, the reason that they looked at English is the vast, vast, vast majority of papers, research articles, conferences, pretty much everything to do with the scientific enterprise are conducted in English. So if English is not your native language, you need to have some level of proficiency to really engage with science. It's not always true. There are papers that are published in other languages, but a vast majority are in English.

Shamini Bundell

So, there are so many people around the world, therefore, who are working in English, who might be really fluent, but it's not their first language. And this is an extra time cost for them, which I suspect that that people for whom English as their first language and they're working in English might find it a bit harder to actually appreciate all that that extra time sunk into writing papers, reading papers, as you say.

Nick Petrić Howe

That certainly seems to be the case. And some people who were interviewed for this article had a couple of quotes. So, for example, one of the study's authors themselves is Japanese. And they said that people might think their papers are a similar level to a native English speaker. But behind the scenes, they have to spend so much time to actually reach that level when they're writing papers and things like that. Another person who was interviewed as well, who is Colombian, actually said that they have had reviewers explicitly say that their English puts into doubt the quality of their research, or just gave them feedback on their English in a harsh way when they're sort of submitting papers. So, it seems like there is certainly an additional cost for people who don't have English as their native language when they are participating in science.

Shamini Bundell

And does this paper put forward any suggestions, any potential solutions because this, this must be a problem affecting so many people.

Nick Petrić Howe

Indeed, and the study's author does suggest a couple of things. So probably hasn't escaped your notice that there's quite a lot of artificial intelligence things going around now. So they suggested that maybe some of these tools could be used to help people with their English, or they could be connected with other scientists who are better English speakers, or maybe they can even present conferences in their native language and have a translator present because that may allow them to actually participate, because some people who responded to this study actually said that they just didn't present at all conferences, because they were so worried about the language barrier.

Shamini Bundell

Well, I'm sure a huge number of our listeners, probably relate to this. And listeners, if you have any anecdotes or anything you want to share about this topic, you can, as usual, get in touch with us podcast@nature.com is our email address, or we're @naturepodcast on Twitter. So get in touch, and we might read out some of your comments on next week's show. Well, thanks for that, Nick, I also have a Nature news story for you today, based on a paper from The JAMA Network Open Journal. And I'm going to warn you before I tell you about my story, that it doesn't have a terribly satisfactory ending. So, I'm just going to warn you now that the ending is kind of a big question mark on this just so you don't get disappointed later. But it's basically about a fascinating link between a certain type of diabetes, and the COVID-19 pandemic. There's been lots of research on this, but this particular paper is a meta-analysis, which has taken the results of 42 other studies, sort of selected studies, and come to a couple of conclusions, including that, yes, there is a sort of notable link between the first couple of years of the pandemic and the incidence of type 1 diabetes in children.

Nick Petrić Howe

Right. Okay. So, this is an established thing that scientists have been looking for that there might be a link between type 1 diabetes and COVID. And if I remember, rightly, this isn't the kind that you get, usually later in life where you become insulin resistant?

Shamini Bundell

Exactly. So yeah, that's type 2 diabetes, which is very known to be associated with lifestyle factors such as diet, whereas type 1 diabetes often first appears in children, and is, in most cases, and this is kind of mostly what we're talking about, an autoimmune situation. So the immune system attacks the cells of the pancreas that make insulin, and thus you are short of insulin, insulin being the hormone that helps you regulate blood sugar.

Nick Petrić Howe

Right. Because the reason I say that is my first thought was, we were all quite inactive and not doing very much during a lot of the COVID pandemic. But I guess here, something else is going on to potentially lead to this rise in type 1 diabetes?

Shamini Bundell

Absolutely. Yeah, that was the interesting thing that I thought is that there's a lot that's known about the causes of type 2 diabetes, that you would think we'd be associated with the kind of lifestyles that many of us had during the pandemic. But that doesn't really apply to type 1. But the numbers seem pretty convincing. Not everyone who was interviewed for this article was completely convinced. But I'll throw some stats at you here. Pooling data from 17 studies, they found that the incidence of type 1 diabetes in under 19-year-olds was 14% higher during 2020, than in 2019. And then in the second year of the pandemic, compared to 2019, it was 27% higher. And this is, I should also say, against a background of type 1 diabetes slowly increasing over the last few years. Already it was at a rate of about 2 to 4% a year growing. And now it's leaped.

Nick Petrić Howe

And do researchers have any ideas why this might be?

Shamini Bundell

This is where the unsatisfactory part of this comes in, because like the ultimate answer is they don't know. And I mean, that's actually fascinating, too. So obviously, there was a worry that the COVID-19 virus was somehow giving people type 1 diabetes, you know, maybe it's sort of somehow causing the immune cells to attack the pancreas. Previous studies have really struggled to show this direct link between the virus and type 1 diabetes. And this study again, didn't find that kind of a direct link. But there are theories of, for example, perhaps some kind of viral infections, it has been thought, can trigger the immune system to attack the pancreas. So maybe COVID-19 had a similar impact. It could be environmental factors, like diet, there could be more environmental factors that changed during COVID that we don't know about. It could be the fact that you're in lockdown and not being exposed to other things. Maybe even other viruses. But one interesting thing is that we also didn't know why this type 1 diabetes has been increasing generally. So, if we can use this possible leap, to sort of try and figure out some of those environmental conditions, scientists could then have a greater insight into what is, in general, causing type 1 diabetes to occur.

Nick Petrić Howe

Well, I guess that's maybe a tiny positive that may have come out of the pandemic, if we can actually discover why this link exists. But I think that's all we've got time for this week on the Briefing. Thanks for chatting to me Shamini. And listeners, for more on those stories and for where you can sign up to the Nature Briefing, check out the show notes for some links.

Shamini Bundell

And as I mentioned in there, get in touch with us if you've got anything to say about language barriers, or otherwise, you can email us podcast@nature.com. Or you can find us on Twitter, if it's still called Twitter, @naturepodcast.

Nick Petrić Howe

That's all for this week show. But keep an eye on your podcast feed late tomorrow. As there's another bit of audio goodness coming your way. We've got an extra story on a new paper that we think you'll be interested in. I'm Nick Petrić Howe.

Shamini Bundell

And I'm Shamini Bundell. Thanks for listening.