Some animals are loners, but many interact in groups: ants collaborate, zebrafish mill about and sometimes fight. Neuroscientists and field ecologists track such behavior with video that is hard to analyze quantitatively. Gonzalo G. de Polavieja, a researcher at Champalimaud Center for the Unknown in Lisbon, has long worked on the theory of group-based decision-making in animals. “At some point we wanted to know who is who,” he says. “We wanted to know ‘who was George,’ the one we trained?” Misidentifying which animals crossed paths is a small mistake, but “very quickly with two to three mistakes, you may be confusing all the animals in your video,” he says.

Gonzalo G. de Polavieja. Credit: A. Azinheira

IdTracker, his previous tool, helped with this disambiguation. He and his team now present, which sounds similar but has an entirely different motor under the hood, one that leverages recent advances in artificial intelligence (AI). Labs can still use the old tool, but for analyzing larger groups of animals more quickly, he recommends trying the new one. This system is species agnostic and can analyze videotaped behavior of up to 100 animals. It’s been tested on the lab’s own videos of zebrafish, medaka (another kind of fish), flies, ants and mice, as well as videos from other labs of ants and more fish species, and de Polavieja knows of labs that have used it with videos of birds, lizards and locusts. has three important components, he says. The system’s two convolutional networks parse video as individual static images. One network ‘decides’ whether a given “blob” in an image corresponds to one or more than one animal, he says. The second network then picks up the analysis, identifying which animal in the group that one blob is. Then the third component comes into play—a set of conditions: heuristics. These mathematical definitions help with computational decisions about whether, for example, a given blob is a single animal or a group, and they support statistical descriptions of size and shape distribution. When the heuristics help make good assignments in any given instance, training of the network continues. If an animal, George, is identified in one scene, it would be a network error if George were identified in another scene at the same time. Positive assignments of a given scene move the analysis forward. “It’s a way of making the training dataset larger and larger in a safe way by obeying some heuristics,” he says. The heuristics make the networks, which are “the magic of deep learning,” more effective, he says.

The team built the system’s components from scratch on top of TensorFlow, an open source deep learning framework originally developed by the Google Brain Team.’s code is written in Python modules, so the system is, he says, “almost like a development platform” that labs can adapt to their needs. The project is built on breakfast-based collaboration: every morning for two years, the team talked things through. De Polavieja’s postdoctoral fellow Mattia Bergomi brought his network knowledge to the breakfast table, and PhD student Francisco Romero-Ferrero his eagerness to apply deep learning.

De Polavieja received his DPhil in physics from the University of Oxford for work on quantum geometric phases and switched to neuroscience as a postdoctoral fellow in Simon Laughlin’s lab at the University of Cambridge. While he was writing his thesis, he sat in on a graduate neuroscience seminar. “For me always, many of the interesting problems were in biology,” he says. But he didn’t want to study biology, where there was too much to learn. “I like the compression of physics,” he says, joking about his possibly limited memory capacity. He felt at home in the “mathematically minded” Laughlin lab. “For me it was not a big change: mathematics to understand a neuron,” he says. He was using mathematics to understand the transformation in photoreceptors. After his postdoc, he joined the theoretical physics department at Universidad Autonoma de Madrid, then moved to the Cajal Institute. There, he met a group working on zebrafish and began watching groups of fish. He had been thinking about decision-making and learning in single individuals: “suddenly I realized I did not understand anything about what happens in a group.” That led to idTracker. He moved to Champalimaud and has kept his affiliation with Cajal. “I am very interested in the mathematics of learning,” he says.

For me it was not a big change: mathematics to understand a neuron.

“Ever since we first collaborated, nearly 20 years ago as research fellows in Cambridge, Gonzalo still keeps amazing me with his uncanny knack to see through the clutter into the heart of a complex problem, describing mathematically how the system might work or how it could be studied at the fundamental level,” says Mikko Juusola, a systems neuroscientist at the University of Sheffield who is friends with de Polavieja. The new species-agnostic solution to track many behaving animals in a video “makes supervised surveillance practically obsolete,” he says. Every morning, de Polavieja swims in a pool near his house. He swims in nearby rivers and the ocean, where the family boats and surfs. “My first job is to be a father,” he says. With his two children, he says, he watches movies, rediscovers the world’s capitals and rational numbers.