As a postdoc, physiologist Valentina Di Santo spent a lot of time scrutinizing high-resolution films of fish.

Di Santo was investigating the motions involved when fish such as skates swim. She filmed individual fish in a tank and manually annotated their body parts frame by frame, an effort that required about a month of full-time work for 72 seconds of footage. Using an open-source application called DLTdv, developed in the computer language MATLAB, she then extracted the coordinates of body parts — the key information needed for her research. That analysis showed, among other things, that when little skates (Leucoraja erinacea) need to swim faster, they create an arch on their fin margin to stiffen its edge1.

But as the focus of Di Santo’s research shifted from individual animals to schools of fish, it was clear a new approach would be required. “It would take me forever to analyse [those data] with the same detail,” says Di Santo, who is now at Stockholm University. So, she turned to DeepLabCut instead.

DeepLabCut is an open-source software package developed by Mackenzie Mathis, a neuroscientist at Harvard University in Cambridge, Massachusetts, and her colleagues, which allows users to train a computational model called a neural network to track animal postures in videos. The publicly available version didn’t have an easy way to track multiple animals over time, but Mathis’ team agreed to run an updated version using the fish data, which Di Santo annotated using a graphical user interface (GUI). The preliminary output looks promising, Di Santo says, although she is waiting to see how the tool performs on the full data set. But without DeepLabCut, she says, the study “would not be possible”.

Researchers have long been interested in tracking animal motion, Mathis says, because motion is “a very good read-out of intention within the brain”. But conventionally, that has involved spending hours recording behaviours by hand. The previous generation of animal-tracking tools mainly determined centre of mass and sometimes orientation, and the few tools that captured finer details were highly specialized for specific animals or subject to other constraints, says Talmo Pereira, a neuroscientist at Princeton University in New Jersey.

Over the past several years, deep learning — an artificial-intelligence method that uses neural networks to recognize subtle patterns in data — has empowered a new crop of tools. Open-source packages such as DeepLabCut, LEAP Estimates Animal Pose (LEAP) and DeepFly3D use deep learning to determine coordinates of animal body parts in videos. Complementary tools perform tasks such as identifying specific animals. These packages have aided research on everything from the study of motion in hunting cheetahs to collective zebrafish behaviour.

Each tool has limitations; some require specific experimental set-ups, or don’t work well when animals always crowd together. But methods will improve alongside advances in image capture and machine learning, says Sandeep Robert Datta, a neuroscientist at Harvard Medical School in Boston, Massachusetts. “What you’re looking at now is just the very beginning of what is certain to be a long-term transformation in the way neuroscientists study behaviour,” he says.

Strike a pose

DeepLabCut is based on software used to analyse human poses. Mathis’ team adapted its underlying neural network to work for other animals with relatively few training data. Between 50 and 200 manually annotated frames are generally sufficient for standard lab studies, although the amount needed depends on factors such as data quality and the consistency of the people doing the labelling, Mathis says. In addition to annotating body parts with a GUI, users can issue commands through a Jupyter Notebook, a computational document popular with data scientists. Scientists have used DeepLabCut to study both lab and wild animals, including mice, spiders, octopuses and cheetahs. Neuroscientist Wujie Zhang at the University of California, Berkeley, and his colleague used it to estimate the behavioural activity of Egyptian fruit bats (Rousettus aegyptiacus) in the lab2.

The deep-learning-based posture tracking package LEAP, developed by Pereira and his colleagues requires 50–100 annotated frames for lab animals, says Pereira. More training data would be needed for wildlife footage, although his team has not yet conducted enough experiments to determine how much. The researchers plan to release another package called Social LEAP (SLEAP) this year to better handle footage of multiple, closely interacting animals.

Jake Graving, a behavioural scientist at the Max Planck Institute of Animal Behavior in Konstanz, Germany, and his colleagues compared the performance of a re-implementation of the DeepLabCut algorithm and LEAP on videos of Grevy’s zebras (Equus grevyi)3. They report that LEAP processed images about 10% faster, but the DeepLabCut algorithm was about three times as accurate.

Graving’s team has developed an alternative tool called DeepPoseKit, which it has used to study behaviours of desert locusts (Schistocerca gregaria), such as hitting and kicking. The researchers report that DeepPoseKit combines the accuracy of DeepLabCut with a batch-processing speed that surpasses LEAP. For instance, tracking one zebra in 1 hour of footage filmed at 60 frames per second takes about 3.6 minutes with DeepPoseKit, 6.4 minutes with LEAP and 7.1 minutes with his team’s implementation of the DeepLabCut algorithm, Graving says.

DeepPoseKit offers “very good innovations”, Pereira says. Mathis disputes the validity of the performance comparisons, but Graving says that “our results offer the most objective and fair comparison we could provide”. Mathis’ team reported an accelerated version of DeepLabCut that can run on a mobile phone in an article posted in September on the arXiv preprint repository4.

Biologists who want to test multiple software solutions can try Animal Part Tracker, developed by Kristin Branson, a computer scientist at the Howard Hughes Medical Institute’s Janelia Research Campus in Ashburn, Virginia, and her colleagues. Users can select any of several posture-tracking algorithms, including modified versions of those used in DeepLabCut and LEAP, as well as another algorithm from Branson’s lab. DeepPoseKit also offers the option to use alternative algorithms, as will SLEAP.

Other tools are designed for more specialized experimental set-ups. DeepFly3D, for instance, tracks 3D postures of single tethered lab animals, such as mice with implanted electrodes or fruit flies walking on a tiny ball that acts as a treadmill. Pavan Ramdya, a neuroengineer at the Swiss Federal Institute of Technology in Lausanne (EPFL), and his colleagues, who developed the software, are using DeepFly3D to help identify which neurons in fruit flies are active when they perform specific actions.

And DeepBehavior, developed by neuroscientist Ahmet Arac at the University of California, Los Angeles, and his colleagues, allows users to track 3D movement trajectories and calculate parameters such as velocities and joint angles in mice and humans. Arac’s team is using this package to assess the recovery of people who have had a stroke and to study the links between brain-network activity and behaviour in mice.

Making sense of movement

Scientists who want to study multiple animals often need to track which animal is which. To address this challenge, Gonzalo de Polavieja, a neuroscientist at Champalimaud Research, the research arm of the private Champalimaud Foundation in Lisbon, and his colleagues developed idtracker.ai, a neural-network-based tool that identifies individual animals without manually annotated training data. The software can handle videos of up to about 100 fish and 80 flies, and its output can be fed into DeepLabCut or LEAP, de Polavieja says. His team has used idtracker.ai to probe, among other things, how zebrafish decide where to move in a group5. However, the tool is intended only for lab videos rather than wildlife footage and requires animals to separate from one another, at least briefly.

Other software packages can help biologists to make sense of animals’ motions. For instance, researchers might want to translate posture coordinates into behaviours such as grooming, Mathis says. If scientists know which behaviour they’re interested in, they can use the Janelia Automatic Animal Behavior Annotator (JAABA), a supervised machine-learning tool developed by Branson’s team, to annotate examples and automatically identify more instances in videos.

An alternative approach is unsupervised machine learning, which does not require behaviours to be defined beforehand. This strategy might suit researchers who want to capture the full repertoire of an animal’s actions, says Gordon Berman, a theoretical biophysicist at Emory University in Atlanta, Georgia. His team developed the MATLAB tool MotionMapper to identify often repeated movements. Motion Sequencing (MoSeq), a Python-based tool from Datta’s team, finds actions such as walking, turning or rearing.

By mixing and matching these tools, researchers can extract new meaning from animal imagery. “It gives you the full kit of being able to do whatever you want,” Pereira says.