Download a PDF of this story.

At full speed, the altercation would have looked like nothing — a brief contact, fractions of a second long, between two flies. But slowed to 1/20th of normal speed it has all the flash and dazzle of an elaborate professional wrestling move. Biologist David Anderson calls the grainy, black and white video on his computer the "fly lucha libre".

One fly, a male, rears up and clamps down on his neighbour. Flipping backwards he whips his opponent into the air, executing two somersaults in the process. The victim, helpless, flaps its wings as it crashes to the floor, then rights itself and flies away. Take-downs like these are just one of dozens of behaviours that Anderson studies at California Institute of Technology (Caltech) in Pasadena (see the video from Martin Heisenberg at the University of Würzburg in Germany at

"It's this whole world you'd have no clue existed," says Anderson's collaborator, Pietro Perona, whose forays into this world have been relatively recent. For 20 years the computational-vision scientist, also at Caltech, has been trying to develop machines capable of detecting and interpreting complex behaviours in humans. It's an ambitious goal, but he has found unlikely allies in Anderson and behavioural neuroscientist Michael Dickinson. Since about 2005, the three have collaborated to create tools that combine the ease of manipulating the fly nervous system with ways to automatically track complex social behaviours such as aggression and courtship. The hope is to understand their neural underpinnings, which may inform studies in humans.

Pietro Perona uses machine vision to track flies. Credit: B. PAZ

This year, the group demonstrated two systems that can do this1, 2. Anderson is now using them to screen thousands of lines of mutant flies to understand the genes and neural circuits that control behaviours, and Dickinson is exploring the social dynamics of flies in large groups — something that has been nearly impossible to approach in the lab.

"This is a very tough problem," says Joel Levine, a neurogeneticist at the University of Toronto in Mississauga, Ontario, who has been using one of the software programs, called Ctrax, since November 2008. "I've been trying for a very long time to get people in my lab to write this kind of software, and no one could do it."

Because the programs are freely available, fly researchers from around the world are using the new technology and taking off in their own directions. And through the flies, Perona is inching closer to his goal of building machines that can interpret human behaviour.

The tools promise to be a boon for the field of fly behaviour, and especially its students, technicians and postdocs who perform a thankless job: watching videos frame by frame, tallying every lunge, timing every chase, classifying every wing threat and repeating.

"It's mind-numbing to sit in front of videos and score them by hand," Anderson says over lunch with Perona and Dickinson at the Caltech faculty restaurant. It's not just a matter of saving time, Perona adds. Accuracy counts, yet observing flies can be a subjective exercise. "Suppose we use the clipboard and stopwatch method. We have a postdoc at Caltech who does it, and another postdoc in Norway who does it." If results don't agree, getting the answer is next to impossible: "Those postdocs are gone, they've got jobs somewhere else, so we're left in this situation of irreconcilably different opinions — opinions that never become fact because there is nothing to ground them on."

Big brother

Perona had been looking to develop machines that can understand peoples' actions and intentions by watching their movements. Machines with such capabilities would have an array of applications. Imagine workplace safety cameras that can warn when a machinist is dozing or a security camera at an airport that flags suspicious behaviour.

To build these systems, machine-vision scientists train computers using a common language: labelled data. Annotate thousands of examples of pedestrians stepping on to a crossing, for example, and one can produce an algorithm that could identify that behaviour in novel situations.

But simple, specific behaviours are of limited use when the goal is to recognize behaviours that would qualify as 'suspicious' or 'dangerous'. Perona is approaching this problem by trying to break down behaviour into its component parts.

Pouring a glass of wine, for example, involves gripping the bottle, lifting it, moving it towards a glass and finally tipping it down and up. Perona has dubbed such elementary motions 'movemes'3, and they form the basis of his behavioural hierarchy. Movemes combine into an action, such as pouring wine. An action that is prolonged in time is an activity, such as having dinner. But testing such a theory, he says, requires enormous sets of labelled videos — videos that are hard to come by.

Humans are boring. How much time do you have to spend in pubs before you finally see a fight? Pietro Perona ,

"You just can't get enough data," Perona says as he counts off the reasons on his fingers. It's illegal to tape without permission, difficult to get approval and then harder, with permission granted, to get natural, candid activity. "Also humans are boring," Perona says. "How much time do you have to spend in pubs before you finally see a fight?"

Perona realized that he could learn a lesson from the life sciences. "When biologists encounter complicated issues such as schizophrenia, what do they do?" he asks. "They use a model organism. Something that lets them do very intrusive, invasive experiments and get answers much more quickly."

A few years ago, he started asking his colleagues at Caltech about their model organisms, searching for one with behaviours that were interesting, but not too complex. Meanwhile, Anderson had just come back from a sabbatical where a lab was using an off-the-shelf tool to measure how much flies were moving around.

Watching flies: Michael Dickinson (left) and David Anderson use computer vision programs to advance their studies of fly behaviour. Credit: P. PERONA

"It worked, but it was clunky," Anderson remembers. Dickinson suggested that he should build something himself. So Anderson approached Perona with descriptions of the flies' intricate behaviours. Perona was intrigued. "I didn't know flies do more than just go buzz and push themselves through the air," he says.

High ambitions

Anderson wanted a program that could automatically detect specific behaviours during aggression and courtship — counting and timing wing threats, chases, wrestling bouts and copulations. He wanted to be able to run many pairs of flies at the same time and have the machine spit out when, where and in what order every action occurred.

Dickinson, who has devoted most of his career to the aerodynamics and biomechanics of fly flight, wanted to understand how flies behave in groups of dozens to hundreds. To do this, he would need to keep track of each fly's movement and maintain their identity over long periods of time — something no one had done before. There were a couple of programs that could follow flies, but they had an 'occlusion problem'. Whenever two insects touched or entered into the same pixel, an experimenter would have to manually tell the machine how to separate the flies out.

Both had tall orders for Perona. "But that's the great thing about collaborations," Anderson says. "Something that to biologists looks nearly impossible may be in the realm of achievability for someone like Pietro."

Anderson's specialized behaviour detector, and Dickinson's exploratory tracker required very different software, but the tasks seemed simple enough. Perona shared one postdoc with Dickinson — Kristin Branson — and another with Anderson — Heiko Dankert. Perona estimated that each program would take a matter of months. "Of course it ended up taking three years, but that's fine," Perona says. Much of the time was spent making it understandable to biologists and also robust enough to work reliably in the hands of others, what Dickinson calls the "fender and body work".

Meanwhile, Dickinson was tinkering away, trying to create an environment for Branson's software — Ctrax — to analyse. The product was a circular arena just large enough to hold a football, surrounded by eight halogen lights.

Inside, up to 50 flies walk around; their wings clipped to prevent them from taking off and confusing the tracking system. A camera hangs directly over the centre of the arena, capturing the movements and relaying them to a nearby computer. On the monitor, each fly leaves a thin, brightly coloured trail as it explores the arena. After a few minutes the screen is so clogged with lines that it looks like a ball of multicoloured yarn.

In essence the program doesn't do much more than follow flies around. The idea was to create an all-purpose tracker and quantifier that would let the experimenter decide which behaviours were interesting — and to tell the machine not only what to look for but how to learn what to look for.

"This is one of the most exciting aspects of it, this discovery phase of research," says Dickinson. Pietro describes one intriguing behaviour they identified. "So there was one fly going slowly, and another fly was overtaking it," Pietro says, demonstrating the motions with his fingers. "As the faster fly was overtaking the slow fly, the slow fly stops. Now it could have been a fluke that the fly decided to stop, but so then Michael goes forward and he finds another example of the same thing happening. And so, now we can very quickly program the software to pull out every encounter of flies going like this." All they have are hunches as to why the flies do it, says Perona.

But once an experimenter recognizes an interesting sequence of movements or interactions, he or she can propose hypotheses and answer them with a new analysis — no new data required. The program will then generate quantitative descriptions of each behaviour for each individual fly. These 'ethograms' allow the user to discover and quantify subtle behavioural differences between populations of flies, such as males versus females or flies with genetic mutations, and even between individuals within a population — fly personalities.

Dickinson's group has already made some novel observations using the ethograms. Male flies, for example, are not shy — they walk close to one another, inspecting, even nudging each other from time to time. But females need their personal space. They obey certain rules of etiquette when they meet each other and when they walk around.

His group is also starting to explore how social interactions change in more complex environments, placing cones and obstacles on the landscape. He has found that flies tend to climb to the top of the cones, and chase each other off in 'king of the mountain' fashion. With these new observations come new hypotheses. Dickinson says the behaviour might have something to do with maintaining a good vantage point for identifying food sources.

In a lab a few hundred metres away, Anderson is using his program, called the Caltech Automated Drosophila Aggression-Courtship Behavioral Repertoire Analysis (CADABRA), to take a quantitative look at behaviours he already knows are there — wrestling, tackling and other manoeuvres that flies attempt during aggressive bouts, and the serenades and dances that lead to mating.

He is now developing a screen in which he has genetically activated and silenced different populations of neurons. These selective disruptions will allow him to dissect the neural circuitry of each behaviour — not just aggression as a whole, but the frequency of each component of the behaviour, and even the frequency at which one behaviour leads to another.

To do this by hand would take 270 hours. We did it in 20 minutes. David Anderson ,

In his set-up, a pair of flies sits in a shallow well about as wide as a golf ball. A plate about a centimetre high holds 12 such wells. Hanging above, a camera films all 12 interactions in parallel. The video can then be run, frame by frame, through CADABRA, which then produces statistics on every behaviour.

CADABRA's ethograms, for example, illustrate the frequency of each lunge and chase, as well as the frequency of transitions — such as how often a chase is followed by a lunge versus a flirtatious wing extension.

"To do this by hand would take 270 person hours," Anderson says. "We basically did it in 20 minutes."

Flight club

Levine learned about Ctrax when he and Dickinson were lecturing at the Marine Biological Laboratory in Woods Hole, Massachusetts, in the summer of 2009. He has now configured the system so that it can follow individual flies in a group over a period of days, without any interference.

"So little is known about group dynamics," he says. "Who interacts with whom? Why do some flies mate more than others? How do social interactions affect the circadian rhythms? This has been very difficult to study. And now along comes Michael with this software, and, finally, we can ask these questions."

Leslie Vosshall, a neuroscientist at Rockefeller University in New York, is using Ctrax to study courtship. She is using a modified set-up, designed in Dickinson's lab, in which the flies wander around in a dish-like arena covered by a plastic lid. This allows them to move about with their wings intact — a crucial modification, because of the involvement of wing signals and wing songs in courtship.

And Michael Reiser — Dickinson's former graduate student, now at the Howard Hughes Medical Institute's Janelia Farm Research Campus in Virginia — is taking advantage of Ctrax's flexibility for his 'Fly Olympiad'. He is putting mutant flies through a barrage of behavioural tests — from visual reflexes to walking behaviour to odour sensation.

Perona says he's pleased that his work has started to bear fruit for biologists. And Anderson's and Dickinson's projects are producing the videos of labelled behaviours of which he had been starved.

The translation of fly behaviours into ethograms is also giving Perona his first opportunity to test his hypothesis of behavioural hierarchies — in which a continuous, extended activity such as courtship can be broken down into actions, which can further be decomposed into elementary but meaningful motions.

He hopes the concept will form the basis of an overall computational theory of behaviour. But although the simplicity of fly behaviour allows Perona to make conceptual progress, it can only take him so far. "Am I closer to building a machine that can figure out what people are doing? Yes and no. Flies have a much simpler repertoire of behaviour than humans and they don't waste time." Watching fly behaviours should allow him to go back to human data with more solid ways of testing his hypotheses. But Perona's goals have also been evolving. He says he has become increasingly interested in fly behaviour for its own rewards. "If it all ended up with me having contributed something to biology, it would still be fantastic," he says.