Credit: Paul jackman/Nature

Even as home experiments go, Hopi Hoekstra’s one was peculiar: she built a giant plywood box in her garage in San Diego, California, filled it with more than a tonne of soil and then let a pet mouse dig away.

“This thing was bursting at its seams and held together with duct tape,” says the evolutionary biologist, now at Harvard University in Cambridge, Massachusetts. “But it worked.” It allowed her to study the genetics of burrowing behaviour in a controlled setting. Armed with plastic casts of the burrows and state-of-the-art sequencing, Hoekstra’s team discovered clusters of genes that partly explain why the oldfield mouse (Peromyscus polionotus) builds elaborate two-tunnel burrows, whereas its close relative, the deer mouse (Peromyscus maniculatus), goes for a simple hole in the ground1.

The findings highlight an underappreciated benefit of a genomics revolution that is moving at breakneck speed. Thanks to cheap and quick DNA sequencing, scientists interested in the genetics of behaviour need not limit themselves to a handful of favourite lab organisms. Instead, they can probe the genetic underpinnings of behaviours observed in the wild, and glean insights into how they evolved. “In my mind, the link between genes and behaviour in natural populations and organisms is the next great frontier in biology,” says Hoekstra.

Oldfield mice are native to the southeastern United States, where they burrow in soils ranging from sandy beaches to silt-rich clays. Wherever they dig, their holes look much the same, with a long entrance tunnel and a second tunnel that stops short of the surface and allows them to escape predators. Such invariability hints that the trait is encoded in DNA, says Hoekstra.

To find out where, she and her Harvard colleagues Jesse Weber and Brant Peterson cross-bred oldfield mice with deer mice, whose burrows are shallow and lack escape routes. The offspring continued to build complex tunnels, suggesting that the oldfield burrowing genes were dominant (see ‘The genetics of burrowing’).

A second round of breeding between the first-generation crosses and deer mice revealed that genes linked to burrow length were distinct from those influencing the escape tunnel. Some offspring produced short tunnels with escape routes, whereas others produced long tunnels without them. DNA analysis revealed that three genetic regions are responsible for much of the variation in tunnel length, and a fourth affects escape-tunnel digging.

“This paper is awesome,” says Cornelia Bargmann, a neurogeneticist at Rockefeller University in New York, noting that it combines cutting-edge molecular-genetics tools with established cross-breeding techniques to study behaviours that have been observed for more than a century in the wild. In the past, geneticists interested in unravelling behaviour had to focus on lab animals for which mutant and transgenic strains and genetic data were available, she says. “But there were always questions we knew would be more interesting in wild animals.” Bargmann and her team studied various wild strains of Caenorhabditis elegans flatworms to identify genes and brain circuits involved in seeking out new sources of food2.

The link between genes and behaviour in natural populations is the next great frontier.

Cheap genome sequencing is the foundation of this new behavioural genetics. In a paper published on Nature’s website this week3, a team led by Laurent Keller at the University of Lausanne, Switzerland, sequenced hundreds of fire ants (Solenopsis invicta) and identified a region that explains why some colonies have only one queen whereas others tolerate more. The region contains hundreds of genes, and Keller thinks that vast behavioural differences among other insects might be explained by such ‘social chromosomes’.

These studies offer clues to the evolution of behaviours seen in the wild, says Catherine Peichel, an evolutionary geneticist at the Fred Hutchinson Cancer Research Center in Seattle, Washington. In unpublished work, Peichel and her postdoc Anna Greenwood have discovered a gene that explains differences in schooling behaviour in stickleback fish. The marine variety swims in tight schools, whereas its freshwater cousin swims solo. Such behaviours were probably selected for by different environments — predator-filled seas with few hiding places versus vegetation-filled streams. And Hoekstra says her findings suggest that complex burrowing in mice evolved piecemeal, by combining the genetic modules responsible for simpler digging behaviours.

Hoekstra’s team has already found that the key regions for burrowing include genes involved in addictive behaviour. “When you watch these mice, you can’t help but think they look like they’re addicted to burrowing,” Hoekstra says. “You drop one of these oldfield mice in one of our burrowing boxes and they immediately start digging.”

The ultimate proof of any gene’s role will be to engineer deer mice to build complex burrows by inserting oldfield mice genes, says Hoekstra. Those experiments would have been unthinkable a few years ago, when genome engineering was possible in only a handful of mammals, but this is now conceivable, says Bargmann (see Nature 455, 160–164; 2008).