Credit: OLIVER MUNDAY

Growing up in the college town of Ames, Iowa, during the 1970s, Stephen Hsu was surrounded by the precocious sons and daughters of professors. Around 2010, after years of work as a theoretical physicist at the University of Oregon in Eugene, Hsu thought that DNA-sequencing technology might finally have advanced enough to help to explain what made those kids so smart. He was hardly the first to consider the genetics of intelligence, but with the help of the Chinese sequencing powerhouse BGI in Shenzhen, he planned one of the largest studies of its kind, aiming to sequence DNA from 2,000 people, most of whom had IQs of more than 150.

He hadn't really considered how negative the public reaction might be until one of the study's participants, New York University psychologist Geoffrey Miller, made some inflammatory remarks to the press. Miller predicted that once the project turned up intelligence genes, the Chinese might begin testing embryos to find the most desirable ones. One article painted the venture as a state-endorsed experiment, selecting for genius kids, and Hsu and his colleagues soon found that their project, which had barely begun, was the target of fierce criticism.

There were scientific qualms over the value of Hsu's work (see Nature 497, 297–299; 2013 ). As with other controversial fields of behavioural genetics, the influence of heredity on intelligence probably acts through myriad genes that each exert only a tiny effect, and these are difficult to find in small studies. But that was only part of the reason for the outrage. For decades, scientists have trodden carefully in certain areas of genetic study for social or political reasons.

At the root of this caution is the widespread but antiquated idea that genetics is destiny — that someone's genes can accurately predict complex behaviours and traits regardless of their environment. The public and many scientists have continued to misinterpret modern findings on the basis of this — fearing that the work will lead to a new age of eugenics, preemptive imprisonment and discrimination against already marginalized groups.

“People can take science and assume it is far more determinative than it is — and, by making that assumption, make choices that we will come to regret as a society,” says Nita Farahany, a philosopher and lawyer at Duke University School of Law in Durham, North Carolina.

But trying to forestall such poor choices by drawing red lines around certain areas subverts science, says Christopher Chabris of Union College in Schenectady, New York. Funding for research in some areas dries up and researchers are dissuaded from entering promising fields. “Any time there's a taboo or norm against studying something for anything other than good scientific reasons, it distorts researchers' priorities and can harm the understanding of related topics,” he says. “It's not just that we've ripped this page out of the book of science; it causes mistakes and distortions to appear in other areas as well.”

Here, Nature looks at four controversial areas of behavioural genetics to find out why each field has been a flashpoint, and whether there are sound scientific reasons for pursuing such studies.

1 INTELLIGENCE

Taboo level: HIGH

The comments that Miller made about Chinese families and the government wanting to select for intelligent babies touched a nerve still raw after many years. In the nineteenth century, British anthropologist Francis Galton founded the eugenics movement on the premise that extraordinary abilities, as well as deficits, were inherited. The movement led to abuses, such as forced sterilization of people deemed mentally inferior — generally minorities, poor people and especially people with mental illnesses — in countries around the world, including Germany, the United States, Belgium, Canada and Sweden.

The term 'intelligence' is also a slippery one. Intelligence tests don't measure a wholly innate ability; it is possible, for example, to improve one's scores with practice. Nevertheless, about 50% of variability in intelligence seems to be inherited, posing an irresistible puzzle to some researchers. No one gene has been linked strongly to intelligence and many that have been weakly linked have also been questioned1.

Earlier this year, in an attempt to find stronger genetic correlations, Chabris and a large international group of colleagues examined the genomes of more than 125,000 people and found three genetic variants, each of which had a small effect on the length of an individual's school career2. The authors speculated that the variants' influence on educational attainment came from their effect on intelligence. But the results triggered the usual rounds of condemnation and concerns over eugenics. Other detractors argued that such studies take the focus and funding away from other, non-genetic, factors such as poverty, which have a much greater effect on social mobility.

Chabris says that the work can actually contribute to greater social mobility — for instance, by helping to identify preschoolers who could be helped by more intensive early childhood education. “The fact that people in the past interpreted the results in a certain way doesn't mean that it shouldn't be studied,” he says. But not everyone buys that potential misuses of the information can be divorced from gathering it. Anthropologist Anne Buchanan at Pennsylvania State University in University Park wrote on the blog The Mermaid's Tale that rather than being purely academic and detached, such studies are “dangerously immoral”.

Critics of the BGI project also point to signs that its data could be misused. After this summer's furore over Miller's interview, Hsu played down the potential for abuse. “There's a big gap between finding a few hits and finding thousands of hits — enough to predict the trait on the basis of the genotype — and we were never saying we were going to get to that point,” he says. But in 2011, before the uproar over the study, Hsu told Nature: “I'm 100% sure that a technology will eventually exist for people to evaluate their embryos or zygotes for quantitative traits, like height or intelligence. I don't see anything wrong with that.”

One of Hsu's collaborators, behavioural geneticist Robert Plomin of King's College London, says that such talk has not been helpful. But after studying intelligence for 40 years, he has high hopes that this project and other sequencing ventures will help to pinpoint the many genetic contributors to the trait. Like Chabris, he says that the work could be used to target educational interventions. Moreover, like all of the intelligence researchers interviewed for this story, he says it is a fundamentally human trait and that it is worth searching for a genetic contribution. “I'm optimistic that we will find it,” he says. “I'm not going to quit until we do.”

2 RACE

Taboo level: VERY HIGH

As far as genetic taboos go, race is probably one of the most heavily policed from within the scientific community, largely because of the way researchers have examined its intersection with other controversial traits, such as intelligence. This is due mostly to suspicion about what motivates the study. There is broad consensus across the social and biological sciences that groups of humans typically referred to as races are not very different from one another. Two individuals from the same race could have more genetic variation between them than individuals from different races. Race is therefore not a particularly useful category to use when searching for the genetics of biological traits or even medical vulnerabilities, despite widespread assumptions.

Most researchers who examine genetic differences between populations take care to point out that the differences they observe reflect the geographic origins, reproductive history and migrations of these groups, not markers of some essential differences between them.

However, some researchers have asked whether the taboo on the genetics of race has become so severe that it bars legitimate research. In 2005, for instance, geneticist Bruce Lahn of the University of Chicago in Illinois published studies3,4 suggesting that variants of two brain-development genes possibly linked to intelligence are evolving differently in white Europeans and African ethnic groups. This provoked a wave of worried comments by scientists about how the studies might be interpreted. Among those who voiced concerns was then-director of the US National Human Genome Research Institute Francis Collins, now director of the National Institutes of Health (NIH) in Bethesda, Maryland.

Lahn and his co-authors eventually found that the gene variants under selection were not linked to elevated intelligence5. But that report garnered little attention compared with the explosive studies that came before it. Lahn says he felt “ambushed” during the debate over his findings. At meetings, even his co-authors did not defend him. “My friends said nothing,” he says.

Some argue that Lahn should have been more cautious. “Science always plays out in a certain socio-political context, and you have to look at the consequences of how the science might play out,” says John Horgan, a journalist who has written widely on the societal implications of science. “Research on race and intelligence is much more prone to supporting racist ideas about the inferiority of certain groups, which plays into racist policies.” Horgan says that institutional review boards should ban or seriously question proposed studies on race and IQ.

Lahn no longer works on the genetics of race and has urged researchers to have a more transparent discussion about whether such studies should proceed at all. “Given the history of the way race has been used in this country, maybe the research shouldn't be encouraged because it touches too many raw nerves. I'm OK with that,” he says. “But I'm not OK with being ambushed by political discussions masquerading as scientific discussions.”

3 VIOLENCE

Taboo level: MILD

A decade ago, forensic psychiatrist Tracy Gunter of Indiana University in Indianapolis was spending her time trying to help people to overcome the behavioural and substance-abuse disorders that had led to their entanglement in the criminal-justice system. But it was becoming increasingly clear to her that once a client fell into an abuse–crime spiral, it was very difficult to bring them back.

It was around this time that researchers reported that people with a certain version of a gene called monoamine oxidase A (MAOA) have some protection from the effects of childhood abuse6. Other people who express low levels of the protein it encodes are more likely to commit crimes if mistreated.

Gunter switched fields to work in behavioural genetics, hoping to find ways to identify and preemptively treat high-risk individuals. She soon found her work complicated by the difficulty of defining criminal behaviour precisely; the impossibility of separating environmental and innate influences; and, again, the emerging consensus that behaviour is influenced by numerous small genetic factors. Ten years on, she says, “the simplistic notions I had about behavioural genetics when I started this work are not true”.

Despite these caveats — and the fact that some studies have failed to replicate the original MAOA finding7 — some lawyers have used MAOA gene tests, combined with history of childhood abuse or life stress, to try to mitigate sentences.

In 2009, such testing led to a lesser charge for a Tennessee man who killed his wife's friend, and it convinced a judge in Italy to reduce a murderer's sentence by one year (see Nature http://doi.org/cttbjt; 2009 ). But juries are often underwhelmed by genetic testimony: in the United States in 2008, for instance, defence lawyers attempted to convince a jury to be lenient towards a boy who had shot a bus driver. They presented evidence that the boy had a variant of a promoter of a serotonin transporter gene, SLC6A4, that is linked to depression in people under stress. The jury found the boy guilty of first-degree murder anyway. Outcomes are mixed, Farahany says, perhaps because the research is so oblique. “It doesn't seem to be enough to persuade judges or juries to change guilt or sentencing,” she says. William Bernet, a forensic psychiatrist in Nashville, Tennessee, adds that, “a genetic result does not directly cause a person to behave in a particular way. Juries seem to understand this”.

That may change as the science progresses, but so far genetics has held no more sway than conventional mitigating factors, which often include the milieu in which a person grew up.

Those two domains are coming together as researchers look for more clues to the environmental factors that interact with genetics in influencing behaviour. Gunter was part of a team that showed that certain epigenetic modifications on the MAOA gene are linked to substance abuse in adult women8, and these modifications are influenced by a history of smoking. “Every year that I work in this field has been a lesson that it's not just genes or environment,” she now says. “It's genes and environment that matter.”

Scientists continue to look at the genetics of violence, and of conditions such as psychopathy, although the tension between those who focus on just genes and those looking for genetic and environmental contributors is high, says James Tabery, a philosopher at the University of Utah in Salt Lake City. “My sense is that we're in a holding pattern; it's not clear what's going to happen next” — specifically because not many genes have been linked to violence and attempts to replicate the MAOA findings have produced mixed results.

4 SEXUALITY

Taboo level: MILD

Sometimes, shifting political winds can destigmatize research. In 1993, for instance, geneticist Dean Hamer, then at the US National Cancer Institute in Bethesda, encountered a firestorm of criticism from political conservatives when he published a report suggesting that a region of the X chromosome might be linked to homosexuality9. Scientists also criticized some aspects of his work. Today, studies on the genetics of sexual orientation have been embraced by the US gay community. The successful campaign to strike down a 2008 California ballot measure that banned same-sex marriage enlisted evidence that homosexuality has some basis in genetics. And the NIH has designated research on lesbian, gay, bisexual, transgender and intersex people a priority. “The tables have turned tremendously,” says geneticist Eric Vilain, director of the Institute for Society and Genetics at the University of California, Los Angeles.

But that does not mean that all research into the genetics of sexuality will be equally welcome, he adds. Vilain, for example, wants to study the epigenetics of homosexuality, in search of environmental influences that might affect the trait. The work hasn't been funded, but he predicts that if it is, it could upset some gay rights activists who have seen their cause benefit from the 'hardwiring' theory. He is keeping his fingers crossed. “I hope that now that there have been significant social advances, that scientists can do their work in peace,” he says.

Such complexities are unavoidable in a democratic society in which citizens have a say on how public money is spent. Researchers must acknowledge that and take part in the broader conversation about the kinds of topics they want to pursue, Farahany says. “You hear this refrain in lots of areas of science, that because people will misuse science we shouldn't engage in scientific inquiry. I think that gets it backwards. If we're worried that people will misuse it, we need to create safeguards — and an open public dialogue that ensures responsible use.” That, rather than censoring science or ignoring its implications, is perhaps the only way that Vilain and other researchers will get their wish: to do their work in peace.

Nature special: The human genome at ten