A study attacking some of the most prominent research in the burgeoning field of social neuroscience is flawed and unfair, according to top scientists who have been accused of overselling their results.

Brain imaging is used to assess neural mechanisms in social behaviour. Credit: SocialBrainLab, Univ. Med. Center Groningen

Social neuroscience is the study of the neuro­biological mechanisms underlying social behaviour. The field frequently uses functional magnetic resonance imaging (fMRI) to reveal which brain areas are activated while a subject is exposed to specific social interactions — for example, situations that may evoke jealousy or the perception of unfairness.

But a no-holds-barred paper1, accepted for publication in Perspectives on Psychological Science and already circulating widely on the Internet, claims that many studies in the field are worthless because brain imaging data have been poorly analysed.

The paper was written by Edward Vul, a PhD student supervised by neuroscientist Nancy Kanwisher at the Massachusetts Institute of Technology in Cambridge, along with psychologists at the University of California, San Diego, including Harold Pashler.

The paper has touched a nerve: brain imaging studies were derided by some as "the new phrenology" when they became common 15 years ago, and interpretations of their highly complex data were denounced as naive. But those directly attacked say they are familiar with, and avoid, the pitfalls.

Vul and his co-authors say they wrote the paper because they were concerned by what they considered to be the "implausibly high correlations" reported between brain activation and particular forms of behaviour, and the lack of methodological details provided. So they selected 54 papers in social neuroscience and sent a brief questionnaire to the authors requesting details of their analyses.

They concluded that in a 'red list' of 31 cases — often in high-profile journals, including Nature and Science — the authors made fundamental errors in data handling and statistics.

They particularly criticize a 'non-independence error', in which bias is introduced by selecting data using a first statistical test and then applying a second non-independent statistical test to those data. This error, they say, arises from selecting small volumes of the brain, called voxels, on the basis of their high correlation with a psychological response, and then going on to report the magnitude of that correlation. "At present, all studies performed using these methods have large question marks over them," they write.

In a rebuttal2, four authors of different red-list papers explain that there was no non-independence error because calculating the size of the correlation is not a statistical test.

Appropriate corrections ensure that the correlations between the selected voxels and psychological responses are likely to be real, and not noise, they add. And the strictness of the correction means that those correlations are necessarily less frequent and of higher magnitude — a situation far from implausible.

Vul and his colleagues also claim that the magnitude of correlation is limited by the average reliability of fMRI data generally. Not so, the accused respond: although the reliability of fMRI is very variable, the upper ceiling on data quality is extremely high.

The swift rebuttal was prompted by scientists' alarm at the speed with which the accusations have spread through the community. The provocative title — 'Voodoo correlations in social neuroscience' — and iconoclastic tone have attracted coverage on many blogs, including that of Newsweek. Those attacked say they have not had the chance to argue their case in the normal academic channels.

"I first heard about this when I got a call from a journalist," comments neuroscientist Tania Singer of the University of Zurich, Switzerland, whose papers on empathy are listed as examples of bad analytical practice. "I was shocked — this is not the way that scientific discourse should take place." Singer says she asked for a discussion with the authors when she received the questionnaire, to clarify the type of information needed, but got no reply.

"We didn't disclose all our potential criticisms before asking these people to tell us things that should have already been in their method sections," says an unrepentant Vul. "Would they have described their methods differently if we had?"

They would indeed, says bio­psychologist Turhan Canli of Stony Brook University, New York, who has four papers on the red list. He argues that the questionnaire, which was billed as taking no more than a few minutes to complete, was not able to capture the rationale for using particular analyses.

The article is scheduled for publication in September, alongside one or more replies. But the accused scientists are concerned that the impression now being established through media reports will be hard to shake after the nine-month delay. "We are not worried about our close colleagues, who will understand the arguments. We are worried that the whole enterprise of social neuroscience falls into disrepute," says neuroscientist Chris Frith of University College London, whose Nature paper3 on response to perceived fairness was called into question.

"On the other hand, we all agree that there is a kernel of truth in what Vul and his colleagues write about some of the literature being shaky," adds Christian Keysers of the University of Groningen in the Netherlands, whose 2007 paper in NeuroImage on empathy was highlighted4. "We can never be reminded often enough of the importance of good statistical practice."