Functional magnetic resonance imaging poses a tricky problem in sorting the wheat from the chaff. Credit: SPL

Nearly half of the neuroimaging studies published in prestige journals in 2008 contain unintentionally biased data that could distort their scientific conclusions, according to scientists at the National Institute of Mental Health in Bethesda, Maryland.

Experts in the field contacted by Nature have been taken aback by the extent of the methodological errors getting through the supposedly strict peer-review systems of the journals in question.

Nikolaus Kriegeskorte, Chris Baker and their colleagues analysed 134 functional magnetic resonance imaging (fMRI) studies published last year in five top journals — Nature, Science, Nature Neuroscience, Neuron and The Journal of Neuroscience. The survey, published in Nature Neuroscience on 26 April (N. Kriegeskorte, W. K. Simmons, P. S. F. Bellgowan and C. I. Baker Nature Neurosci. 12, 535–540; 2009), found that 57 of these papers included at least one so-called 'non-independent selective analysis'; another 20 may also have done so, but did not provide enough information to confirm suspicions.

The non-independence of the analysis lies in using the same data to set up the conditions to test a hypothesis, then to confirm it. "We are not saying that the papers draw wrong conclusions, because in some cases the error will not have been critical," says Baker. "But in other cases we don't know, and this creates an ambiguity."

"It is a poor reflection on the quality of peer review of prestige journals — they really need to up their game in terms of rigour," says Karl Friston, scientific director of the Wellcome Trust Centre for Neuroimaging at University College London.

Brain imaging provides vast quantities of data in the form of voxels (three-dimensional pixels), from the entire brain. Neuroscientists sometimes focus on an area of interest by searching for voxels that are activated when subjects perform different tasks in an experiment — for example, looking at a face or an inanimate object.

The issue of selection bias doesn't require special understanding of statistics, just the following of good practice — it is not rocket science. Karl Friston , Wellcome Trust Centre for Neuroimaging, University College London

But fMRI data are intrinsically very noisy, producing many 'false voxels'. Problems arise when researchers use the same data to select a particular brain region and then to quantify the experimental effects there — for example, by asking how much more strongly the region responds to a face compared with an inanimate object.

"It is crucial to analyse your results with a set of data that are independent of that used in the earlier selection process," says Chris Baker. "It is even OK to split your total data and use one half to select the voxels, and the other to further analyse the response in these voxels."

A similar type of error has been addressed by Edward Vul of the Massachussetts Institute of Technology in Cambridge and his colleagues (E. Vul, C. Harris, P. Winkielman and H. Pashler, Perspect. Psychol. Sci. 4, 274–290; 2009). A preprint of their research caused uproar in the field earlier this year by referring to 'voodoo correlations' and naming labs it considered guilty of circular analysis (see Nature 457, 245; 2009).

In contrast, the study by Kriegeskorte and Baker does not single out any researchers. "We didn't name names because the error is just too common," says Baker. "And we saw no reason to be personal — our idea was to highlight a problem so people are less likely to fall into the trap."

"This new paper is less controversial, but potentially more worrying," says Friston. "The issue of selection bias doesn't require special understanding of statistics, just the following of good practice — it is not rocket science."

Baker points out that circularity errors creep into many areas of neuroscience. "It applies equally to single-unit electrophysiology, electroencephalography, gene microarray studies or even behavioural data," he says. But fMRI data are particularly vulnerable because of the complex analysis demanded by their huge volumes, and because so many untrained outsiders are entering the field. "For those of us with a few years of fMRI experience the issue is entirely passé, but there will always be a substantial minority on a steep learning curve," says Friston. "What surprised me is how frequent the errors are."

Baker notes that the increasing complexity of the data "probably leads people to take their eye off the ball so that the more fundamental aspects don't get taken care of".