Access
This article is part of Nature's premium content.
Published online 27 April 2009 | Nature 458, 1087 (2009) | doi:10.1038/4581087a
News
Brain imaging skewed
Double dipping of data magnifies errors in functional MRI scans.
Nearly half of the neuroimaging studies published in prestige journals in 2008 contain unintentionally biased data that could distort their scientific conclusions, according to scientists at the National Institute of Mental Health in Bethesda, Maryland.
Experts in the field contacted by Nature have been taken aback by the extent of the methodological errors getting through the supposedly strict peer-review systems of the journals in question.
To read this story in full you will need to login or make a payment (see right).
Comments
Reader comments are usually moderated after posting. If you find something offensive or inappropriate, you can speed this process by clicking 'Report this comment' (or, if that doesn't work for you, email webadmin@nature.com). For more controversial topics, we reserve the right to moderate before comments are published.
I want to add this information about brain scans: http://sciencenow.sciencemag.org/cgi/content/full/2009/121/1 if you don't have access you can read here: http://www.vetscite.org/publish/items/005021/index.html
Baker says "we saw no reason to be personal". Has he forgotten that science is supposed to be a public and self-correcting enterprise? To view noting flaws in published literature as "being personal" is a very unfortunate and unhelpful way to look at things. As a result of the authors' decision, no one knows which fMRI papers from 2008 can be believed, and which cannot. On a lighter note: another good reason to have published the list would be to see whether Friston's seemingly rather self-congratulatory tone (as when he says "For those of us with a few years of fMRI experience the issue is entirely passé, but there will always be a substantial minority on a steep learning curve") is justified or not. So none of the defective papers from 2008 were written by anyone on his staff? No, they must have all reached very lofty elevations on those learning curves. :-)
This means that fMRI is not a black box and it should be used with knowledge. Besides, there?s a need for a deeper reflection on the use of brain imaging technologies, its role in society and the presentation of the results in the media (fMRI often appears as the ultimate explainer of our brain functions). The scientific community should be open to discussion, in order not to lose its credibility. Key stakeholders discussed these issues in a recent meeting (see www.neuromedia.eu/NewsData.aspx?IdNews=108&IdType=261&type=Actual and www.neuromedia.eu/NewsData.aspx?IdNews=104&IdType=261&type=Actual).
Functional magnetic resonance imaging and optical imaging as well as the subtractive water method used with positron emission tomography do not detect changes in neural activity directly. They permit us to map local changes in cerebral blood flow from which neural function is inferred. Inferences provide ample room for ambiguity. In normal physiological circumstances, cerebral blood flow is tightly associated with the neural and glial demand for glucose. The electrical discharge activity of neurons and the uptake of glutamate by astrocytes are known to increase glucose utilization. However, the mechanisms that couple blood flow to glucose utilization are not yet understood. Molecular pathways involving nitric oxide, adenosine and glutamate in as much as neurogenic control are potential candidates. Adding to this ambiguity, the imaging methods above require spatial and temporal averaging of the recorded blood flow signals. Changes in flow are considered significant above probabilistic thresholds based on statistical assumptions. The identified regions may contain hundreds of thousands of neurons, and the methods do not reveal the neurons, the activity of which ultimately influenced the blood flow. Exploration with micro-electrode recordings of single- or multi-neuron activity may be of limited use. The neurons driving the flow may be widely dispersed and, thus, easily missed. Therefore, the imaging of task-related changes in blood flow may help us identify the regions of cerebral cortex involved, and the statistical methods used are crucial for correct identification (hence this article). Yet, how the neurons in these regions are engaged in the examined process must remain hypothetical and may be difficult to test in the human condition. I have written more about functional brain imaging here: http://brainmindinst.blogspot.com/2009/03/fmri-mental-processing.html
2Peter Melzer's "the imaging methods above require spatial and temporal averaging of the recorded blood flow signals". I would disagree -- imaging methods are not the ones to be blamed, it is the analysis strategies which are at fault. Single trial analysis strategies became more popular lately, and for an example of single-trial analysis strategy applied to 4 different imaging (and not) neural data modalities, I would refer the reader to "PyMVPA: a unifying approach to the analysis of neuroscientific data" ( http://dx.doi.org/10.3389/neuro.11.003.2009 ).
The situation is may be a bit more complicated then presented here, which is possibly one reason why it does get by reviewers so often. The core of the problem is not the 'double dipping' in itself, but the fact that the first dip skews the result for the second dip. While you indeed can not go wrong if you use independent data for pre-selection and hypothesis testing, you can use the same data for both processes, as long as the selection criteria are not biased in regard to the hypothesis being tested.
Example: If you want to compare the fMRI signal level in a certain brain region between two task conditions, you can preselect a group of voxels in that region that reach a certain threshhold in EITHER of the two tasks, compared to a baseline. Although this is by definition 'double dipping' it first dip does not bias in regard to the hypothesis, so it is statistically not incorrect.
It may not be rocket science, but it still may be a bit more tricky then suggested...:)