Published online 17 March 2010 | Nature | doi:10.1038/news.2010.129

News

Reproducibility of brainscan studies questioned

Some magnetic resonance imaging studies could be less reliable than has been presumed.

MRI brain scanAnother study has raised doubts over the validity of some fMRI findings.A. Herreid / iStockphoto

Brain-scanning studies that claim to link facets of personality, memory and emotion to specific brain activity may not be easy to reproduce, a new study concludes.

The analysis, accepted for publication in Annals of the New York Academy of Sciences1, questions the reliability of certain functional magnetic resonance imaging (fMRI) studies, which image brain activity while people perform mental tasks.

Nobody disputes the validity of the basic method, which uses MRI scanners to monitor the brain's oxygen usage with millimetre-scale precision. But the new study, led by cognitive neuroscientist Michael Miller of the University of California, Santa Barbara, casts doubt on some studies that have reported 80 or 90% correlations between specific regions of brain activity and personality traits and emotions, such as laboratory-induced feelings of rejection.

Reviewing 63 journal articles describing fMRI studies in which tests had been performed twice on the same people at different times, it concludes that, on average, there is only a 50% correlation between the brain regions that light up in the first and second tests in the same person.

"The thrust of our experiment is that it challenges the assumptions of how reliable the results from any one fMRI study may be," says Craig Bennett, a postdoctoral researcher in cognitive neuroscience at the University of California, Santa Barbara, and the co-author of the study. "The end question we want to know is: if you did your experiment again, would the results — and the conclusions — be the same?"

“If you get the same results twice it is a very powerful statement.”


The new study is the third in little more than a year to challenge the reliability of some fMRI studies.

Early last year, a team led by Edward Vul, an experimental psychologist at the Massachusetts Institute of Technology in Cambridge, argued that high correlations between brain activity and inherently slippery concepts such as personality and emotion just aren't possible(see 'Brain imaging studies under fire')2.

Vul's study provoked what Bennett describes as a "Wild West shoot-out" among fMRI researchers — a battle further inflamed when a team led by Nikolaus Kriegeskorte of the Laboratory of Brain and Cognition at the US National Institute of Mental Health in Bethesda, Maryland, published a study showing that many fMRI papers in top-tier publications unintentionally included biased data (see 'Brain imaging skewed').

Bennett says that the new study is not an indictment of fMRI research in general. "If you get the same results twice," he says, "it is a very powerful statement."

Repeat performance

Other scientists' reactions are mixed. Kriegeskorte doesn't think that the new study will stir up much controversy because it isn't claiming to have found major methodological errors. Rather, it's a "useful review" that gives a better understanding of the inherent level of variability of fMRI. "The reliability of a technique like fMRI is not 'all or none'," Kriegeskorte said in an e-mail. "Measurements are always noisy, thus data are always somewhat unreliable."

Alexander Stevens, a neuroscientist at Oregon Health and Science University in Portland, Oregon, calls the study overdue, although he thinks that it is more relevant to researchers looking at whole-brain activity than to those examining specific regions. In the latter cases, he says, "you can essentially ignore what happens elsewhere in the brain".

“Measurements are always noisy, thus data are always somewhat unreliable.”


Dardo Tomasi, head of the MRI laboratory at Brookhaven National Laboratory in Upton, New York, believes the finding is an important wake-up call for MRI researchers. "I think that most in the field do not know anything about the reliability of the studies," he says. "We need to think more about if a particular result is reproducible or not."

What's needed, he says, is for researchers to make sure they have enough research subjects for the results to have statistical power, plus experimental protocols that control for such things as the time of day, drug use (or abuse), age and gender. Careful use of the fMRI scanner is also important, he says. "You can get significant changes just by a rotation of two degrees of the head. Some people are not aware of these things."

But Karl Friston, scientific director of the Wellcome Trust Centre for Neuroimaging at University College London, believes the new study will have little impact. "I am sure the authors have a point," he said in an e-mail, but it won't affect the way many researchers do their experiments.

ADVERTISEMENT

Basically, he says, questions about reliability are better answered by looking to see if repeated studies get the same result. "The measure of reliability the authors are searching for," he says, "can be found in the 250,000 scientific reports on statistically significant brain activations in the peer-reviewed literature."

Bennett concedes that many fMRI studies are finding valid correlations between brain activity and behaviour. Still, he says, there is extensive variability in brain activity both between people and within the same person. "These factors affect the precision and have to be taken into account," he says. 

  • References

    1. Bennett, C. M. & Miller, M. B. Ann. NY Acad Sci. (in the press).
    2. Vul, E. et al. Perspect. Psychol. Sci. 4, 274-290 (2009).

Comments

If you find something abusive or inappropriate or which does not otherwise comply with our Terms or Community Guidelines, please select the relevant 'Report this comment' link.

Comments on this thread are vetted after posting.

Commenting is now closed.