Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reproducibility of brainscan studies questioned

Subjects

Some magnetic resonance imaging studies could be less reliable than has been presumed.

Another study has raised doubts over the validity of some fMRI findings. Credit: A. Herreid / iStockphoto

Brain-scanning studies that claim to link facets of personality, memory and emotion to specific brain activity may not be easy to reproduce, a new study concludes.

The analysis, accepted for publication in Annals of the New York Academy of Sciences1, questions the reliability of certain functional magnetic resonance imaging (fMRI) studies, which image brain activity while people perform mental tasks.

Nobody disputes the validity of the basic method, which uses MRI scanners to monitor the brain's oxygen usage with millimetre-scale precision. But the new study, led by cognitive neuroscientist Michael Miller of the University of California, Santa Barbara, casts doubt on some studies that have reported 80 or 90% correlations between specific regions of brain activity and personality traits and emotions, such as laboratory-induced feelings of rejection.

Reviewing 63 journal articles describing fMRI studies in which tests had been performed twice on the same people at different times, it concludes that, on average, there is only a 50% correlation between the brain regions that light up in the first and second tests in the same person.

"The thrust of our experiment is that it challenges the assumptions of how reliable the results from any one fMRI study may be," says Craig Bennett, a postdoctoral researcher in cognitive neuroscience at the University of California, Santa Barbara, and the co-author of the study. "The end question we want to know is: if you did your experiment again, would the results — and the conclusions — be the same?"

If you get the same results twice it is a very powerful statement. ,

The new study is the third in little more than a year to challenge the reliability of some fMRI studies.

Early last year, a team led by Edward Vul, an experimental psychologist at the Massachusetts Institute of Technology in Cambridge, argued that high correlations between brain activity and inherently slippery concepts such as personality and emotion just aren't possible(see 'Brain imaging studies under fire')2.

Vul's study provoked what Bennett describes as a "Wild West shoot-out" among fMRI researchers — a battle further inflamed when a team led by Nikolaus Kriegeskorte of the Laboratory of Brain and Cognition at the US National Institute of Mental Health in Bethesda, Maryland, published a study showing that many fMRI papers in top-tier publications unintentionally included biased data (see 'Brain imaging skewed').

Bennett says that the new study is not an indictment of fMRI research in general. "If you get the same results twice," he says, "it is a very powerful statement."

Repeat performance

Other scientists' reactions are mixed. Kriegeskorte doesn't think that the new study will stir up much controversy because it isn't claiming to have found major methodological errors. Rather, it's a "useful review" that gives a better understanding of the inherent level of variability of fMRI. "The reliability of a technique like fMRI is not 'all or none'," Kriegeskorte said in an e-mail. "Measurements are always noisy, thus data are always somewhat unreliable."

Alexander Stevens, a neuroscientist at Oregon Health and Science University in Portland, Oregon, calls the study overdue, although he thinks that it is more relevant to researchers looking at whole-brain activity than to those examining specific regions. In the latter cases, he says, "you can essentially ignore what happens elsewhere in the brain".

Measurements are always noisy, thus data are always somewhat unreliable. ,

Dardo Tomasi, head of the MRI laboratory at Brookhaven National Laboratory in Upton, New York, believes the finding is an important wake-up call for MRI researchers. "I think that most in the field do not know anything about the reliability of the studies," he says. "We need to think more about if a particular result is reproducible or not."

What's needed, he says, is for researchers to make sure they have enough research subjects for the results to have statistical power, plus experimental protocols that control for such things as the time of day, drug use (or abuse), age and gender. Careful use of the fMRI scanner is also important, he says. "You can get significant changes just by a rotation of two degrees of the head. Some people are not aware of these things."

But Karl Friston, scientific director of the Wellcome Trust Centre for Neuroimaging at University College London, believes the new study will have little impact. "I am sure the authors have a point," he said in an e-mail, but it won't affect the way many researchers do their experiments.

Basically, he says, questions about reliability are better answered by looking to see if repeated studies get the same result. "The measure of reliability the authors are searching for," he says, "can be found in the 250,000 scientific reports on statistically significant brain activations in the peer-reviewed literature."

Bennett concedes that many fMRI studies are finding valid correlations between brain activity and behaviour. Still, he says, there is extensive variability in brain activity both between people and within the same person. "These factors affect the precision and have to be taken into account," he says.

References

  1. Bennett, C. M. & Miller, M. B. Ann. NY Acad Sci. (in the press).

  2. Vul, E. et al. Perspect. Psychol. Sci. 4, 274-290 (2009).

    Article  Google Scholar 

Download references

Authors

Related links

Related links

Related external links

Ed Vul's website, on Voodoo Correlations

Miller and Bennett's paper

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Lovett, R. Reproducibility of brainscan studies questioned. Nature (2010). https://doi.org/10.1038/news.2010.129

Download citation

  • Published:

  • DOI: https://doi.org/10.1038/news.2010.129

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing