A neuroscientist's fishy discovery earns him an Ig Nobel Prize and helps to improve data interpretation.
Craig Bennett, a postdoc in cognitive neuroscience at the University of California, Santa Barbara, has highlighted the prevalence of statistical errors in interpretations of functional magnetic resonance imaging (fMRI) data. He showed that such errors can lead to ambiguous and sometimes absurd results, such as registering brain activity in a dead salmon (C. M. Bennett et al. J. Serendipitous Unexpected Results 1, 1–5; 2010). On 20 September, Bennett accepted an Ig Nobel Prize — a riff on the Nobel prize that recognizes unusual research — for his work. He describes how the fishy findings have affected the field and his career.
Did you conduct a 'dead salmon' experiment?
No. We do a lot of calibration and testing before undertaking fMRI studies, to make sure that the scanner is working before we put a human into it to measure their brain activity. One of my PhD advisers and I made a game out of finding all kinds of things to scan, from a pumpkin to a game hen to a salmon. But there was no reason to dig into the results until about five years ago, when my other adviser was lecturing on false positives and asked me to find some good data. It was pure luck that we found them in the salmon's brain.
How did the story evolve from there?
False positives in fMRI studies had caused increasing concern in neuroscience. The dead salmon was a good example of how an improperly corrected threshold value — which separates the signal from noise — could compromise results. We submitted our findings as a poster to the 2009 Human Brain Mapping conference in San Francisco, California, and at first colleagues thought it was a joke. Ultimately they included it, but we realized that we would face challenges in trying to publish.
What sort of challenges?
We knew that the article could mean that our statistics would receive closer scrutiny. We have experienced some of that, but it keeps us on our toes. Our work has improved — we are meticulous about analyses.
What happened when you submitted the dead-salmon paper?
We tried to get it published in two major neuroimaging journals. One rejected it and the other sent it out for review. One reviewer said it was fantastic; the other gave us a hateful, livid review that sunk it. But less-mainstream journals were clamouring for the paper. We went with the Journal of Serendipitous and Unexpected Results, which led to other publications and fostered a debate on statistical errors.
Has the field changed?
In the salmon paper, we did a meta-analysis of major journal articles and found that 25–40% of neuroimaging papers that we studied were not properly correcting for threshold values. We surveyed a couple of journals last year as a follow-up, and found that fewer than 10% of people are now using incorrect statistics. The decline is not all attributable to the salmon paper, but it is all progress. We gave the field a kick in the pants — and I've heard that a lot of groups reviewed the paper in lab meetings.
Are you concerned that being an Ig Nobel winner might harm your career prospects?
We were a tad worried, given that the Ig Nobel's original criterion was to recognize research that “cannot or should not be replicated”. We didn't feel that the salmon fell into that category. But when we got a call from Marc Abrahams, founder of the awards, he described their new context: recognizing science that “first makes people laugh, then makes them think”. That seemed appropriate, and we worried less.
What is next for you?
I am keeping track of job openings while I complete some research. I'm convinced that a big part of a successful job search is shaping your personal story. You need a coherent thread of research. I study how teens and adults make decisions, and I am interested in how best to analyse fMRI data to answer these questions. I just co-authored a big review on fMRI reliability, and I will make clear in interviews that all my work tells a coherent story — otherwise I would just be the 'dead fish' guy.
Interview by Virginia Gewin