Perspective abstract
Nature Neuroscience 12, 535 - 540 (2009)
Published online: 26 April 2009 | doi:10.1038/nn.2303
Circular analysis in systems neuroscience: the dangers of double dipping
Nikolaus Kriegeskorte1, W Kyle Simmons1, Patrick S F Bellgowan1 & Chris I Baker1
Abstract
A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.
- Laboratory of Brain and Cognition, US National Institute of Mental Health, Bethesda, Maryland, USA.
Correspondence to: Nikolaus Kriegeskorte1 e-mail: nikokriegeskorte@gmail.com
Correspondence to: Chris I Baker1 e-mail: bakerchris@mail.nih.gov
MORE ARTICLES LIKE THIS
These links to content published by NPG are automatically generated.
RESEARCH
Odor quality coding and categorization in human posterior piriform cortexNature Neuroscience Article (01 Jul 2009)
Predicting the orientation of invisible stimuli from activity in human primary visual cortexNature Neuroscience Article (01 May 2005)
Does the fusiform face area contain subregions highly selective for nonfaces?Nature Neuroscience Correspondence (01 Jan 2007)
Decoding the visual and subjective contents of the human brainNature Neuroscience Article (01 May 2005)
