Key Points

Low statistical power undermines the purpose of scientific research; it reduces the chance of detecting a true effect.

Perhaps less intuitively, low power also reduces the likelihood that a statistically significant result reflects a true effect.

Empirically, we estimate the median statistical power of studies in the neurosciences is between ∼8% and ∼31%.

We discuss the consequences of such low statistical power, which include overestimates of effect size and low reproducibility of results.

There are ethical dimensions to the problem of low power; unreliable research is inefficient and wasteful.

Improving reproducibility in neuroscience is a key priority and requires attention to wellestablished, but often ignored, methodological principles.

We discuss how problems associated with low power can be addressed by adopting current bestpractice and make clear recommendations for how to achieve this.
Abstract
A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to wellestablished but often ignored methodological principles.
Similar content being viewed by others
Main
It has been claimed and demonstrated that many (and possibly most) of the conclusions drawn from biomedical research are probably false^{1}. A central cause for this important problem is that researchers must publish in order to succeed, and publishing is a highly competitive enterprise, with certain kinds of findings more likely to be published than others. Research that produces novel results, statistically significant results (that is, typically p < 0.05) and seemingly 'clean' results is more likely to be published^{2,3}. As a consequence, researchers have strong incentives to engage in research practices that make their findings publishable quickly, even if those practices reduce the likelihood that the findings reflect a true (that is, nonnull) effect^{4}. Such practices include using flexible study designs and flexible statistical analyses and running small studies with low statistical power^{1,5}. A simulation of genetic association studies showed that a typical dataset would generate at least one false positive result almost 97% of the time^{6}, and two efforts to replicate promising findings in biomedicine reveal replication rates of 25% or less^{7,8}. Given that these publishing biases are pervasive across scientific practice, it is possible that false positives heavily contaminate the neuroscience literature as well, and this problem may affect at least as much, if not even more so, the most prominent journals^{9,10}.
Here, we focus on one major aspect of the problem: low statistical power. The relationship between study power and the veracity of the resulting finding is underappreciated. Low statistical power (because of low sample size of studies, small effects or both) negatively affects the likelihood that a nominally statistically significant finding actually reflects a true effect. We discuss the problems that arise when lowpowered research designs are pervasive. In general, these problems can be divided into two categories. The first concerns problems that are mathematically expected to arise even if the research conducted is otherwise perfect: in other words, when there are no biases that tend to create statistically significant (that is, 'positive') results that are spurious. The second category concerns problems that reflect biases that tend to cooccur with studies of low power or that become worse in small, underpowered studies. We next empirically show that statistical power is typically low in the field of neuroscience by using evidence from a range of subfields within the neuroscience literature. We illustrate that low statistical power is an endemic problem in neuroscience and discuss the implications of this for interpreting the results of individual studies.
Low power in the absence of other biases
Three main problems contribute to producing unreliable findings in studies with low power, even when all other research practices are ideal. They are: the low probability of finding true effects; the low positive predictive value (PPV; see Box 1 for definitions of key statistical terms) when an effect is claimed; and an exaggerated estimate of the magnitude of the effect when a true effect is discovered. Here, we discuss these problems in more detail.
First, low power, by definition, means that the chance of discovering effects that are genuinely true is low. That is, lowpowered studies produce more false negatives than highpowered studies. When studies in a given field are designed with a power of 20%, it means that if there are 100 genuine nonnull effects to be discovered in that field, these studies are expected to discover only 20 of them^{11}.
Second, the lower the power of a study, the lower the probability that an observed effect that passes the required threshold of claiming its discovery (that is, reaching nominal statistical significance, such as p < 0.05) actually reflects a true effect^{1,12}. This probability is called the PPV of a claimed discovery. The formula linking the PPV to power is:
where (1 − β) is the power, β is the type II error, α is the type I error and R is the prestudy odds (that is, the odds that a probed effect is indeed nonnull among the effects being probed). The formula is derived from a simple twobytwo table that tabulates the presence and nonpresence of a nonnull effect against significant and nonsignificant research findings^{1}. The formula shows that, for studies with a given prestudy odds R, the lower the power and the higher the type I error, the lower the PPV. And for studies with a given prestudy odds R and a given type I error (for example, the traditional p = 0.05 threshold), the lower the power, the lower the PPV.
For example, suppose that we work in a scientific field in which one in five of the effects we test are expected to be truly nonnull (that is, R = 1 / (5 − 1) = 0.25) and that we claim to have discovered an effect when we reach p < 0.05; if our studies have 20% power, then PPV = 0.20 × 0.25 / (0.20 × 0.25 + 0.05) = 0.05 / 0.10 = 0.50; that is, only half of our claims for discoveries will be correct. If our studies have 80% power, then PPV = 0.80 × 0.25 / (0.80 × 0.25 + 0.05) = 0.20 / 0.25 = 0.80; that is, 80% of our claims for discoveries will be correct.
Third, even when an underpowered study discovers a true effect, it is likely that the estimate of the magnitude of that effect provided by that study will be exaggerated. This effect inflation is often referred to as the 'winner's curse'^{13} and is likely to occur whenever claims of discovery are based on thresholds of statistical significance (for example, p < 0.05) or other selection filters (for example, a Bayes factor better than a given value or a falsediscovery rate below a given value). Effect inflation is worst for small, lowpowered studies, which can only detect effects that happen to be large. If, for example, the true effect is mediumsized, only those small studies that, by chance, overestimate the magnitude of the effect will pass the threshold for discovery. To illustrate the winner's curse, suppose that an association truly exists with an effect size that is equivalent to an odds ratio of 1.20, and we are trying to discover it by performing a small (that is, underpowered) study. Suppose also that our study only has the power to detect an odds ratio of 1.20 on average 20% of the time. The results of any study are subject to sampling variation and random error in the measurements of the variables and outcomes of interest. Therefore, on average, our small study will find an odds ratio of 1.20 but, because of random errors, our study may in fact find an odds ratio smaller than 1.20 (for example, 1.00) or an odds ratio larger than 1.20 (for example, 1.60). Odds ratios of 1.00 or 1.20 will not reach statistical significance because of the small sample size. We can only claim the association as nominally significant in the third case, where random error creates an odds ratio of 1.60. The winner's curse means, therefore, that the 'lucky' scientist who makes the discovery in a small study is cursed by finding an inflated effect.
The winner's curse can also affect the design and conclusions of replication studies. If the original estimate of the effect is inflated (for example, an odds ratio of 1.60), then replication studies will tend to show smaller effect sizes (for example, 1.20), as findings converge on the true effect. By performing more replication studies, we should eventually arrive at the more accurate odds ratio of 1.20, but this may take time or may never happen if we only perform small studies. A common misconception is that a replication study will have sufficient power to replicate an initial finding if the sample size is similar to that in the original study^{14}. However, a study that tries to replicate a significant effect that only barely achieved nominal statistical significance (that is, p ∼ 0.05) and that uses the same sample size as the original study, will only achieve ∼50% power, even if the original study accurately estimated the true effect size. This is illustrated in Fig. 1. Many published studies only barely achieve nominal statistical significance^{15}. This means that if researchers in a particular field determine their sample sizes by historical precedent rather than through formal power calculation, this will place an upper limit on average power within that field. As the true effect size is likely to be smaller than that indicated by the initial study — for example, because of the winner's curse — the actual power is likely to be much lower. Furthermore, even if power calculation is used to estimate the sample size that is necessary in a replication study, these calculations will be overly optimistic if they are based on estimates of the true effect size that are inflated owing to the winner's curse phenomenon. This will further hamper the replication process.
Low power in the presence of other biases
Low power is associated with several additional biases. First, lowpowered studies are more likely to provide a wide range of estimates of the magnitude of an effect (which is known as 'vibration of effects' and is described below). Second, publication bias, selective data analysis and selective reporting of outcomes are more likely to affect lowpowered studies. Third, small studies may be of lower quality in other aspects of their design as well. These factors can further exacerbate the low reliability of evidence obtained in studies with low statistical power.
Vibration of effects^{13} refers to the situation in which a study obtains different estimates of the magnitude of the effect depending on the analytical options it implements. These options could include the statistical model, the definition of the variables of interest, the use (or not) of adjustments for certain potential confounders but not others, the use of filters to include or exclude specific observations and so on. For example, a recent analysis of 241 functional MRI (fMRI) studies showed that 223 unique analysis strategies were observed so that almost no strategy occurred more than once^{16}. Results can vary markedly depending on the analysis strategy^{1}. This is more often the case for small studies — here, results can change easily as a result of even minor analytical manipulations. In small studies, the range of results that can be obtained owing to vibration of effects is wider than in larger studies, because the results are more uncertain and therefore fluctuate more in response to analytical changes. Imagine, for example, dropping three observations from the analysis of a study of 12 samples because posthoc they are considered unsatisfactory; this manipulation may not even be mentioned in the published paper, which may simply report that only nine patients were studied. A manipulation affecting only three observations could change the odds ratio from 1.00 to 1.50 in a small study but might only change it from 1.00 to 1.01 in a very large study. When investigators select the most favourable, interesting, significant or promising results among a wide spectrum of estimates of effect magnitudes, this is inevitably a biased choice.
Publication bias and selective reporting of outcomes and analyses are also more likely to affect smaller, underpowered studies^{17}. Indeed, investigations into publication bias often examine whether small studies yield different results than larger ones^{18}. Smaller studies more readily disappear into a file drawer than very large studies that are widely known and visible, and the results of which are eagerly anticipated (although this correlation is far from perfect). A 'negative' result in a highpowered study cannot be explained away as being due to low power^{19,20}, and thus reviewers and editors may be more willing to publish it, whereas they more easily reject a small 'negative' study as being inconclusive or uninformative^{21}. The protocols of large studies are also more likely to have been registered or otherwise made publicly available, so that deviations in the analysis plans and choice of outcomes may become obvious more easily. Small studies, conversely, are often subject to a higher level of exploration of their results and selective reporting thereof.
Third, smaller studies may have a worse design quality than larger studies. Several small studies may be opportunistic experiments, or the data collection and analysis may have been conducted with little planning. Conversely, large studies often require more funding and personnel resources. As a consequence, designs are examined more carefully before data collection, and analysis and reporting may be more structured. This relationship is not absolute — small studies are not always of low quality. Indeed, a bias in favour of small studies may occur if the small studies are meticulously designed and collect highquality data (and therefore are forced to be small) and if large studies ignore or drop quality checks in an effort to include as large a sample as possible.
Empirical evidence from neuroscience
Any attempt to establish the average statistical power in neuroscience is hampered by the problem that the true effect sizes are not known. One solution to this problem is to use data from metaanalyses. Metaanalysis provides the best estimate of the true effect size, albeit with limitations, including the limitation that the individual studies that contribute to a metaanalysis are themselves subject to the problems described above. If anything, summary effects from metaanalyses, including power estimates calculated from metaanalysis results, may also be modestly inflated^{22}.
Acknowledging this caveat, in order to estimate statistical power in neuroscience, we examined neuroscience metaanalyses published in 2011 that were retrieved using 'neuroscience' and 'metaanalysis' as search terms. Using the reported summary effects of the metaanalyses as the estimate of the true effects, we calculated the power of each individual study to detect the effect indicated by the corresponding metaanalysis.
Methods. Included in our analysis were articles published in 2011 that described at least one metaanalysis of previously published studies in neuroscience with a summary effect estimate (mean difference or odds/risk ratio) as well as study level data on group sample size and, for odds/risk ratios, the number of events in the control group.
We searched computerized databases on 2 February 2012 via Web of Science for articles published in 2011, using the key words 'neuroscience' and 'metaanalysis'. All of the articles that were identified via this electronic search were screened independently for suitability by two authors (K.S.B. and M.R.M.). Articles were excluded if no abstract was electronically available (for example, conference proceedings and commentaries) or if both authors agreed, on the basis of the abstract, that a metaanalysis had not been conducted. Full texts were obtained for the remaining articles and again independently assessed for eligibility by two authors (K.S.B. and M.R.M.) (Fig. 2).
Data were extracted from forest plots, tables and text. Some articles reported several metaanalyses. In those cases, we included multiple metaanalyses only if they contained distinct study samples. If several metaanalyses had overlapping study samples, we selected the most comprehensive (that is, the one containing the most studies) or, if the number of studies was equal, the first analysis presented in the article. Data extraction was independently performed by K.S.B. and either M.R.M. or C.M. and verified collaboratively.
The following data were extracted for each metaanalysis: first author and summary effect size estimate of the metaanalysis; and first author, publication year, sample size (by groups), number of events in the control group (for odds/risk ratios) and nominal significance (p < 0.05, 'yes/no') of the contributing studies. For five articles, nominal study significance was unavailable and was therefore obtained from the original studies if they were electronically available. Studies with missing data (for example, due to unclear reporting) were excluded from the analysis.
The main outcome measure of our analysis was the achieved power of each individual study to detect the estimated summary effect reported in the corresponding metaanalysis to which it contributed, assuming an α level of 5%. Power was calculated using G^{*}Power software^{23}. We then calculated the mean and median statistical power across all studies.
Results. Our search strategy identified 246 articles published in 2011, out of which 155 were excluded after an initial screening of either the abstract or the full text. Of the remaining 91 articles, 48 were eligible for inclusion in our analysis^{24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71}, comprising data from 49 metaanalyses and 730 individual primary studies. A flow chart of the article selection process is shown in Fig. 2, and the characteristics of included metaanalyses are described in Table 1.
Our results indicate that the median statistical power in neuroscience is 21%. We also applied a test for an excess of statistical significance^{72}. This test has recently been used to show that there is an excess significance bias in the literature of various fields, including in studies of brain volume abnormalities^{73}, Alzheimer's disease genetics^{70,74} and cancer biomarkers^{75}. The test revealed that the actual number (349) of nominally significant studies in our analysis was significantly higher than the number expected (254; p < 0.0001). Importantly, these calculations assume that the summary effect size reported in each study is close to the true effect size, but it is likely that they are inflated owing to publication and other biases described above.
Interestingly, across the 49 metaanalyses included in our analysis, the average power demonstrated a clear bimodal distribution (Fig. 3). Most metaanalyses comprised studies with very low average power — almost 50% of studies had an average power lower than 20%. However, seven metaanalyses comprised studies with high (>90%) average power^{24,26,31,57,63,68,71}. These seven metaanalyses were all broadly neurological in focus and were based on relatively small contributing studies — four out of the seven metaanalyses did not include any study with over 80 participants. If we exclude these 'outlying' metaanalyses, the median statistical power falls to 18%.
Small sample sizes are appropriate if the true effects being estimated are genuinely large enough to be reliably observed in such samples. However, as small studies are particularly susceptible to inflated effect size estimates and publication bias, it is difficult to be confident in the evidence for a large effect if small studies are the sole source of that evidence. Moreover, many metaanalyses show smallstudy effects on asymmetry tests (that is, smaller studies have larger effect sizes than larger ones) but nevertheless use randomeffect calculations, and this is known to inflate the estimate of summary effects (and thus also the power estimates). Therefore, our power calculations are likely to be extremely optimistic^{76}.
Empirical evidence from specific fields
One limitation of our analysis is the underrepresentation of metaanalyses in particular subfields of neuroscience, such as research using neuroimaging and animal models. We therefore sought additional representative metaanalyses from these fields outside our 2011 sampling frame to determine whether a similar pattern of low statistical power would be observed.
Neuroimaging studies. Most structural and volumetric MRI studies are very small and have minimal power to detect differences between compared groups (for example, healthy people versus those with mental health diseases). A cl ear excess significance bias has been demonstrated in studies of brain volume abnormalities^{73}, and similar problems appear to exist in fMRI studies of the bloodoxygenleveldependent response^{77}. In order to establish the average statistical power of studies of brain volume abnormalities, we applied the same analysis as described above to data that had been previously extracted to assess the presence of an excess of significance bias^{73}. Our results indicated that the median statistical power of these studies was 8% across 461 individual studies contributing to 41 separate metaanalyses, which were drawn from eight articles that were published between 2006 and 2009. Full methodological details describing how studies were identified and selected are available elsewhere^{73}.
Animal model studies. Previous analyses of studies using animal models have shown that small studies consistently give more favourable (that is, 'positive') results than larger studies^{78} and that study quality is inversely related to effect size^{79,80,81,82}. In order to examine the average power in neuroscience studies using animal models, we chose a representative metaanalysis that combined data from studies investigating sex differences in water maze performance (number of studies (k) = 19, summary effect size Cohen's d = 0.49) and radial maze performance (k = 21, summary effect size d = 0.69)^{80}. The summary effect sizes in the two metaanalyses provide evidence for medium to large effects, with the male and female performance differing by 0.49 to 0.69 standard deviations for water maze and radial maze, respectively. Our results indicate that the median statistical power for the water maze studies and the radial maze studies to detect these medium to large effects was 18% and 31%, respectively (Table 2). The average sample size in these studies was 22 animals for the water maze and 24 for the radial maze experiments. Studies of this size can only detect very large effects (d = 1.20 for n = 22, and d = 1.26 for n = 24) with 80% power — far larger than those indicated by the metaanalyses. These animal model studies were therefore severely underpowered to detect the summary effects indicated by the metaanalyses. Furthermore, the summary effects are likely to be inflated estimates of the true effects, given the problems associated with small studies described above.
The results described in this section are based on only two metaanalyses, and we should be appropriately cautious in extrapolating from this limited evidence. Nevertheless, it is notable that the results are so consistent with those observed in other fields, such as the neuroimaging and neuroscience studies that we have described above.
Implications
Implications for the likelihood that a research finding reflects a true effect. Our results indicate that the average statistical power of studies in the field of neuroscience is probably no more than between ∼8% and ∼31%, on the basis of evidence from diverse subfields within neuroscience. If the low average power we observed across these studies is typical of the neuroscience literature as a whole, this has profound implications for the field. A major implication is that the likelihood that any nominally significant finding actually reflects a true effect is small. As explained above, the probability that a research finding reflects a true effect (PPV) decreases as statistical power decreases for any given prestudy odds (R) and a fixed type I error level. It is easy to show the impact that this is likely to have on the reliability of findings. Figure 4 shows how the PPV changes for a range of values for R and for a range of v alues for the average power in a field. For effects that are genuinely nonnull, Fig. 5 shows the degree to which an effect size estimate is likely to be inflated in initial studies — owing to the winner's curse phenomenon — for a range of values for statistical power.
The estimates shown in Figs 4,5 are likely to be optimistic, however, because they assume that statistical power and R are the only considerations in determining the probability that a research finding reflects a true effect. As we have already discussed, several other biases are also likely to reduce the probability that a research finding reflects a true effect. Moreover, the summary effect size estimates that we used to determine the statistical power of individual studies are themselves likely to be inflated owing to bias — our excess of significance test provided clear evidence for this. Therefore, the average statistical power of studies in our analysis may in fact be even lower than the 8–31% range we observed.
Ethical implications. Low average power in neuroscience studies also has ethical implications. In our analysis of animal model studies, the average sample size of 22 animals for the water maze experiments was only sufficient to detect an effect size of d = 1.26 with 80% power, and the average sample size of 24 animals for the radial maze experiments was only sufficient to detect an effect size of d = 1.20. In order to achieve 80% power to detect, in a single study, the most probable true effects as indicated by the metaanalysis, a sample size of 134 animals would be required for the water maze experiment (assuming an effect size of d = 0.49) and 68 animals for the radial maze experiment (assuming an effect size of d = 0.69); to achieve 95% power, these sample sizes would need to increase to 220 and 112, respectively. What is particularly striking, however, is the inefficiency of a continued reliance on small sample sizes. Despite the apparently large numbers of animals required to achieve acceptable statistical power in these experiments, the total numbers of animals actually used in the studies contributing to the metaanalyses were even larger: 420 for the water maze experiments and 514 for the radial maze experiments.
There is ongoing debate regarding the appropriate balance to strike between using as few animals as possible in experiments and the need to obtain robust, reliable findings. We argue that it is important to appreciate the waste associated with an underpowered study — even a study that achieves only 80% power still presents a 20% possibility that the animals have been sacrificed without the study detecting the underlying true effect. If the average power in neuroscience animal model studies is between 20–30%, as we observed in our analysis above, the ethical implications are clear.
Low power therefore has an ethical dimension — unreliable research is inefficient and wasteful. This applies to both human and animal research. The principles of the 'three Rs' in animal research (reduce, refine and replace)^{83} require appropriate experimental design and statistics — both too many and too few animals present an issue as they reduce the value of research outputs. A requirement for sample size and power calculation is included in the Animal Research: Reporting In Vivo Experiments (ARRIVE) guidelines^{84}, but such calculations require a clear appreciation of the expected magnitude of effects being sought.
Of course, it is also wasteful to continue data collection once it is clear that the effect being sought does not exist or is too small to be of interest. That is, studies are not just wasteful when they stop too early, they are also wasteful when they stop too late. Planned, sequential analyses are sometimes used in large clinical trials when there is considerable expense or potential harm associated with testing participants. Clinical trials may be stopped prematurely in the case of serious adverse effects, clear beneficial effects (in which case it would be unethical to continue to allocate participants to a placebo condition) or if the interim effects are so unimpressive that any prospect of a positive result with the planned sample size is extremely unlikely^{85}. Within a significance testing framework, such interim analyses — and the protocol for stopping — must be planned for the assumptions of significance testing to hold. Concerns have been raised as to whether stopping trials early is ever justified given the tendency for such a practice to produce inflated effect size estimates^{86}. Furthermore, the decision process around stopping is not often fully disclosed, increasing the scope for researcher degrees of freedom^{86}. Alternative approaches exist. For example, within a Bayesian framework, one can monitor the Bayes factor and simply stop testing when the evidence is conclusive or when resources are expended^{87}. Similarly, adopting conservative priors can substantially reduce the likelihood of claiming that an effect exists when in fact it does not^{85}. At present, significance testing remains the dominant framework within neuroscience, but the flexibility of alternative (for example, Bayesian) approaches means that they should be taken seriously by the field.
Conclusions and future directions
A consequence of the remarkable growth in neuroscience over the past 50 years has been that the effects we now seek in our experiments are often smaller and more subtle than before as opposed to when mostly easily discernible 'lowhanging fruit' were targeted. At the same time, computational analysis of very large datasets is now relatively straightforward, so that an enormous number of tests can be run in a short time on the same dataset. These dramatic advances in the flexibility of research design and analysis have occurred without accompanying changes to other aspects of research design, particularly power. For example, the average sample size has not changed substantially over time^{88} despite the fact that neuroscientists are likely to be pursuing smaller effects. The increase in research flexibility and the complexity of study designs^{89} combined with the stability of sample size and search for increasingly subtle effects has a disquieting consequence: a dramatic increase in the likelihood that statistically significant findings are spurious. This may be at the root of the recent replication failures in the preclinical literature^{8} and the correspondingly poor translation of these findings into humans^{90}.
Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings. As we have shown, these factors result in biases that are exacerbated by low power. Ultimately, these biases reduce the reproducibility of neuroscience findings and negatively affect the validity of the accumulated findings. Unfortunately, publishing and reporting practices are unlikely to change rapidly. Nonetheless, existing scientific practices can be improved with small changes or additions that approximate key features of the idealized model^{4,91,92}. We provide a summary of recommendations for future research practice in Box 2.
Increasing disclosure. False positives occur more frequently and go unnoticed when degrees of freedom in data analysis and reporting are undisclosed^{5}. Researchers can improve confidence in published reports by noting in the text: “We report how we determined our sample size, all data exclusions, all data manipulations, and all measures in the study.”^{7} When such a statement is not possible, disclosure of the rationale and justification of deviations from what should be common practice (that is, reporting sample size, data exclusions, manipulations and measures) will improve readers' understanding and interpretation of the reported effects and, therefore, of what level of confidence in the reported effects is appropriate. In clinical trials, there is an increasing requirement to adhere to the Consolidated Standards of Reporting Trials (CONSORT), and the same is true for systematic reviews and metaanalyses, for which the Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) guidelines are now being adopted. A number of reporting guidelines have been produced for application to diverse study designs and tools, and an updated list is maintained by the EQUATOR Network^{93}. A tenitem checklist of study quality has been developed by the Collaborative Approach to MetaAnalysis and Review of Animal Data in Experimental Stroke (CAMARADES), but to the best of our knowledge, this checklist is not yet widely used in primary studies.
Registration of confirmatory analysis plan. Both exploratory and confirmatory research strategies are legitimate and useful. However, presenting the result of an exploratory analysis as if it arose from a confirmatory test inflates the chance that the result is a false positive. In particular, pvalues lose their diagnostic value if they are not the result of a prespecified analysis plan for which all results are reported. Preregistration — and, ultimately, full reporting of analysis plans — clarifies the distinction between confirmatory and exploratory analysis, encourages wellpowered studies (at least in the case of confirmatory analyses) and reduces the filedrawer effect. These subsequently reduce the likelihood of false positive accumulation. The Open Science Framework (OSF) offers a registration mechanism for scientific research. For observational studies, it would be useful to register datasets in detail, so that one can be aware of how extensive the multiplicity and complexity of analyses can be^{94}.
Improving availability of materials and data. Making research materials available will improve the quality of studies aimed at replicating and extending research findings. Making raw data available will improve data aggregation methods and confidence in reported results. There are multiple repositories for making data more widely available, such as The Dataverse Network Project and Dryad) for data in general and others such as OpenfMRI, INDI and OASIS for neuroimaging data in particular. Also, commercial repositories (for example, figshare) offer means for sharing data and other research materials. Finally, the OSF offers infrastructure for documenting, archiving and sharing data within collaborative teams and also making some or all of those research materials publicly available. Leading journals are increasingly adopting policies for making data, protocols and analytical codes available, at least for some types of studies. However, these policies are uncommonly adhered to^{95}, and thus the ability for independent experts to repeat published analysis remains low^{96}.
Incentivizing replication. Weak incentives for conducting and publishing replications are a threat to identifying false positives and accumulating precise estimates of research findings. There are many ways to alter replication incentives^{97}. For example, journals could offer a submission option for registered replications of important research results (see, for example, a possible new submission format for Cortex^{98}). Groups of researchers can also collaborate on performing one or many replications to increase the total sample size (and therefore the statistical power) achieved while minimizing the labour and resource impact on any one contributor. Adoption of the gold standard of largescale collaborative consortia and extensive replication in fields such as human genome epidemiology has transformed the reliability of the produced findings. Although previously almost all of the proposed candidate gene associations from small studies were false^{99} (with some exceptions^{100}), collaborative consortia have substantially improved power, and the replicated results can be considered highly reliable. In another example, in the field of psychology, the Reproducibility Project is a collaboration of more than 100 researchers aiming to estimate the reproducibility of psychological science by replicating a large sample of studies published in 2008 in three psychology journals^{92}. Each individual research study contributes just a small portion of time and effort, but the combined effect is substantial both for accumulating replications and for generating an empirical estimate of reproducibility.
Concluding remarks. Small, lowpowered studies are endemic in neuroscience. Nevertheless, there are reasons to be optimistic. Some fields are confronting the problem of the poor reliability of research findings that arises from lowpowered studies. For example, in genetic epidemiology sample sizes increased dramatically with the widespread understanding that the effects being sought are likely to be extremely small. This, together with an increasing requirement for strong statistical evidence and independent replication, has resulted in far more reliable results. Moreover, the pressure for emphasizing significant results is not absolute. For example, the Proteus phenomenon^{101} suggests that refuting early results can be attractive in fields in which data can be produced rapidly. Nevertheless, we should not assume that science is effectively or efficiently selfcorrecting^{102}. There is now substantial evidence that a large proportion of the evidence reported in the scientific literature may be unreliable. Acknowledging this challenge is the first step towards addressing the problematic aspects of current scientific practices and identifying effective solutions.
Change history
15 April 2013
On page 2 of this article, the definition of R should have read: "R is the prestudy odds (that is, the odds that a probed effect is indeed nonnull among the effects being probed)". This has been corrected in the online version.
References
Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power.
Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012).
Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20 (1975).
Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012).
Simmons, J. P., Nelson, L. D. & Simonsohn, U. Falsepositive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false.
Sullivan, P. F. Spurious genetic associations. Biol. Psychiatry 61, 1121–1126 (2007).
Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).
Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discov. 10, 712 (2011).
Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855–3859 (2011).
Munafo, M. R., Stothart, G. & Flint, J. Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119–120 (2009).
Sterne, J. A. & Davey Smith, G. Sifting the evidence — what's wrong with significance tests? BMJ 322, 226–231 (2001).
Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The falsepositive to falsenegative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011).
Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008).
Tversky, A. & Kahneman, D. Belief in the law of small numbers. Psychol. Bull. 75, 105–110 (1971).
Masicampo, E. J. & Lalande, D. R. A peculiar prevalence of p values just below .05. Q. J. Exp. Psychol. 65, 2271–2279 (2012).
Carp, J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage 63, 289–300 (2012). This article reviews methods reporting and methodological choices across 241 recent fMRI studies and shows that there were nearly as many unique analytical pipelines as there were studies. In addition, many studies were underpowered to detect plausible effects.
Dwan, K. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081 (2008).
Sterne, J. A. et al. Recommendations for examining and interpreting funnel plot asymmetry in metaanalyses of randomised controlled trials. BMJ 343, d4002 (2011).
JoyGaba, J. A. & Nosek, B. A. The surprisingly limited malleability of implicit racial evaluations. Soc. Psychol. 41, 137–146 (2010).
Schmidt, K. & Nosek, B. A. Implicit (and explicit) racial attitudes barely changed during Barack Obama's presidential campaign and early presidency. J. Exp. Soc. Psychol. 46, 308–314 (2010).
Evangelou, E., Siontis, K. C., Pfeiffer, T. & Ioannidis, J. P. Perceived information gain from randomized trials correlates with publication in highimpact factor journals. J. Clin. Epidemiol. 65, 1274–1281 (2012).
Pereira, T. V. & Ioannidis, J. P. Statistically significant metaanalyses of clinical trials have modest credibility and inflated effects. J. Clin. Epidemiol. 64, 1060–1069 (2011).
Faul, F., Erdfelder, E., Lang, A. G. & Buchner, A. G^{*}Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175–191 (2007).
Babbage, D. R. et al. Metaanalysis of facial affect recognition difficulties after traumatic brain injury. Neuropsychology 25, 277–285 (2011).
Bai, H. Metaanalysis of 5, 10methylenetetrahydrofolate reductase gene poymorphism as a risk factor for ischemic cerebrovascular disease in a Chinese Han population. Neural Regen. Res. 6, 277–285 (2011).
BjorkhemBergman, L., Asplund, A. B. & Lindh, J. D. Metformin for weight reduction in nondiabetic patients on antipsychotic drugs: a systematic review and metaanalysis. J. Psychopharmacol. 25, 299–305 (2011).
Bucossi, S. et al. Copper in Alzheimer's disease: a metaanalysis of serum, plasma, and cerebrospinal fluid studies. J. Alzheimers Dis. 24, 175–185 (2011).
Chamberlain, S. R. et al. Translational approaches to frontostriatal dysfunction in attentiondeficit/hyperactivity disorder using a computerized neuropsychological battery. Biol. Psychiatry 69, 1192–1203 (2011).
Chang, W. P., Arfken, C. L., Sangal, M. P. & Boutros, N. N. Probing the relative contribution of the first and second responses to sensory gating indices: a metaanalysis. Psychophysiology 48, 980–992 (2011).
Chang, X. L. et al. Functional parkin promoter polymorphism in Parkinson's disease: new data and metaanalysis. J. Neurol. Sci. 302, 68–71 (2011).
Chen, C. et al. Allergy and risk of glioma: a metaanalysis. Eur. J. Neurol. 18, 387–395 (2011).
Chung, A. K. & Chua, S. E. Effects on prolongation of Bazett's corrected QT interval of seven secondgeneration antipsychotics in the treatment of schizophrenia: a metaanalysis. J. Psychopharmacol. 25, 646–666 (2011).
Domellof, E., Johansson, A. M. & Ronnqvist, L. Handedness in preterm born children: a systematic review and a metaanalysis. Neuropsychologia 49, 2299–2310 (2011).
Etminan, N., Vergouwen, M. D., Ilodigwe, D. & Macdonald, R. L. Effect of pharmaceutical treatment on vasospasm, delayed cerebral ischemia, and clinical outcome in patients with aneurysmal subarachnoid hemorrhage: a systematic review and metaanalysis. J. Cereb. Blood Flow Metab. 31, 1443–1451 (2011).
Feng, X. L. et al. Association of FK506 binding protein 5 (FKBP5) gene rs4713916 polymorphism with mood disorders: a metaanalysis. Acta Neuropsychiatr. 23, 12–19 (2011).
Green, M. J., Matheson, S. L., Shepherd, A., Weickert, C. S. & Carr, V. J. Brainderived neurotrophic factor levels in schizophrenia: a systematic review with metaanalysis. Mol. Psychiatry 16, 960–972 (2011).
Han, X. M., Wang, C. H., Sima, X. & Liu, S. Y. Interleukin6–74G/C polymorphism and the risk of Alzheimer's disease in Caucasians: a metaanalysis. Neurosci. Lett. 504, 4–8 (2011).
Hannestad, J., DellaGioia, N. & Bloch, M. The effect of antidepressant medication treatment on serum levels of inflammatory cytokines: a metaanalysis. Neuropsychopharmacology 36, 2452–2459 (2011).
Hua, Y., Zhao, H., Kong, Y. & Ye, M. Association between the MTHFR gene and Alzheimer's disease: a metaanalysis. Int. J. Neurosci. 121, 462–471 (2011).
Lindson, N. & Aveyard, P. An updated metaanalysis of nicotine preloading for smoking cessation: investigating mediators of the effect. Psychopharmacology 214, 579–592 (2011).
Liu, H. et al. Association of 5HTT gene polymorphisms with migraine: a systematic review and metaanalysis. J. Neurol. Sci. 305, 57–66 (2011).
Liu, J. et al. PITX3 gene polymorphism is associated with Parkinson's disease in Chinese population. Brain Res. 1392, 116–120 (2011).
MacKillop, J. et al. Delayed reward discounting and addictive behavior: a metaanalysis. Psychopharmacology 216, 305–321 (2011).
Maneeton, N., Maneeton, B., Srisurapanont, M. & Martin, S. D. Bupropion for adults with attentiondeficit hyperactivity disorder: metaanalysis of randomized, placebocontrolled trials. Psychiatry Clin. Neurosci. 65, 611–617 (2011).
Ohi, K. et al. The SIGMAR1 gene is associated with a risk of schizophrenia and activation of the prefrontal cortex. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 1309–1315 (2011).
Olabi, B. et al. Are there progressive brain changes in schizophrenia? A metaanalysis of structural magnetic resonance imaging studies. Biol. Psychiatry 70, 88–96 (2011).
Oldershaw, A. et al. The socioemotional processing stream in Anorexia Nervosa. Neurosci. Biobehav. Rev. 35, 970–988 (2011).
Oliver, B. J., Kohli, E. & Kasper, L. H. Interferon therapy in relapsingremitting multiple sclerosis: a systematic review and metaanalysis of the comparative trials. J. Neurol. Sci. 302, 96–105 (2011).
Peerbooms, O. L. et al. Metaanalysis of MTHFR gene variants in schizophrenia, bipolar disorder and unipolar depressive disorder: evidence for a common genetic vulnerability? Brain Behav. Immun. 25, 1530–1543 (2011).
Pizzagalli, D. A. Frontocingulate dysfunction in depression: toward biomarkers of treatment response. Neuropsychopharmacology 36, 183–206 (2011).
Rist, P. M., Diener, H. C., Kurth, T. & Schurks, M. Migraine, migraine aura, and cervical artery dissection: a systematic review and metaanalysis. Cephalalgia 31, 886–896 (2011).
Sexton, C. E., Kalu, U. G., Filippini, N., Mackay, C. E. & Ebmeier, K. P. A metaanalysis of diffusion tensor imaging in mild cognitive impairment and Alzheimer's disease. Neurobiol. Aging 32, 2322.e5–2322.e18 (2011).
Shum, D., Levin, H. & Chan, R. C. Prospective memory in patients with closed head injury: a review. Neuropsychologia 49, 2156–2165 (2011).
Sim, H. et al. Acupuncture for carpal tunnel syndrome: a systematic review of randomized controlled trials. J. Pain 12, 307–314 (2011).
Song, F. et al. Metaanalysis of plasma amyloidβ levels in Alzheimer's disease. J. Alzheimers Dis. 26, 365–375 (2011).
Sun, Q. L. et al. Correlation of Eselectin gene polymorphisms with risk of ischemic stroke A metaanalysis. Neural Regen. Res. 6, 1731–1735 (2011).
Tian, Y., Kang, L. G., Wang, H. Y. & Liu, Z. Y. Metaanalysis of transcranial magnetic stimulation to treat poststroke dysfunction. Neural Regen. Res. 6, 1736–1741 (2011).
Trzesniak, C. et al. Adhesio interthalamica alterations in schizophrenia spectrum disorders: a systematic review and metaanalysis. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 877–886 (2011).
Veehof, M. M., Oskam, M. J., Schreurs, K. M. & Bohlmeijer, E. T. Acceptancebased interventions for the treatment of chronic pain: a systematic review and metaanalysis. Pain 152, 533–542 (2011).
Vergouwen, M. D., Etminan, N., Ilodigwe, D. & Macdonald, R. L. Lower incidence of cerebral infarction correlates with improved functional outcome after aneurysmal subarachnoid hemorrhage. J. Cereb. Blood Flow Metab. 31, 1545–1553 (2011).
Vieta, E. et al. Effectiveness of psychotropic medications in the maintenance phase of bipolar disorder: a metaanalysis of randomized controlled trials. Int. J. Neuropsychopharmacol. 14, 1029–1049 (2011).
Wisdom, N. M., Callahan, J. L. & Hawkins, K. A. The effects of apolipoprotein E on nonimpaired cognitive functioning: a metaanalysis. Neurobiol. Aging 32, 63–74 (2011).
Witteman, J., van Ijzendoorn, M. H., van de Velde, D., van Heuven, V. J. & Schiller, N. O. The nature of hemispheric specialization for linguistic and emotional prosodic perception: a metaanalysis of the lesion literature. Neuropsychologia 49, 3722–3738 (2011).
Woon, F. & Hedges, D. W. Gender does not moderate hippocampal volume deficits in adults with posttraumatic stress disorder: a metaanalysis. Hippocampus 21, 243–252 (2011).
Xuan, C. et al. No association between APOE ε 4 allele and multiple sclerosis susceptibility: a metaanalysis from 5472 cases and 4727 controls. J. Neurol. Sci. 308, 110–116 (2011).
Yang, W. M., Kong, F. Y., Liu, M. & Hao, Z. L. Systematic review of risk factors for progressive ischemic stroke. Neural Regen. Res. 6, 346–352 (2011).
Yang, Z., Li, W. J., Huang, T., Chen, J. M. & Zhang, X. Metaanalysis of Ginkgo biloba extract for the treatment of Alzheimer's disease. Neural Regen. Res. 6, 1125–1129 (2011).
Yuan, H. et al. Metaanalysis of tau genetic polymorphism and sporadic progressive supranuclear palsy susceptibility. Neural Regen. Res. 6, 353–359 (2011).
Zafar, S. N., Iqbal, A., Farez, M. F., Kamatkar, S. & de Moya, M. A. Intensive insulin therapy in brain injury: a metaanalysis. J. Neurotrauma 28, 1307–1317 (2011).
Zhang, Y. G. et al. The −1082G/A polymorphism in IL10 gene is associated with risk of Alzheimer's disease: a metaanalysis. J. Neurol. Sci. 303, 133–138 (2011).
Zhu, Y., He, Z. Y. & Liu, H. N. Metaanalysis of the relationship between homocysteine, vitamin B(12), folate, and multiple sclerosis. J. Clin. Neurosci. 18, 933–938 (2011).
Ioannidis, J. P. & Trikalinos, T. A. An exploratory test for an excess of significant findings. Clin. Trials 4, 245–253 (2007). This study describes a test that evaluates whether there is an excess of significant findings in the published literature. The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies.
Ioannidis, J. P. Excess significance bias in the literature on brain volume abnormalities. Arch. Gen. Psychiatry 68, 773–780 (2011).
Pfeiffer, T., Bertram, L. & Ioannidis, J. P. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 6, e18362 (2011).
Tsilidis, K. K., Papatheodorou, S. I., Evangelou, E. & Ioannidis, J. P. Evaluation of excess statistical significance in metaanalyses of 98 biomarker associations with cancer risk. J. Natl Cancer Inst. 104, 1867–1878 (2012).
Ioannidis, J. Clarifications on the application and interpretation of the test for excess significance and its extensions. J. Math. Psychol. (in the press).
David, S. P. et al. Potential reporting bias in small fMRI studies of the brain. PLoS Biol. (in the press).
Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010).
Ioannidis, J. P. Extrapolating from animals to humans. Sci. Transl. Med. 4, 151ps15 (2012).
Jonasson, Z. Metaanalysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data. Neurosci. Biobehav. Rev. 28, 811–825 (2005).
Macleod, M. R. et al. Evidence for the efficacy of NXY059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008).
Sena, E., van der Worp, H. B., Howells, D. & Macleod, M. How can we improve the preclinical development of drugs for stroke? Trends Neurosci. 30, 433–439 (2007).
Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Methuen, 1958).
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).
Bassler, D., Montori, V. M., Briel, M., Glasziou, P. & Guyatt, G. Early stopping of randomized clinical trials for overt efficacy is problematic. J. Clin. Epidemiol. 61, 241–246 (2008).
Montori, V. M. et al. Randomized trials stopped early for benefit: a systematic review. JAMA 294, 2203–2209 (2005).
Berger, J. O. & Wolpert, R. L. The Likelihood Principle: A Review, Generalizations, and Statistical Implications (ed. Gupta, S. S.) (Institute of Mathematical Sciences, 1998).
Vesterinen, H. M. et al. Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the Journal of Cerebral Blood Flow and Metabolism. J. Cereb. Blood Flow Metab. 31, 1064–1072 (2011).
Smith, R. A., Levine, T. R., Lachlan, K. A. & Fediuk, T. A. The high cost of complexity in experimental design and data analysis: type I and type II error rates in multiway ANOVA. Hum. Comm. Res. 28, 515–530 (2002).
Perel, P. et al. Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 334, 197 (2007).
Nosek, B. A. & BarAnan, Y. Scientific utopia: I. Opening scientific communication. Psychol. Inquiry 23, 217–243 (2012).
OpenScienceCollaboration. An open, largescale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657–660 (2012). This article describes the Reproducibility Project — an open, largescale, collaborative effort to systematically examine the rate and predictors of reproducibility in psychological science. This will allow the empirical rate of replication to be estimated.
Simera, I. et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 8, 24 (2010).
Ioannidis, J. P. The importance of potential studies that have not existed and registration of observational data sets. JAMA 308, 575–576 (2012).
AlsheikhAli, A. A., Qureshi, W., AlMallah, M. H. & Ioannidis, J. P. Public availability of published research data in highimpact journals. PLoS ONE 6, e24357 (2011).
Ioannidis, J. P. et al. Repeatability of published microarray gene expression analyses. Nature Genet. 41, 149–155 (2009).
Ioannidis, J. P. & Khoury, M. J. Improving validation practices in “omics” research. Science 334, 1230–1232 (2011).
Chambers, C. D. Registered Reports: A new publishing initiative at Cortex. Cortex 49, 609–610 (2013).
Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The falsepositive to falsenegative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011).
Siontis, K. C., Patsopoulos, N. A. & Ioannidis, J. P. Replication of past candidate loci for common diseases and phenotypes in 100 genomewide association studies. Eur. J. Hum. Genet. 18, 832–837 (2010).
Ioannidis, J. P. & Trikalinos, T. A. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J. Clin. Epidemiol. 58, 543–549 (2005).
Ioannidis, J. Why science is not necessarily selfcorrecting. Perspect. Psychol. Sci. 7, 645–654 (2012).
Zollner, S. & Pritchard, J. K. Overcoming the winner's curse: estimating penetrance parameters from casecontrol data. Am. J. Hum. Genet. 80, 605–615 (2007).
Acknowledgements
M.R.M. and K.S.B. are members of the UK Centre for Tobacco Control Studies, a UK Public Health Research Centre of Excellence. Funding from British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council and the UK National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged. The authors are grateful to G. Lewis for his helpful comments.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Related links
Related links
FURTHER INFORMATION
Rights and permissions
About this article
Cite this article
Button, K., Ioannidis, J., Mokrysz, C. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14, 365–376 (2013). https://doi.org/10.1038/nrn3475
Published:
Issue Date:
DOI: https://doi.org/10.1038/nrn3475
This article is cited by

Association between HbA1c and deep sternal wound infection after coronary artery bypass: a systematic review and metaanalysis
Journal of Cardiothoracic Surgery (2024)

Benchmark findings from a veteran electronic patientreported outcomes evaluation from a chronic pain management telehealth program
BMC Health Services Research (2024)

Minimal clinically important difference (MCID), substantial clinical benefit (SCB), and patientacceptable symptom state (PASS) in patients who have undergone total knee arthroplasty: a systematic review
Knee Surgery & Related Research (2024)

Exploring the steps of learning: computational modeling of initiatoryactions among individuals with attentiondeficit/hyperactivity disorder
Translational Psychiatry (2024)

ezBIDS: Guided standardization of neuroimaging data interoperable with major data archives and platforms
Scientific Data (2024)