Credit: Image Source/Getty Images
5 tips for dealing with non-significant results
It might look like failure, but don’t let go just yet.
16 September 2019
Image Source/Getty Images
When researchers fail to find a statistically significant result, it’s often treated as exactly that – a failure. Non-significant results are difficult to publish in scientific journals and, as a result, researchers often choose not to submit them for publication.
This means that the evidence published in scientific journals is biased towards studies that find effects.
A study published in Science by a team from Stanford University who investigated 221 survey-based experiments funded by the National Science Foundation found that nearly two-thirds of the social science experiments that produced null results were filed away, never to be published.
By comparison, 96% of the studies with statistically strong results were written up.
“These biases imperil the robustness of scientific evidence,” says David Mehler, a psychologist at the University of Münster in Germany. “But they also harm early career researchers in particular who depend on building up a track record.”
Mehler is the co-author of a recent article published in the Journal of European Psychology Students about appreciating the significance of non-significant findings.
So, what can researchers do to avoid unpublishable results?
#1: Perform an equivalence test
The problem with a non-significant result is that it’s ambiguous, explains Daniël Lakens, a psychologist at Eindhoven University of Technology in the Netherlands.
It could mean that the null hypothesis is true – there really is no effect. But it could also indicate that the data are inconclusive either way.
Lakens says performing an ‘equivalence test’ can help you distinguish between these two possibilities. It can’t tell you that there is no effect, but it can tell you that an effect – if it exists – is likely to be of negligible practical or theoretical significance.
Bayesian statistics offer an alternative way of performing this test, and in Lakens’ experience, “either is better than current practice”.
#2 Collaborate to collect more data
Equivalence tests and Bayesian analyses can be helpful, but if you don’t have enough data, their results are likely to be inconclusive.
“The root problem remains that researchers want to conduct confirmatory hypothesis tests for effects that their studies are mostly underpowered to detect,” says Mehler.
This, he adds, is a particular problem for students and early career researchers, whose limited resources often constrain them to small sample sizes.
One solution is to collaborate with other researchers to collect more data. In psychology, the StudySwap website is one way for researchers to team up and combine resources.
#3 Use directional tests to increase statistical power
If resources are scarce, it’s important to use them as efficiently as possible. Lakens suggests a number of ways in which researchers can tweak their research design to increase statistical power – the likelihood of finding an effect if it really does exist.
In some circumstances, he says, researchers should consider ‘directional’ or ‘one-sided’ tests.
For example, if your hypothesis clearly states that patients receiving a new drug should have better outcomes than those receiving a placebo, it makes sense to test that prediction rather than looking for a difference between the groups in either direction.
“It’s basically free statistical power just for making a prediction,” says Lakens.
#4 Perform sequential analyses to improve data collection efficiency
Efficiency can also be increased by conducting sequential analyses, whereby data collection is terminated if there is already enough evidence to support the hypothesis, or it’s clear that further data will not lead to it being supported.
This approach is often taken in clinical trials where it might be unethical to test patients beyond the point that the efficacy of the treatment can already be determined.
A common concern is that performing multiple analyses increases the probability of finding an effect that doesn’t exist. However, this can be addressed by adjusting the threshold for statistical significance, Lakens explains.
#5 Submit a Registered Report
Whichever approach is taken, it’s important to describe the sampling and analyses clearly to permit a fair evaluation by peer reviewers and readers, says Mehler.
Ideally, studies should be preregistered. This allows authors to demonstrate that the tests were determined before rather than after the results were known. In fact, Mehler argues, the best way to ensure that results are published is to submit a Registered Report.
In this format, studies are evaluated and provisionally accepted based on the methods and analysis plan. The paper is then guaranteed to be published if the researchers follow this preregistered plan – whatever the results.
In a recent investigation, Mehler and his colleague, Chris Allen from Cardiff University in the UK, found that Registered Reports led to a much increased rate of null results: 61% compared with 5 to 20% for traditional papers.
Read next:
First analysis of ‘pre-registered’ studies shows sharp rise in null findings