Jessica Butler is a Senior Research Fellow at the University of Aberdeen's Centre for Health Data Science, where she works with patient data from the National Health Service looking for ways to improve public health. She has a keen interest in tackling rigour and reproducibility issues in research. Jessica joined the Scientific Reports Editorial Board in 2021 as one of the inaugural Board Members on the Registered Reports section.
On our 10th anniversary, Rafal Marszalek (Deputy Editor at Scientific Reports) talked to Jessica about the challenges of public health research during the pandemic, issues with researcher evaluation and reproducibility problems, and why scientists should turn to Registered Reports for help with these.
Could you tell me a bit about your current research?
Since COVID, a number of people on our team at the university have been seconded to the NHS. We're just finishing an analysis of people with chronic diseases who were asked to isolate during the first lockdown because they were vulnerable to a poor outcome from COVID. The NHS wants to know how these people's healthcare changed while isolating to understand if there will be 'care debt' to prioritise for this important population. Our centre runs a high-security setting for safely analysing anonymised patient data from a wide variety of sources.
Has the pandemic changed the way you do research and publish results?
Rather than the traditional two-year study, followed by a year to get a paper written up and published, we've moved into rapid-response mode. For example, I recently wrote a bit of software for the NHS that calculates contact networks for the Coronavirus Track and Trace team. They needed it immediately, and it was a joy to be able to produce something quickly and have it be used right away.
It's also been the first time in my life that I've been able to work and produce things that aren't papers. I was under no pressure to have my results fit into the format of an academic journal article, and that's been an exciting and enjoyable challenge. But this rapid response doesn't mesh with the typical academic reward system and publication timeline.
Also during COVID, I've really enjoyed Twitter as an excellent source for discovery and peer review before traditional publication. There's a bunch of lovely nerds out there picking apart all the latest studies, figuring out which ones are reliable, which ones are worth looking harder at.
Do you think there will be more emphasis on dissemination of other research outputs in the future?
I think there will be a move away from only evaluating traditional publications when evaluating how productive a scientist is, but it will be a slow and difficult process.
Let's say you love Registered Reports, I love Registered Reports, our team loves Registered Reports, but this doesn't matter until our institution loves Registered Reports. With all the goodwill in the world, we still have to do the consciousness raising and make the infrastructure to fit research that is different, and that takes work.
Since the emphasis on journal articles is still so strong, how do you think we fit into that?
Journals can do so much. I think Registered Reports are the way forward: saying that the journal will evaluate research blind to the outcome, to the significance of the p-value, helps. And the more journals say they'll evaluate research results-blind, the more we can judge the research on its methods rather than some sort of novelty aspect.
Another thing, and it seems so small but is so important, is journals not just asking for but forcing open data with articles as much as possible, and open code as well. Today there are all sorts of ways we can make it easier to evaluate the quality of our research and journals play a big role in encouraging this.
The Registered Reports format originated in cognitive sciences, but how do you think it can be employed in other disciplines? We hope to be able to advocate effectively to researchers in other disciplines for this format and would hope to see it grow in medicine and public health in particular.
Every randomized controlled trial should be a Registered Report. RCTs already do the upfront analytical planning that epidemiologists don't always do, so requiring the submission of this work as a Registered Report is an obvious thing to do.
There is a moral obligation for any medical research (including research using medical records) to be the most rigorous research possible. Any studies that use patient data or other routinely-collected data ought to use a Registered Report format to increase rigour and the reliability of results. Otherwise it's a waste of resource and a waste of privacy risk.
Luckily, we have the safe-setting infrastructure in place for handling patient data anonymously that we could use to do what Registered Reports lend themselves to: keeping researchers blind to the data until they have a very specific research plan that can then be peer reviewed and accepted at a journal.
Do you think publishers could collaborate with institutions, institutional review boards, or funders on this format?
Yes. An IRB is generally considered a stick or a hurdle to get over, where a Registered Report is a carrot: submit your study design, get it peer reviewed, and you're more or less guaranteed publication. That is a massive incentive for working in this more rigorous way. And making a commitment to publish as a Registered Report a condition of funding, that's multiple carrots. Submit a study, define it to the Registered Reports standard, then get the money to do it and a guarantee of publication. To me that's a scientist's dream, and if that's the way science worked, I would have a different life and it would be marvellous.
What are the biggest challenges for future research and how it is disseminated? What are the things that the scientific community will just have to face?
The way we're evaluated and how journals pick articles encourage research practices that mean a good deal of the current scientific record isn't reliable. We don't have a standard for evaluating rigour. We don't put a lot of stress on making sure our research is true. And that is astonishing!
We have to admit this without becoming completely paralysed. Instead say: what can we do? Things like Registered Reports, things like automated checking of all analysis code, like making machine-readable hypotheses so that everything in the research process is easily evaluable. The system isn't set up to reward working with rigour, nor evaluating rigour. We need to acknowledge that and then advocate for things like Registered Reports, which are radically different and better.
Happily, I know there are a lot of people working hard at improving our science now – a lot of journals are interested, a lot of funders are interested, and a lot of scientists are interested, especially young researchers. So I am fundamentally optimistic that things will improve.