Clinical trial of the Chinese vaccine CORONAVAC

Evidence from randomized controlled trials has been essential to showing that COVID-19 vaccines work.Credit: Aloisio Mauricio/Fotoarena/ZUMA Press

The past year was a turbulent one for Archie Cochrane, despite his having been dead for more than 30 years. A Scottish doctor and seminal figure in the history of medicine, Cochrane questioned the standard way doctors decide how to treat disease — which was based largely on their opinion. Fifty years ago, Cochrane proposed that decisions should instead be based on rigorous evidence1 — particularly randomized controlled trials — and he later challenged medics to build useful summaries of such studies.

Billions of people alive today have probably benefited from these ideas. Cochrane and other pioneers inspired a movement called evidence-based medicine, in which research-based evidence informs doctors and patients in decisions about care. In 1993, an international network known as the Cochrane Collaboration (now called just Cochrane) was founded; this group and others have assembled a vast library of systematic reviews in medicine and other disciplines, providing a foundation of evidence that has helped to save many lives.

But the COVID-19 pandemic has been one of the biggest tests yet of evidence-based medicine — and has shown that the current system falls short in a fast-moving global emergency. There have, of course, been huge wins. Randomized controlled trials have been crucial to testing the safety and efficacy of drugs, as well as the vaccines that will bring the pandemic to an end. But as a Feature in Nature reports, the pandemic also resulted in many wasteful clinical trials that were too small to produce useful results — and an accompanying wave of systematic reviews, many of them of low quality, repetitive and quickly out of date.

Time for change

It’s important that researchers, doctors and global leaders assess what worked, what didn’t and why, and make recommendations for change. They must fix the evidence pipeline so that it is stronger and better able to supply timely, high-quality evidence — not just for the next pandemic, but for everyday health emergencies, from malaria to heart disease.

A prime opportunity will arrive in October, at a meeting of global-health leaders organized by Cochrane and the World Health Organization (WHO), as well as other members of a group called COVID-END (COVID-19 Evidence Network to support Decision-making). The heads of COVID-END are also planning to convene a global commission of thought leaders to assess how best to supply evidence to address social challenges, such as climate change and inequality, that go well beyond health. This group has a unique opportunity to refine, re-imagine and re-engineer processes for the generation, supply and use of research-based evidence and so ensure that our future world is one better informed by rationality and fact.

In a systematic review, researchers typically define a question. Does keyhole surgery help knee pain in middle-aged people? Does hydroxychloroquine prevent COVID-19 deaths? They then search for and collate all the relevant, high-quality studies that can help to answer the question, and analyse the pooled results. A systematic review should make sense of the balance of evidence when individual studies conflict.

But during the pandemic, the challenges of producing such reviews were exposed. A rigorous systematic review often takes a year or two to complete, which is too long to wait when urgent answers are required. The pace of research and volume of results produced during the pandemic made it impossible to keep some reviews up to date; yet systematic reviews and other evidence syntheses have poured out. In 2010, researchers estimated that about 11 systematic reviews were published per day, in a paper2 that asked despairingly, ‘how will we ever keep up?’. By 2019, that figure had exceeded 65; today, one database contains some 9,000 evidence syntheses related to COVID-19 alone, about 21 for every day since the WHO characterized the outbreak as a pandemic. Taken together, this means that doctors, policymakers and others who are desperate for authoritative reviews of evidence can struggle to find what they need.

Keeping up

The fundamental principles of evidence-based medicine stand firm; it’s the processes that need to evolve. When the Cochrane Collaboration was formed, its founders knew that reviews must be regularly updated with the latest research. But, in practice, this is often difficult because of the laborious nature of the literature searches and data synthesis required.

New methods can help. Last year, a group at the Institute for Evidence-Based Healthcare at Bond University in Gold Coast, Australia, published a full systematic review completed in two weeks, using a skilled team and automated tools to search for and extract data3. And during the pandemic, scientists collaborated to quickly produce ‘living’ systematic reviews on potential COVID-19 therapies, which are updated as new studies come out. Researchers must now evaluate the best methods for generating fast reviews and living ones, as well as deciding on which topics it’s worth investing in them.

The pandemic has shown that large, coordinated clinical trials that span hospitals and test multiple treatments against one condition offer an excellent way to include sufficient numbers of patients to provide firm conclusions about what works. The United Kingdom’s RECOVERY trial and the WHO’s SOLIDARITY trial are exemplars of this approach. It would be a powerful legacy of the pandemic if this model were widely adopted on an ongoing basis to provide the numbers necessary for trials in many health conditions. This would have the added bonus of involving many doctors and researchers, helping to educate them in what a well-designed trial looks like — and so ensuring that fewer poorly designed ones are done.

Cochrane and the other organizers of the October meeting say they hope to take any recommendations that emerge to the World Health Assembly in May 2022, to discuss with member states. It’s important for countries to demand — and fund — changes. All these efforts must include diverse perspectives from patients, citizens and policymakers. This will help to ensure that evidence is equitably available, and that research and reviews address the needs of communities worldwide, rather than just earning scientists more career-boosting papers. Getting there will require organizations such as Cochrane to take a hard look at their processes and be willing to change what they do.

As Hilda Bastian, an independent scientist who studies evidence-based medicine in Victoria, Australia, rightly argues, we need to ensure that when the next pandemic strikes and everyone is googling for evidence, that the high-quality reviews rise to the top, leaving the tide of questionable ones behind.

There is a danger that everyone will be so keen to move on and forget the trauma of the pandemic that we won’t take time to reflect and improve. But Archie Cochrane’s 50-year-old plea that decisions be based on rigorous evidence is more important than ever, and, tired though everyone is, we need to put in the work now so that we can deliver better, quicker evidence next time around.