Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • COMMENT

Decision makers need constantly updated evidence synthesis

Cartoon showing a pile of science papers in human form walking from the past to the future.

Illustration by David Parkins

As the world began to respond to the COVID-19 pandemic last year, there was an explosion of guidelines, position statements and protocols — many of low quality and contradictory. In March 2020, several of us approached the Australian National Health and Medical Research Council, worried that the cacophony would create confusion and anxiety among already-stressed clinicians. We argued for key bodies to come together quickly and use robust, evidence-based processes to find signals in the noisy flow of COVID-19 research. Two weeks later, we had formed a task force and produced the first version of national, evidence-based COVID-19 guidelines for Australia. We made a commitment to update the guidelines every week, but this had never been done before. Our challenge was to work out how.

Typically, national guidelines draw on formal summaries of research evidence called systematic reviews, but the pandemic ‘broke the evidence pipeline’1. Take the example of remdesivir, an intravenous treatment originally developed for Ebola virus. In May 2020, weak but promising data suggested it could be used to treat COVID-19. Over the next 18 months, 52 papers from 14 randomized trials were published. Clinicians and policymakers had to make decisions on the basis of this shifting, and often contradictory, body of evidence. To help them, scholars produced systematic reviews — 30 in 2020. Many were out of date before publication because they left out recently published primary studies; most of the rest became out of date within weeks.

Out-of-date systematic reviews are common any time there is a flood of new research. In the absence of up-to-date summaries of accumulating knowledge, decision makers’ attention often jumps from study to study. This muddles policymaking, fuels controversy and erodes trust in science.

A better system would keep summaries of research evidence up to date. That’s the system we built for the Australian National COVID‑19 Clinical Evidence Taskforce, for the World Health Organization’s (WHO’s) Therapeutics and COVID-19: living guideline, and for the COVID-NMA repository of COVID-19 research. The system can be applied beyond COVID-19, and indeed beyond health. Here’s what that will take.

Evidence building

The process of identifying and combining data across studies to create a clear understanding of a body of research is known as evidence synthesis. The resulting publications are generally called systematic reviews, and often include a meta-analysis. Since the 1980s, the practice of evidence synthesis has grown to become the foundation for high-impact decision-making in disease prevention, diagnosis and treatment, and in other aspects of health. Evidence synthesis also helps to tackle questions in education, economics, environment, criminal justice, global development and more. The number of published academic systematic reviews has risen from around 6,000 in 2011 to more than 45,000 in 2021.

University departments, international bodies such as Cochrane (a leading producer of systematic reviews on health topics), and numerous conferences and journals have established scientific methods, conventions and production systems for evidence synthesis. The approach is now routine across many national and global decision-making bodies responsible for policy and practice guidance. For example, rather than relying on single studies, drug regulators and insurers worldwide use syntheses of all relevant studies to evaluate safety and effectiveness and decide whether to approve or pay for a drug.

Evidence synthesis can be applied to the most pressing global challenges: climate change, energy transitions, biodiversity loss, antimicrobial resistance, poverty eradication and so on. But in practice, synthesis projects are often under-resourced, of poor quality, uncoordinated, duplicative and out of date.

‘Living evidence’ is an approach to synthesis that we and others have developed to address these problems2. It produces ‘ready-to-go’ evidence summaries that better serve the needs of decision makers because they are both rigorous (all relevant research has been carefully appraised) and up to date. Living evidence is particularly important in topics for which research evidence is emerging rapidly, current evidence is uncertain and new research might change policy or practice. For example, the Australian national guidelines for treating people who have had a stroke are now updated every 3–4 months instead of every 5–7 years (see ‘Evidence accelerated’). Clinicians can consult the most up-to-date evidence, and this has led to greater trust and use of the guidelines.

Evidence accelerated: Charts showing that living-evidence allows more frequent guideline update than conventional methods.

Source: Kelvin Hill, Heidi Li, Simon Turner, Jordi Elliott, Andrew Duan

Researchers practising living evidence establish protocols that commit to regular updates and continuously monitor bibliographic and other databases to identify new research. These projects are often enabled by technologies such as natural-language processing, machine learning and crowdsourcing and, sometimes, by FAIR data sharing (which ensures the findability, accessibility, interoperability and reuse of digital assets). Researchers rapidly incorporate studies in frequent cycles that follow established methods for high-quality synthesis and guidelines (for instance, predefined protocols and careful appraisal of the research included).

Even with a deluge of new research, it is feasible to update syntheses and treatment guidelines frequently. For COVID-19, we pushed that limit lower than ever, updating some living guidelines weekly (see ‘Evidence accelerated’). Frequent updating assured policymakers and the public that all relevant research had been taken into account when recommending against the use of the drug hydroxychloroquine, for example. And the COVID-END collaboration used living evidence to keep Canada’s public-health agency updated on COVID‑19 vaccine effectiveness as variants of concern emerged.

This year, the UK National Institute for Health and Care Excellence (NICE) announced that living guidelines will be a pillar of its improvement strategy over the next five years. After piloting efforts for several years, the WHO this year began promoting living systematic reviews and living guidelines as standard methodology. It has used the approach for COVID-19, maternal and perinatal care and contraception. In the academic literature, about 100 living systematic reviews have been published since the start of 2020, compared with perhaps 15 in the 4 years previously.

Many global challenges can benefit from living evidence. Consider climate policy. The Intergovernmental Panel on Climate Change (IPCC) has been very effective in synthesizing evidence on the anthropogenic causes of climate change, its impacts and the potential effects of long-term emission-reduction strategies. But there is no comparable culture of evidence synthesis on the effectiveness of climate-change mitigation or adaptation3. Living evidence could help to firmly establish this capacity to ensure that the thousands of decisions needed to achieve net-zero emissions are supported by reliable and up-to-date scientific evidence. There is real urgency for funders and policymakers to incentivize this cultural shift.

Stakeholders and end-users should be involved in both the design and production of living evidence. Depending on the topic, this might mean including communities that are most likely to be affected by extreme weather, or vocational training programmes or a national disease- treatment guideline. For example, early in the pandemic, the COVID‑19 Core Outcome Set initiative identified the most important COVID-19 outcomes for patients and clinicians from around the world4. This was used to guide how living-guidelines panels judged the potential benefits of new treatments.

Living evidence is particularly relevant for research funders. To direct resources to important questions, funders must identify current knowledge gaps. Living evidence reveals these gaps by mapping published and ongoing research and by keeping up to date with new research. Investments in living evidence can therefore benefit society twice — directly by funding synthesis efforts, and indirectly by enhancing research investments. Funders should invest in systems, tools and partnerships that will build overall capacities beyond any specific project.

Four steps

How best can living evidence expand? Four considerations are key.

Reduce unit costs. Cutting the time and effort required for systematic review is crucial to the scale-up of living evidence. At present, an initial systematic review typically requires more than 200 person-hours of manual work to sift through titles and abstracts of potentially relevant studies. Many of us are developing technologies to make workflows more efficient and cost-effective. These include natural-language processing and machine learning5, crowdsourcing6, FAIR data and non-profit software tools, including Covidence and EPPI-Reviewer for systematic review, and MAGICapp for guideline development (J.E. and P.O.V are chief executives at Covidence and MAGICapp, respectively). Together, these technologies could slash the time required by two-thirds. The more others get involved, the better and faster these tools will get. Publications should include better metadata to improve discovery, and should follow FAIR practices to provide open and machine-readable research data.

Prevent research waste. As systematic reviews became widely recognized as a form of publication, there was an increase in wasteful, duplicated, low-quality reviews. Some researchers and journals pursue ‘quick’ systematic reviews as a way to boost their publication and citation counts. As with the remdesivir example, the number of systematic reviews summarizing a body of randomized trials can often outnumber the number of relevant trials.

Funders could help by focusing on a set of high-quality living systematic reviews and guidelines. Another approach could be to establish larger collaborative projects, analogous to global multisite clinical trials or ‘big science’ collaborations in physics, as a mechanism to reduce redundancy.

Optimize publishing. Publishers such as F1000, Cochrane, The BMJ and Annals of Internal Medicine have shown that academic publishers can support frequent updating of living systematic reviews and guidelines with multiple updates of essentially the same document. Most are minor and only occasionally major. Workflows were modified so that editorial and peer review, as well as copyediting and production, supported frequent revisions. This required differentiating between small and large alterations and ensuring links between versions. Minor updates can be incorporated in the original publication, or as an addendum. When authors and editors judge that changes to a paper warrant a major update, this can trigger a new article version (linked to previous versions through CrossRef and other links on the journal website) and a new bibliographic database listing and digital object identifier (DOI).

The research community must rethink what the ‘version of record’ means and how to optimize the incentives and rewards for authors and peer reviewers of frequently updated systematic reviews. Peer review of these continuously updated documents should be neither slow and onerous nor a quick rubber stamp. The underlying data from living systematic reviews and guidelines should be FAIR, to support replicability and transparency and maximize reuse.

Facilitate implementation. Living evidence has the potential to speed up the incorporation of science into practice and policy. Decisions do not depend solely on evidence, however, and changes in practice generally lag behind changes in guidance. Research on implementation is therefore essential to understand how best living evidence can serve policy and practice. So far, experience with implementation has been limited to fewer than a dozen case studies such as those mentioned here.

Issues that need to be investigated include: assessing the impact of new models of dissemination (for example, continuous ‘what’s new’ feeds rather than conventional, intermittent announcements); deciding when changes in evidence should trigger implementation; and whether a supply of reliable, current evidence affects how it is incorporated into decisions. In many instances, implementation activities will not need to change, and living evidence will give them a head start.

Living-evidence outputs should also be set up to integrate with data systems. For example, health-care recommendations in living guidelines could flow into clinical tools to support physicians’ decisions. Living systematic reviews of low-carbon technologies could update climate-change-mitigation models.

Keep moving

Decisions relevant to global challenges must be informed by the best available evidence. It should no longer be acceptable for evidence to be out of date, biased or selective. Without trustworthy and up-to-date summaries, the world risks making ill-informed decisions and wasting investment.

Living evidence as a practice has been around for barely five years, and there is much to learn. Our experience so far is counter to common initial concerns. Rather than being confused by changing guidelines, clinicians value a resource they know is up to date. Interest from living-evidence practitioners can be sustained and team membership can gradually shift without interrupting updates. The cost of continuously updating guidelines seems about the same as gearing up for big revisions every few years.

The advances made in evidence systems during the COVID-19 pandemic should extend beyond health. We call on researchers in all scientific fields, and their funders, to test the living-evidence model across diverse domains. Trialling the approach with different types of evidence, for a wide range of decision makers, will contribute to the advancement and scale-up of the model. Science does not stand still; neither should its synthesis and translation into action.

Nature 600, 383-385 (2021)

doi: https://doi.org/10.1038/d41586-021-03690-1

The contents of this article are the responsibility of the authors and do not reflect the views of the World Health Organization, for which O.T.O. works.

References

  1. Nature 593, 182–185 (2021).

    PubMed  Article  Google Scholar 

  2. Elliott, J. H. et al. PLoS Med 11, e1001603 (2014).

    PubMed  Article  Google Scholar 

  3. Minx, J. C., Callaghan, M., Lamb, W. F., Garard, J. & Edenhofer, O. Environ. Sci. Policy 77, 252–259 (2017).

    Article  Google Scholar 

  4. Tong, A. et al. Crit. Care Med. 48, 1622–1635 (2020).

    PubMed  Article  Google Scholar 

  5. Thomas, J. et al. J. Clin. Epidemiol. 133, 140–151 (2021).

    PubMed  Article  Google Scholar 

  6. Noel-Storr, A. et al. J. Clin. Epidemiol. 133, 130–139 (2021).

    PubMed  Article  Google Scholar 

Download references

Competing Interests

J.E. is chief executive of Covidence, a non-profit software platform for systematic review production. R.L. is managing director of F1000. P.R. is co-chair of the Cochrane Scientific Committee and member of the Steering Committee of COVID-NMA, a living mapping and living systematic review of COVID-19 trials. P.O.V. is chief executive of the non-profit MAGIC Evidence Ecosystem Foundation (www.magicevidence.org) that hosts an online platform (www.magicapp.org) to support living guidelines. J.M.G. receives a small honorarium as president of the board of the Campbell Collaboration.

Subjects

Nature Careers

Jobs

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing

Search

Quick links