The evolution of evidence synthesis
Increasing interest in promoting evidence-based clinical practice has led to methodological advancements in evidence syntheses [1, 2]. Narrative reviews have been superseded by systematic reviews, which may include meta-analysis—statistical pooling of treatment effect estimates across similar trials to improve precision [3,4,5]. Systematic reviews minimize the risk of selection bias by considering all evidence relevant to a clinical question; however, an important limitation of conventional meta-analyses is that they only inform treatments that have been directly compared in clinical trials. Moreover, many trials compare active interventions against placebo, usual, or standard care, whereas patients and clinicians are typically concerned with the relative effectiveness of competing interventions. Network meta-analysis (NMA) has emerged to address these limitations by allowing for calculation of the comparative effects of more than two competing interventions, even when they have not been directly compared in clinical trials [6, 7].
What is network meta-analysis?
NMA requires the same steps as a conventional meta-analysis which include a systematic search of the literature, assessment of risk of bias among eligible trials, statistical pooling of reported pairwise comparisons for all outcomes of interest, and assessment of the overall certainty of evidence on an outcome-by-outcome basis. This provides the “direct” evidence for treatments that have been compared against each other, which is graphically represented by a network map. An NMA then identifies all interventions that are connected by virtue of a common comparator. For example, two different active treatments may have been compared against placebo in different trials. An NMA allows for a theoretical trial to be created that compares these active treatments against each other, based on their effect against a common comparator (placebo), which provides “indirect” evidence. Indirect comparisons provide an opportunity to fill knowledge gaps within the available evidence, providing a more comprehensive understanding of treatment options for the clinician. The network estimate is the pooled result of the direct and indirect evidence for a given comparison, or only the indirect evidence if no direct evidence is available [6, 8, 9]. Once all treatments have been compared within a network, there are different methods for ranking treatments to convey their relative net effectiveness. Limitations and advancements in the ranking methodology will be discussed in greater detail within the example provided below.
Network meta-analysis in practice
An example network map on first-line medications effects on intra-ocular pressure (IOP) for primary open angle glaucoma (POAG) is shown in Fig. 1, which represents all pharmacologic treatments that have been directly evaluated in 114 clinical trials for this condition . Traditional meta-analysis would be limited in comparing two of these treatments at a time, and could not inform the effectiveness of treatments that have not been directly compared; however, this NMA provides the relative effectiveness of all 15 treatments in a single investigation, even when no RCT is available to make a direct comparison between two treatments. The network map uses circles, or nodes, for each included treatment, that increase in size relative to the number of patients treated with that medication within included RCTs. The lines connecting different treatments are weighted by the number of RCTs comparing them (i.e., thicker lines convey more direct trials) . In this particular study, the authors color coded their treatment nodes by drug class to improve interpretation. The network is specific to one outcome, in this case IOP, and the network assumes that the baseline characteristics of patients enrolled across trials are similar.
As Fig. 1 demonstrates, there are many RCTs assessing pharmacotherapy for POAG. Some treatments, such as Timolol or Latanoprost, have large bodies of evidence, while many others have far fewer – and smaller – trials assessing their efficacy . This network enables the comparison of 14 active medications, as well as placebo, for POAG.
While the ability to summarize large bodies of evidence is also possible for traditional meta-analyses, NMAs provide comparative effectiveness data between competing treatments. It is important to note that the evidence provided by an NMA is subject to the limitations of the individual RCTs included within the network . In addition, the ranking of interventions by NMAs using methods such as the Surface Under the Cumulative Ranking Curve (SUCRA) approach is problematic – despite this currently being the most common form of treatment ranking in NMAs. This approach ranks all treatments within a network from “best” to “worst” for each analyzed outcome, but only considers the effect estimate and not the associated precision or the certainty of evidence . Thus, interventions supported by small, low-quality trials that report large effects are ranked highly. Minimally or partially contextualized approaches, instead, consider the magnitude of effect in the context of patient importance as well as the certainty of evidence [13, 14].
How can you have certainty in the findings of an NMA?
Like all study designs, there are considerations when evaluating the credibility of the findings of an NMA. These include the same issues that should be considered when evaluating a traditional pairwise meta-analysis, such as the rigor of the literature search, risk of bias among included trials, consistency of effect estimates contributing to pooled effects (heterogeneity), precision of the pooled effect estimate, publication bias, and directness of the included evidence in relation to the primary research question [8, 9, 15, 16]. However, there are two additional considerations that are specific to NMAs: incoherence and transitivity [8, 9, 15, 17].
Incoherence exists when the direct and indirect estimates for a comparison are not consistent with one another . A meta-epidemiological study of 112 published NMAs found inconsistent direct and indirect treatment effects in 14% of the comparisons made . This means that while in most cases it is appropriate to combine indirect and direct evidence, this is not always the case, and review authors should formally explore this issue. In the presence of incoherence, the higher certainty evidence should be presented rather than the network estimate. If the direct and indirect effects are both supported by the same certainty of evidence, then the network estimate can be used but should be downgraded one level for incoherence. The GRADE approach is increasingly used for rating the certainty in evidence for network estimates, which incorporates these aforementioned criteria [11, 15,16,17]. A GRADE rating can assign high, moderate, low, or very low certainty in the evidence [11, 15,16,17]. Clinicians should take the certainty of the evidence in consideration when determining the impact findings would have on their clinical practice, as lower certainty evidence provides less confidence in the results.
Transitivity refers to the similarity between study characteristics that allows indirect effect comparisons to be made with the assurance that there are limited factors that could modify treatment effects, aside from the intervention under investigation [6, 15]. Essentially, transitivity refers to the inclusion of studies that fundamentally address the same research questions within the same population . Intransitivity can result in biased indirect estimates, which would then impact the overall findings of the network estimates [15, 17]. As previously discussed, incoherence exists when discrepancies between direct and indirect estimates are present, thus, transitivity is a common cause of incoherence .
Clinicians cannot be expected to evaluate transitivity and incoherence within an NMA and authors should clearly report on these two important aspects. Indeed, the absence of reporting should lead readers to question the findings. Table 1 provides an example and overview of the core items for readers to identify for critical appraisal of published NMAs, as applied to the Li et al. (2016) POAG study [10, 19]. These criteria are based on the Users’ Guides to the Medical Literature: Essentials of Evidence-Based Clinical Practice .
Rigorously conducted and reported NMA may provide helpful information for advancing evidence-based ophthalmology, specifically in the common scenario in which multiple treatment options exist. However, clinicians should appraise the quality of NMAs before accepting the results, and even rigorously conducted NMAs cannot provide high certainty evidence if the primary trials eligible for review are flawed.
Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet Lond Engl. 2017;390:415–23. https://doi.org/10.1016/S0140-6736(16)31592-6.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009;6:e1000097. https://doi.org/10.1371/journal.pmed.1000097.
Egger M, Smith GD, Phillips AN. Meta-analysis: principles and procedures. BMJ. 1997;315:1533–7. https://doi.org/10.1136/bmj.315.7121.1533.
Crowther M, Lim W, Crowther MA. Systematic review and meta-analysis methodology. Blood. 2010;116:3140–6. https://doi.org/10.1182/blood-2010-05-280883.
Leucht S, Kissling W, Davis JM. How to read and understand and use systematic reviews and meta-analyses. Acta Psychiatr Scand. 2009;119:443–50. https://doi.org/10.1111/j.1600-0447.2009.01388.x.
Mills E, Thorlund K, Ioannidis J. Demystifying trial networks and network meta-analysis. BMJ. 2013;346:f2914 https://doi.org/10.1136/bmj.f2914.
Neupane B, Richer D, Bonner AJ, Kibret T, Beyene J. Network meta-analysis using r: a review of currently available automated packages. PLOS ONE. 2014;9:e115065. https://doi.org/10.1371/journal.pone.0115065.
Foote CJ, Chaudhry H, Bhandari M, Thabane L, Furukawa TA, Petrisor B, et al. Network meta-analysis: users’ guide for surgeons: Part I – credibility. Clin Orthop Relat Res. 2015;473:2166–71. https://doi.org/10.1007/s11999-015-4286.
Chaudhry H, Foote CJ, Guyatt G, Thabane L, Furukawa TA, Petrisor B, et al. Network meta-analysis: users’ guide for surgeons: Part II - certainty. Clin Orthop. 2015;473:2172–8. https://doi.org/10.1007/s11999-015-4287-9.
Li T, Lindsley K, Rouse B, Hong H, Shi Q, Friedman DS, et al. Comparative effectiveness of first-line medications for primary open-angle glaucoma: a systematic review and network meta-analysis. Ophthalmology. 2016;123:129–40. https://doi.org/10.1016/j.ophtha.2015.09.005.
Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630. https://doi.org/10.1136/bmj.g5630.
Mbuagbaw L, Rochwerg B, Jaeschke R, Heels-Andsell D, Alhazzani W, Thabane L, et al. Approaches to interpreting and choosing the best treatments in network meta-analyses. Syst Rev. 2017;6:1–5. https://doi.org/10.1186/s13643-017-0473-z.
Brignardello-Petersen R, Florez ID, Izcovich A, Santesso N, Hazlewood G, Alhazanni W, et al. GRADE approach to drawing conclusions from a network meta-analysis using a minimally contextualised framework. BMJ. 2020;371:m3900. https://doi.org/10.1136/bmj.m3900
Brignardello-Petersen R, Izcovich A, Rochwerg B, Florez ID, Hazlewood G, Alhazanni W, et al. GRADE approach to drawing conclusions from a network meta-analysis using a partially contextualised framework. BMJ. 2020;371:m3907. https://doi.org/10.1136/bmj.m3907.
Brignardello-Petersen R, Bonner A, Alexander PE, Siemieniuk RA, Furukawa TA, Rochwerg B, et al. Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. J Clin Epidemiol. 2018;93:36–44. https://doi.org/10.1016/j.jclinepi.2017.10.005.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94. https://doi.org/10.1016/j.jclinepi.2010.04.026.
Brignardello-Petersen R, Mustafa RA, Siemieniuk RAC, Murad MH, Agoritsas T, Izcovich A, et al. GRADE approach to rate the certainty from a network meta-analysis: addressing incoherence. J Clin Epidemiol. 2019;108:77–85. https://doi.org/10.1016/j.jclinepi.2018.11.025.
Song F, Xiong T, Parekh-Bhurke S, Loke YK, Sutton AJ, Eastwood AJ, et al. Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study. BMJ. 2011;343:d4909. https://doi.org/10.1136/bmj.d4909.
Mills E, Ioannidis JPA, Thorlund K, Schünemann HJ, Puhan MA, Guyatt GH. Chapter 24: network meta-analysis. In: Guyatt G, Rennie D, Meade MO, Cook DJ (eds). Users’ guides to the medical literature: a manual for evidence-based clinical practice. 3rd ed. McGraw Hill Education; 2015. 327–56.
MRP: Nothing to disclose. DHS: Consultant: Gyroscope, Roche, Alcon, BVI; Research Funding for IIS: Alcon, Bayer, DORC, Gyrsocope, Boehringer-Ingelheim – unrelated to this study. CCW: Consultant: Acuela, Adverum Biotechnologies, Inc, Aerpio, Alimera Sciences, Allegro Ophthalmics, LLC, Allergan, Apellis Pharmaceuticals, Bayer AG, Chengdu Kanghong Pharmaceuticals Group Co, Ltd, Clearside Biomedical, DORC (Dutch Ophthalmic Research Center), EyePoint Pharmaceuticals, Gentech/Roche, GyroscopeTx, IVERIC bio, Kodiak Sciences Inc, Novartis AG, ONL Therapeutics, Oxurion NV, PolyPhotonix, Recens Medical, Regeron Pharmaceuticals, Inc, REGENXBIO Inc, Santen Pharmaceutical Co, Ltd, and Takeda Pharmaceutical Company Limited; Research funds: Adverum Biotechnologies, Inc, Aerie Pharmaceuticals, Inc, Aerpio, Alimera Sciences, Allergan, Apellis Pharmaceuticals, Chengdu Kanghong Pharmaceutical Group Co, Ltd, Clearside Biomedical, Gemini Therapeutics, Genentech/Roche, Graybug Vision, Inc, GyroscopeTx, Ionis Pharmaceuticals, IVERIC bio, Kodiak Sciences Inc, Neurotech LLC, Novartis AG, Opthea, Outlook Therapeutics, Inc, Recens Medical, Regeneron Pharmaceuticals, Inc, REGENXBIO Inc, Samsung Pharm Co, Ltd, Santen Pharmaceutical Co, Ltd, and Xbrane Biopharma AB – unrelated to this study. JWB: Nothing to disclose. RRB: Research funds: Pfizer – unrelated to this study. LT: Nothing to disclose. MB: Research funds: Pendopharm, Bioventus, Acumed – unrelated to this study. VC: Advisory Board Member: Alcon, Roche, Bayer, Novartis; Grants: Bayer, Novartis – unrelated to this study.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Phillips, M.R., Steel, D.H., Wykoff, C.C. et al. A clinician’s guide to network meta-analysis. Eye 36, 1523–1526 (2022). https://doi.org/10.1038/s41433-022-01943-5