Introduction

The review article has long been a staple of journals and a ready source of information for clinicians. With the rapid increase in dental publications referenced in Medline from fewer than 1000 per year in 1960 to over 18,000 in 2008 (see figure 1) reliance on review articles is unlikely to diminish. Concerns that traditional narrative may result in biased conclusions1, 2, 3 along with this explosion of healthcare publishing and the development of evidence-based healthcare have led to the development of the systematic review.

Figure 1
figure 1

Dental papers in Medline 1960-2008

Full size image

The systematic review aims to identify, evaluate and summarise the findings of all relevant individual studies on a specific topic. An explicit systematic approach is used to minimise bias and provide reliable findings on which to base conclusions. The Cochrane Handbook4 (www.cochrane-handbook.org/) identifies five key characteristics of the format:

  • a clearly stated set of objectives with pre-defined eligibility criteria for studies;

  • an explicit, reproducible methodology;

  • a systematic search that attempts to identify all studies that would meet the eligibility criteria;

  • an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and

  • a systematic presentation, and synthesis, of the characteristics and findings of the included studies

Meta-analysis, the use of statistical approaches to summarise the results of independent studies, is also often included in systematic reviews.

Guidance for undertaking reviews has been developed over the years and two key documents are the Cochrane Handbook4 and the University of York's Centre for Reviews and Dissemination guidance document.5 There are also clear guidelines on how systematic reviews should be reported: the current guidelines, PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)6 have replaced the QUOROM statement (Quality Of Reporting of Meta-analyses). Guidelines are also available for meta-analysis of observational studies.7

Despite the availability of these guidance papers there is still a great deal of variation in the quality of systematic reviews, for example Glenny et al.8 highlighted that only 19% of dental systematic reviews attempted to identify all relevant studies. Because of this it is important that readers should be in a position to critically appraise systematic reviews.

Appraisal questions and checklists

As with randomised controlled trials9 a number of critical appraisal questions or checklists are available (see Table 1). There are also a number of textbooks and websites that provide information on critical appraisal (for links, see www.cebd.org/practising-ebd/appraise/resources-for-appraising/).

Table 1 Main appraisal questions from Centre for Evidence-based Medicine (CEBM) and Critical Appraisal Skills Programme (CASP) worksheets
Full size table

Despite the range of materials available for appraisal of papers there are only three essential questions that need to be asked of any paper:

  • Is the study valid?

  • What are the results?

  • Are the results relevant locally?

Is the study valid?

One of the first issues to address is whether the review's authors have clearly stated the question or questions they are trying to answer. The majority of systematic reviews address a narrowly focussed questions; for example, ‘Is vitamin B12 effective for the treatment of minor oral ulceration?’ compared with a broader question as might be found in a more traditional review such as; ‘a review of the management of oral ulceration’. Systematic reviews will tend to have clearly defined the population group, the types of intervention being compared and the specific outcomes being considered.

Depending on the question being addressed you need to assess if the right type of study designs have been included. If the authors are looking at the effectiveness of a particular treatment this will ideally be randomised controlled trials. If however, they are looking at the potential for malignant change in oral lesions they should be looking to include cohort studies. That is, the type of study to be included should be based on the question being asked.

When undertaking a systematic review it is important to search a range of databases to ensure that all potentially relevant studies can be identified. For example, the York review of water fluoridation11 searched 25 electronic databases: the majority of reviews’ searches seem to utilise just one or two. The number of databases that needs to be searched will depend on the topic under review but for most dental reviews the use of Medline, Embase and the CENTRAL database of the Cochrane library seems to me to be a minimum requirement. Ideally, additional contact with experts in the field together with the use of grey literature (material that is not formally published, such as institutional or technical reports, working papers, conference proceedings) and not restricting results to English-language papers only will increase the potential to identify studies. If only some of these are undertaken the reader must consider the likely impact on the review's conclusions.

Assessing the quality of the studies included in the review is key stage and a wide range of tools have been devised.12, 13 Most of these tools are scales that include criteria which the Cochrane handbook4 considers do not directly relate to internal validity. The Cochrane Handbook recommends the use of a domain-based evaluation to assess the risk of bias in studies. This should be conducted independently by two reviewers.

The final validity question is related to whether the studies have been combined in a meta-analysis and whether it was reasonable to do this. What should be considered is whether the results of each of the individual studies have been clearly displayed. If different studies designs have been included it is good to see this clearly identified or presented separately. Check to see whether the results of the study are similar from study to study, this is potential heterogeneity and can be tested for statistically. If heterogeneity is marked meta-analysis may not have been carried out, whether or not a meta-analysis has been carried out the review should include discussion of any reasons for variability between studies.

Sometimes the results of studies are presented in graphical format, a forest plot (figure 2). Each study is represented and the mean value (square box) and 95% confidence intervals (lines either side of box) are shown for each study together with the mean for all the studies (diamond). As a rule of thumb if it is possible to connect each of the studies by a vertical line this is an indication of whether the studies are similar enough to combine. This is possible for most of the studies in the example shown.

Figure 2
figure 2

Schematic representation of a forest plot

Full size image

What are the results?

If a meta-analysis has been conducted the results may have been presented using a range of effects measures, eg. odds ratios, risk ratios, standardised mean difference, prevented fraction. A detailed appraisal of each of these is beyond the scope of this article and will be dealt with in future issues of Evidence based Dentistry. Key points to look for in the presentation of the results, however, are: that clear results are presented for each of the outcomes considered; and that these results are presented together with confidence intervals (CI) to allow the reader to estimate the precision of the result. For example, in the Cochrane review of fluoride toothpastes14 it was found that the pooled PF for decayed, missing or filled surfaces was 24%, with a 95% CI of 21–28%, which is narrow and therefore precise. If the CI had been 9–39%, in contrast, this wider range would show a lack of precision.

Another issue to consider is the type of outcome under consideration. Suppose we are looking at studies to reduce caries. One outcome measure may be a reduction in the number of decayed missing and filled teeth. Looking at a forest plot we would want to see the summary diamond on the left. One way of remembering this is to think; less on the left, in this case less decay. Another potential positive outcome would be an increase in the number of children with no disease. In this case we would be looking for the summary diamond to be on the right, indicating more children free of disease. To remember this think of the R in more to indicate right.

Are the results relevant locally?

Once you are happy with the methodology and the results the next thing to consider is whether these results can be applied to your own patients, The CEBM appraisal questions in table 1 only use this one important question whereas the CASP questions have some additional bullet points in the full worksheet that can help. Two of the most important are, to consider if your patients or setting is so different for that in the review that you could not apply the approach, or whether there are any important outcomes, for example potential side effects that have not been considered that would impact upon you using this approach in your practice. If not, the next thing to consider is how you might implement this treatment or approach in your practice and whether they may be any barriers to you doing so.

As with critically appraising randomised controlled trials10 regular practice with like minded colleagues is a good way of developing your critical appraisal skills.