Introduction

In recent years there has been increased emphasis on considering evidence derived from reputable research to make clinical decisions.1 The directive to base clinical decisions on quality evidence, however, has presented some difficult challenges to practising clinicians: finding the evidence, appraising it, and integrating it. This is a difficult and lengthy process.1 Review articles must be looked to as the source of research summaries: narratives formerly filled this role but have been shown to be highly prone to bias.2, 3, 4, 5 SR also find, appraise and integrate the evidence, but ideally do so in a nonbiased fashion and therefore can be a high-quality source of evidence-based recommendations for clinical decisions.5, 6, 7, 8

For a review to provide an accurate, complete and unbiased synopsis of the evidence, intuitively one understands that it must first find all the evidence. It was shown recently that the quality of the literature-search and selection methods reported in dentistry SR as a whole have consistently improved over the last 5 years.9 Although it is reassuring for clinicians that significant efforts have been made to find all available literature, it is not known how the trends break down within the individual branches of dentistry.

The profession of dentistry is broad and further analysis is required to more accurately advise clinicians of the shortcomings in different disciplines. Furthermore, the knowledge gained by analysis of each speciality can help identify and guide where further improvement is needed most.

It is the purpose of this study to investigate and compare the reported literature-search and selection methods according to dental speciality. The null hypothesis to be tested is that there is no difference in the reported SR literature-search and selection methods between the dental specialities.

Methods

To assess the reported literature-search and selection methods of dentistry SR, all those reviews published in the English language between January 2000 and June 2006 were located, then categorised by speciality, and finally evaluated according to an adaptation9 of the guidelines outlined in the Cochrane Handbook for Systematic Reviews of Interventions.10

To identify all dentistry SR, the search strategy employed by Major et al.9 was utilised with additional MeSH (Medical Subject Headings) and key word search terms. The truncation pattern proposed by Stamm and Hohoff11 was used for all pertinent dental specialities. The following search strategy was developed for the Ovid interface of Medline with the help of a senior health sciences librarian:

(systematic review$.mp OR meta-analys$.mp OR exp meta-analysis/ OR exp “review literature”/ OR exp evidence-based medicine/) AND (exp dentistry/ OR dentistry.mp OR dental.mp OR tooth.mp OR teeth.mp OR (stomatognathic disease.mp NOT pharyngeal disease.mp) OR endodon$.mp OR orthodon$.mp OR oral surger$.mp OR oral surgic$.mp OR periodon$.mp OR prosthodon$.mp OR pedodon$.mp OR pediatric dentistry.mp OR paediatric dentistry.mp OR dental public health.mp OR oral pathology.mp OR oral medicine.mp)

Searches were limited to human studies and those published in the English language. Key words (example.mp) were used for all databases. For Embase all key words were used but Medline MeSH terms (exp example/) were changed to the following EM-tree (Elsevier's Life Science Thesaurus) terms: (exp meta-analysis/ OR exp “systematic review”/) AND (exp dentistry/ OR exp preventative dentistry/ OR reparative dentistry/).

Specific parts of the Cochrane Library (Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, and the Health Technology Assessment Database), Medline, Embase, Pascal and Web of Science electronic databases were searched from 1 January 2000 to 14 June 2006. Electronic searches were followed up with secondary searches including: hand searching bibliographies of identified SR and other articles pertaining to evidence-based dentistry, SR known to the authors prior to the study and SR encountered while performing literature searches for other research projects.

Reviews were included as SR if a search and selection procedure was outlined in the Methods.12 Initial screening of articles by title and abstract was performed independently by two researchers (MM and CF) to identify potential SR. Discrepancies were resolved by discussion until all parties were in agreement. Identified potential SR were then retrieved in full for a second selection process (by MM and CF) to verify potential reviews as being truly systematic.

Categories for classification were derived from the ADA's recognised specialities.13 Each SR was allocated to one of the following categories: DPH, endodontics, oral and maxillofacial pathology and medicine (OMPM), OMR, OMS, orthodontics, paediatric dentistry, periodontology, or prosthodontics and restorative dentistry (PROS). The following potentially ‘grey' topics were allocated as follows: fluoride treatment and all preventative dentistry to DPH; temporal mandibular dysfunctions and orofacial pain to OMPM; implant placement to OMS; and biomaterials and caries diagnosis to PROS.

Evaluation of search and selection methods used in the identified SR was performed according to Major et al.9 and therefore followed an adaptation of the guidelines outlined in the Cochrane Handbook for Systematic Reviews of Interventions.10 The criteria for evaluation were:

  1. 1

    The database names and date ranges searched were documented

  2. 2

    Electronic searches were performed in at least two databases

  3. 3

    Electronic searches were followed up with secondary searches

  4. 4

    All electronic search terms and applicable truncations were presented in a clear and reproducible manner

  5. 5

    How the terms were combined (ie, Boolean operators AND, OR, NOT) were presented in a clear and reproducible manner

  6. 6

    Clear and reproducible inclusion–exclusion criteria were provided

  7. 7

    Included articles were selected by a team of two or more reviewers

  8. 8

    There was inclusion of all languages

The listed criteria were evaluated dichotomously; if a criterion was met by the SR it was marked “1” whereas reviews failing that criterion were marked “0”. Evaluation of every article identified was performed independently by two researchers (MM and CF). Assessments were recorded using Excel (Microsoft, Redmond, Washington, USA) spreadsheets and results were systematically compared. Inclusion of all languages in the abstracts/ articles searched in the SR was considered important. This will increase the possibilities of including all evidence when writing a SR for clinical decision-making.

Pearson's chi-squared analysis was used to compare the quality of search and selection methods between dental specialities. The proportion of SR per speciality that met each criterion was calculated and each speciality assigned a rank within each category relative to the other specialities. An average rank was then calculated to determine which speciality had the largest proportion of SR meeting the search and selection methods.

Results

Two hundred and seventy-four published reviews (full list available online) were found to meet the definition of a dental systematic review. The majority of dental SR were published either in the fields of periodontology (57 reviews; 20.8% of SR) or DPH (50 reviews; 18.2% of SR). Of the remaining specialities the numbers of published SR were as follows: 40 in OMPM; 38 in OMS, including placement-of-implant reviews; 34 in PROS; 27 in orthodontics, 15 in endodontics, seven in OMR, and six in paediatric dentistry (Fig. 1). Two SR were not included for further analysis because they could not be classified into one of the nine recognised specialities (Table 1). In the case of duplicates involving a Cochrane Systematic Review and a peer-reviewed publication generated from it, only the Cochrane Systematic Review was considered for the analysis.

Figure 1
figure 1

Systematic reviews by dental speciality published January 2000–June 2006 DPH, Dental Public Health; Endo, endodontics; OMPM, oral and maxillofacial pathology and medicine; OMR, oral and maxillofacial radiology; OMS, oral and maxillofacial surgery; Ortho, Orthodontics; Paed, paediatric dentistry; Perio, periodontology; Pros, prosthodontics and restorative dentistry.

Full size image
Table 1 Articles qualifying as systematic reviews but not eligible for analysis
Full size table

Four categories of search and selection methodology were found to be significantly different between dental specialities (Table 2): the documented electronic search-date ranges (P 0.003); documentation of inclusion–exclusion criteria (P 0.017); article selection by two or more reviewers (P 0.014); and inclusion of all languages (P 0.014). No significant differences were found between the other four categories of methodology (Table 2), ie, searches in two or more databases (P 0.306), searches by hand (P 0.640), documentation of search terms (P 0.160), and documentation of search strategy with Boolean operators (P 0.118).

Table 2 Systematic review (SR) evaluation summary: literature search and selection criteria by dental speciality
Full size table

On average, 76.6% of reviews documented the range of dates within which searches of databases were made. Within several specialities the figures were much lower, however: only 42.9% of OMR SR, 33.3% of paediatric dentistry SR and 58.8% of PROS reviews properly documented their database search-dates. Periodontology reviews were particularly good, in contrast, with 89.5% documenting their database search-dates (Table 2).

A mean of 91.2% of reviews documented their inclusion–exclusion criteria, but only 50% of paediatric dentistry SR reported them (Table 2).

On average 51.5% of reviews used two or more reviewers to select their articles for inclusion. Periodontology and DPH used more than the average number of reviewers to select articles (two or more reviewers in 63.2% and 60.0% of SR, respectively), whereas reviews in the areas of OMPM (37.5%), OMR (0%), paediatric dentistry (33.3%) and PROS (38.2%) were less likely than average to use two or more reviewers (Table 2).

Regarding the inclusion of all languages, the mean was only 25.7% of SR doing this; those in OMR (0%), periodontology (12.3%) and PROS (14.7%) had even fewer with below-average numbers of reviews utilising literature from all languages. Of note, OMS (44.7%) and DPH (32.0%) were the two areas that included all languages with better than average frequency (Table 2).

Once rank orders for each speciality had been assigned for each search and selection category and an average rank calculated, it could be determined which specialities' SR followed the most literature-search and selection methods. The areas were thus ranked as follows: periodontology reviews met the most criteria followed by OMS, DPH, orthodontics, endodontics, OMPM, paediatric dentistry, PROS and then OMR (Table 2).

Four key areas of SR still require improvement in all dental specialities. Only 65.7% of SR search more databases than Medline alone; only 50.4% document their search strategies with Boolean operators; only 51.5% select articles with two or more reviewers; and only 25.7% include all languages (Table 2).

Discussion

The growing emphasis on evidence-based practice in dentistry has led to new challenges for clinicians, particularly finding strong evidence on which to base their clinical decisions. SR are a powerful tool for clinicians because ideally they attempt to remove bias by systematically finding all available evidence on a topic, evaluating it, and integrating all the available research into an evidence-based conclusion.5, 6, 7, 8 Because of the potential utility of SR, their publication has been increasing in all areas of dentistry.8, 9, 14 What has not yet been examined, however, is whether all areas of dentistry have been equally rigorous in reporting their searches for and selection of literature for review.

The evaluation of SR by speciality showed that there were significant differences between dental specialities in some aspects of literature search and selection methods. These were in documenting database search-dates (P 0.003), documenting inclusion–exclusion criteria (P 0.017), selection of articles by two or more reviewers (P 0.014), and inclusion of all languages (P 0.014). A rank-order analysis derived from the proportion of SR per speciality that met the criteria here results in dental specialities listed in the following (descending) order: periodontology, OMS, DPH, orthodontics, endodontics, OMPM, paediatric dentistry, PROS and OMR.

A previous study showed that dental SR have been improving as a whole over the last 5 years,9 but the current analysis demonstrates how some specialities have performed better at meeting the mandated criteria for literature-search and selection methodology. OMR and PROS SR have lagged behind, although the results for OMR and paediatric dentistry must be interpreted with caution because of the small number of published reviews in these specialities (Fig. 1). The poor results for PROS, however, are more certain since many more SR have been published in that area.

In general, for dental disciplines where treatment is the primary clinical concern, the number of SR per speciality (Fig. 1) was comparable with the number of randomised clinical trials (RCT) per speciality identified by Niederman et al.15 In detail, these then rank as follows:

  • number of RCT: implant placement < endodontics < orthodontics < restorative dentistry < OMS < periodontology < OMPM plus OMR

  • number of SR: paediatric dentistry < OMR < endodontics < orthodontics < PROS < OMS (plus implant placement) < OMPM < DPH < periodontology

Given that there are relatively few well-executed primary research studies in endodontics it is understandable why there are fewer endodontics SR available; the same logic explains the large number of periodontology SR. The low number of reviews in paediatric dentistry cannot be fully understood since Niederman et al.15 did not consider them in his analysis, whether because they overlooked the speciality or because no paediatric dentistry RCT existed. Furthermore, DPH and OMR SR numbers cannot be contextualised with the other specialities because these disciplines usually require different study designs. For DPH, case–control or cohort studies may be the gold standard design depending on the question asked;16 for OMR, diagnosis studies are often the gold standard.17 Although it is true that a case–control or cohort study may be the gold standard in any of the specialities, DPH and OMR in particular do not often lend themselves to RCT design. In any given speciality the selection of the most appropriate research design would vary according to the question being asked and SR of study designs other than RCT are undertaken using established methodologies such as MOOSE (Meta-analysis of Observational Studies in Epidemiology).

Periodontology SR have met the most search and selection methods since 2000, with OMS and DPH following close behind (Table 2). The strong findings for periodontology were anticipated as a natural extension of the speciality's commitment to evidence-based dentistry, as substantiated by the large number of periodontology RCT.15 The strong findings for DPH were also not entirely unexpected because DPH is frequently involved in dental policymaking decisions,18 decisions that require high-quality reviews. There may also be additional factors motivating the use of good SR protocols unknown to the authors of this paper.

Neither the strong findings nor the large number of OMS reviews were expected. Most SR classified as OMS were not on the topics traditionally associated with oral surgery, such as orthognathic surgery or third molar extraction, but rather dealt with implant placement. Perhaps implant review-writing has been a larger question of interest and attracted more thorough SR authors. Also, implant placement SR were frequently published in periodontology journals, journals known to have greater impact and probably more stringent revision processes. There is normally an association between journal impact and quality of their manuscripts.19 It is also possible that the same community of authors publishing periodontology SR is the one doing so for implant SR.

In general, there was a positive correlation between how many SR were published in the speciality (Fig. 1) and how frequently the reviews of a given discipline met the literature-search and selection methods (Table 2).

  • Number of SR: paediatric dentistry < OMR < endodontics < orthodontics < PROS < OMS (plus implant placement) < OMPM < DPH < periodontology

  • SR rank: OMR < PROS < paediatric dentistry = OMPR < endodontics < orthodontics < DPH < OMS < periodontology

This finding suggests that familiarity with SR may be the most important factor in improving review quality. With greater familiarity and exposure, review authors learn to perform and report their methods better, the editorial benchmark rises and, most importantly, the readers demand more methodologically rigorous reviews.

A few important words of caution must be kept in mind when interpreting these results. First, the conclusions drawn in this paper do not allow the reader to say that PROS SR are fundamentally worse than periodontology SR. Instead, periodontology review authors have either done a better job at searching and selecting literature by following more of the mandated criteria, or they have been able to report it better.

Second, there is the unexplored possibility that the peer review and editorial process of publication has influenced the reporting of methodology. Differences in journal styling could account for apparently missing information; similarly, in an effort to reduce article length, authors may have not reported protocols that were performed. This may be especially pronounced with a printed publication where space is more limited than an electronic version. Readers should remember that extra information about methods may be available online from the journals or upon request from the authors. Whereas some caution should be exercised when rating SR based only on the published information, there is increasing expectation from journals that checklists be used, such as QUORUM and MOOSE (www.consort-statement.org/index.aspx?o=1065). These checklists greatly increase the chances that important information has been included in the SR.

Third, it must be kept in mind that other elements not considered by the present study ought to be considered when evaluating overall quality of a review. In addition to the criteria listed here, a high-quality review should also have a protocol for critically appraising the selected articles, must clearly document how the data were extracted, and choose the appropriate method for analysis.10

Nevertheless, a good literature-search and selection protocol is the key and fundamental initial step to a reliable SR. Even when well-conducted, SR that do not sufficiently utilise the evidence may have conclusions misguided beyond reconciliation. Conversely, an outstanding job finding all available evidence may be followed by a substandard critical appraisal or the need for a better or different method of data analysis. For the astute reader this second problem can be more easily overcome as, provided sufficient information is provided, they can draw their own conclusions even if the inferences drawn by the SR authors must be disregarded.

Conclusions

Within the context of the current findings, it can be stated that current periodontology SR present better-reported search and selection methodology than PROS or OMR reviews. Fortunately, the three highest-ranking specialities (periodontology, OMS and DPH; Table 2) are also the three most published specialities.

Although this analysis provides useful guidelines and identifies key areas of concern, it does not replace the need for critical reading of all scientific literature. Therefore dental clinicians should consider SR in their speciality with appropriate levels of awareness. All dental SR require improvement in four key areas: searching more than just Medline, documenting the search strategies with Boolean operators, article selection by two or more reviewers, and inclusion of all languages.