Objective

This article describes where papers come from in a systematic review of complex evidence.

Method

An audit was carried out of how the 495 primary sources for a review were originally identified.

Results

Searches of databases defined in the protocol at the outset of the study were only able to find 30% of sources ultimately identified. Fifty-one per cent were identified by "snowballing" (ie, pursuing references from references lists and electronic citation tracking), and 24% by personal knowledge or personal contacts (Table 1).

Table 1 - Methods of identifying primary data sources in a systematic review of complex evidence.

Full table

Conclusion

Systematic reviews of complex evidence cannot rely solely on protocol-driven search strategies.

Commentary

Systematic reviews have an important place in the development of evidence-based healthcare and the methodology for conducting reviews is constantly developing and improving. This is largely the result of the work by organisations such as the Cochrane Collaboration, Campbell Collaboration, Centres of Evidence-based Health (Medicine, Mental Health, etc) and universities, along with an army of researchers increasingly experienced in systematic review methodology. Important standards for systematic reviews have been developed by the University of York¹ and the Cochrane Collaboration (Cochrane Handbook for Systematic Reviews of Interventions, see www.cochrane.org/resources/handbook/index.htm). Standards for reporting reviews of different study designs have been published.^{2, 3}

One of the key features of these standards for systematic reviews is a focus on thorough searching to identify all the relevant primary sources. Failure to do this impacts upon the quality of the systematic review, a fact noted by Glenny in a recent assessment of quality systematic reviews in dentistry.⁴ She found that only 12 out of 65 reviews (19%) demonstrated that they had attempted to identify all relevant studies — disappointing, considering they were published between 1996 and 2001.

Medline was used by 97% of Glenny's sample, 26% used Embase, 15% used the Cochrane Clinical Trials database and 6% used BIDS (the Bath Information and Data Services). Only eight reviews had searched for published and unpublished literature and considered all languages and, of these eight, only five had carried out hand-searching in addition to electronic database searching.

The findings of this simple audit of primary source identification, during the conduct of a systematic review of the diffusion of innovations,^{5, 6} yet again highlight the limitations of only searching electronic databases to identify primary sources for systematic reviews.

Practice point

• Systematic reviewers need to consider a wide range of search methods to ensure comprehensive identification of the available literature when conducting reviews.

References

1. Centre for Reviews and Dissemination Guidance for Those Carrying Out or Commissioning Reviews. Undertaking systematic reviews of research on effectiveness. In CRD Report No. 4. 2nd edn. Edited by KS Khan, Gerben ter Riet, Julie Glanville, Amanda J Sowden and Jos Kleijnen. York: University of York; 2001.
2. Moher D, Cook D, Eastwood S, Olkin I, Rennie D, Stroup D, for the Quorom Group. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999; 354:1896–1899. | Article | PubMed | ISI | ChemPort |
3. Stroup D, Berlin J, Morton S et al., for the meta-analysis of observational studies in epidemiology (MOOSE) group. Meta-analysis of observational studies in epidemiology. A proposal for reporting. J Am Med Assoc 2000; 283:2008–2012. | Article | ISI | ChemPort |
4. Glenny AM, Esposito M, Coulthard P, Worthington HV. The assessment of systematic reviews in dentistry. Eur J Oral Sci 2003; 111:85–92. | Article | PubMed | ISI | ChemPort |
5. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med 2005; 61:417–430. | Article | PubMed | ISI |
6. Greenhalgh T, Robert G, Bate P, Kyriakidou O, Macfarlane F. Diffusion of Innovations in Health Service Organisations: A Systematic Literature Review, Oxford: Blackwell; 2005.