Although the research process, extending from primary evidence creation through to guideline production for delivery at the point of care, should functionally act as a continuum, it consists of different stages. There are many points along this route at which research can fail to deliver its promises of improved patient care. One of these is where the findings of clinical trials cannot be compared to, or combined with, other trials. Failure to combine trials into systematic reviews, with or without meta-analyses, means it is not possible to determine the relative overall benefit of a particular intervention with any certainty. This can hamper the uptake of new evidence into practice or might prevent discontinuing use of an ineffective treatment.

The role of outcomes and outcome measures in trials

If we look at a simple two arm clinical trial comparing intervention A to intervention B for managing dental caries, the outcomes are the endpoint(s) that we want to measure to assess if there is a difference between the two (A or B). Examples of a few typical outcomes in caries clinical trials are shown in Table 1. Outcomes play an important role in research and they are used to calculate the number of patients required (sample size) to potentially show a difference between two (or more) interventions. They should be determined prior to the start of the trial to decrease the potential introduction of bias from an ad hoc selection of outcomes that show a particular difference.

Table 1 Examples of outcomes and outcome measures commonly used in clinical trials related to caries management
Full size table

An outcome measure is the index or measurement instrument used to quantify the outcome (see Table 1) and is equally important. Outcome measures should have: face, content and structural validity; internal consistency; reliability; responsiveness; low measurement error; construct validity; cross-cultural validity; and be feasible to use.

Outcomes can be thought of as WHAT is measured in the trial and outcome measures as HOW that is measured.

Types of outcomes and outcome measures

The most commonly used endpoints in dentistry are clinical outcomes, surrogate outcomes or composite outcomes.

Surrogate outcome measures are often used for outcomes that are undesirable (death) or when the number of events is quite small or impractical to use in a clinical trial (for example, requiring a long time to manifest); loss of clinical attachment is a commonly used surrogate outcome for tooth loss. Surrogate outcomes may correlate with primary endpoints but do not necessarily have a guaranteed relationship.

Composite measures involve multiple variables/data items being combined to give a single score; ‘success’ is a commonly used composite outcome in dentistry which can involve (1) lack of pain/discomfort; (2) lack of caries progression; (3) lack of radiographic progression and (4) no replacement restoration required.

Difficulties with outcomes and outcome measures

The sheer diversity of outcomes and outcome measures can lead to difficulties when trying to combine study results in a systematic review. Systematic reviews aim to identify, collect, appraise and systhesise data from relevant clinical trials for a particular clinical problem; the diversity of outcomes can lead to complications at this stage of the research process. Although the trials identified may be of high quality, if the outcomes they have investigated are not similar, they may not be able to combine their data (comparing apples with oranges); this can result in wasted opportunities to improve patient care. It can be difficult or impossible to meaningfully compare or combine the findings of one study with those of another for the same intervention because both use different outcomes; how does a restoration survival rate compare with the modified United States Public Health Service/Clinical criteria.1 or with caries incidence?

Even if the trials do measure the same outcome, each may have used different outcome measures. In a study of an intervention for prevention of caries, where the outcomes in both studies measure caries progression, how can the International Caries Detection and Assessment System (ICDAS)2 be combined with Nyvad's criteria?3

To improve the efficiency of clinical trials and outcome measures, the COMET (Core Outcome Measures in Effectiveness Trials) database (http://www.comet-initiative.org/), an internet based open source knowledge base, brings together researchers and trialists to develop core outcome sets (COS).

Reporting bias

Even when a group of studies share outcomes and outcome measures enabling them to be brought together into a review and meta-analysis, where outcomes have not been stated prior to the study commencing, there is a potential for reporting bias. This subject has received little attention, yet can bias results to give an overly optimistic picture of the effect of an intervention. Outcomes may have been measured but then not included in the study report because the findings were not positive or statistically significant or did not fit with the researchers' hypotheses. It can be legitimate to change an outcome during the course of a study but this has to stated clearly in the report and the reasons explained. These changes can be extensive and often go unreported. In one review4 of publications 40-62% failed to report on at least one primary outcome, changed it, or introduced a new one when compared to the published protocol. This extensive changing or under-reporting of outcomes is indicative that, at the design stage of trials, there is often inadequate care taken in choosing the most important outcomes. This has been corroborated in interviews with 59 trialists5 where outcome choice was viewed as a challenging area in trial design. Outcomes were often chosen on the basis of the trial designers' own experiences and through knowing experts rather than through rational and systematic methodology. An assessment of Cochrane Oral Health Group protocols and their published systematic reviews held in the Cochrane Database of Systematic Reviews, was carried out recently. Although there was no evidence of selective outcome reporting in the 152 reviews assessed, 11% of the primary outcomes were downgraded to secondary outcomes, 10% of primary outcomes were not reported at all and 19% were new in the final publication. The use of standardized outcomes was suggested by the authors as a solution to reduce these discrepancies in outcome reporting6.

Current outcomes and outcome measures in caries clinical trials and systematic reviews

Restorative studies traditionally have assessed materials used to restore caries but more recently investigations have looked at different ways of managing caries itself prior to placement of a restoration. These include selective (also known as partial, incomplete, ultraconservative), stepwise or no, caries removal.

A search for ‘Dentistry and oral health’ in the COMET database (http://www.comet-initiative.org/ accessed 1st July 2015) returned 22 articles. Five of these were reviews of the literature (one on orthodontics, two on periodontology, two on implantology), six were recommendations from groups discussing improvements to general trial methodology (two on implantology, three on periodontitis, one on plaque and gingivitis) and 10 were on the development of a COS. However, of these, four were unpublished or still underway (three being undertaken by the Cochrane Oral Health Group, in the areas of periodontology, oral medicine and oral mucositis in cancer patients and one in traumatology). Of the remaining published COS, three were in periodontology and the remaining three dealt with caries.

The first of these three7 deals specifically with the development of a single COS in a trial comparing safety of amalgam. The outcomes of interest were developed by the group to specifically test performance in neurobehavioural domains, nerve conduction measurement and urinary glutathione transferase levels.

The second paper8 is a consensus statement from an international group looking at the conduct of caries clinical trials as a whole. They present recommendations for consideration around outcomes rather than an outcome set.

The final one9 takes a Delphi consensus approach to developing a COS for clinical trials of pulp treatments in primary teeth. The aim was to ‘Develop a core set of component outcomes to be part of a composite outcome defining the failure of a pulp treatment’. A report on a search of the literature included 47 randomised control trials with 83 reported outcomes (a median of 11 per trial). These were grouped into 24 overarching outcome categories and, following three rounds of a Delphi process involving 210 researchers/ clinicians (participation rate 25-30%) five component outcomes were left: soft-tissue pathology; pain; pathologic mobility; pathologic radiolucency; and pathologic root resorption. However, the consensus process was built on clinicians and researchers with no patient or other stakeholder involvement. Giving a voice to patients increases the likelihood that the outcomes will be of relevance to the end-user – the patient.

Twelve reviews in the Cochrane Oral Health Group database (accessed 1st July 2015), deal with ‘Dental caries treatment’. Of these, all four reviews where more than one clinical trial was included, commented on the inadequacy of outcomes and outcome measures in the trials: ‘lack of consistency between different outcome measures’;10 ‘insufficient outcome data on cost...’;11 ‘future research should also investigate patient-centred outcomes .... health economic measures should be used to determine the cost of treatment and patient willingness to pay’;12 ‘it was disappointing to note that so few trials could be included into this review that sought to compare different dental fillings for the same type of outcome’.13

There is clearly a need for standardisation of outcomes and outcome measures in clinical trials related to management of dental caries, both for what to measure as well as how to measure it.

Barriers to developing a core outcome set and standardised outcome measures

There are a number of barriers to developing, agreeing and universally using a COS and measures. Although it is widely acknowledged that researchers alone should not develop these and that clinicians, industry, funders, patients and other stakeholders should be included in this process, these can be difficult groups to engage in something fairly intangible and with limited immediate benefit. Patients might be an especially difficult group to access, engage and explain the need to solicit their opinions. Dental caries is so widespread that it is not viewed as disease but almost normalised, the impact on daily life is accepted and, as such, there are no specific patient interest groups to access as there are in the cases of diseases such as diabetes, orofacial clefting etc.

Further hurdles to be crossed might be that researchers who have developed their own outcome measures are likely to feel more positively towards their own than another one. Also, familiarity with measures might give them more weighting in being considered as part of a standardised set.

Once a COS and standardised measures have been agreed, it is essential that a comprehensive process of dissemination is undertaken to ensure that researchers know about them and use them.

Future steps

Outcome set development for studies in oral healthcare is progressing but there is a need for a significant move away from the recommendations of groups of specialists towards formal development involving a wide range of stakeholders' opinions.

There is no COS for dental trials of interventions to manage caries, and without this we are wasting a huge amount of time and effort, not to mention money, by continuing to produce single studies that cannot be combined or compared with others.

Initial steps have been taken to develop a COS for management of dental caries.14 It is hoped that when this work is complete, it will encourage trialists to use outcomes in their trials chosen by consensus, that are relevant to patients and can be compared and combined with similar studies.