Introduction

In recent years there has been growing public concern that dental care might not be meeting required standards. Additionally there has been a greater willingness on the part of patients to complain and to litigate, with a doubling of complaints and claims to one dental protection society during the 1990s.1 As a result regulatory authorities have moved to improve protection of the public from poorly performing and out of touch healthcare professionals. Poor performance procedures,2 recertification3 and the commitment to revalidation,4,6 have all been part of the dental profession's response.

The term 'revalidation' has sometimes been held to mean the same as recertification.5 However, developments in the regulation of the practice of dentistry have introduced clear differences in the respective processes, which are now distinguished as separate strands of a quality assurance process for the practice of dentistry in the United Kingdom. Recertification for dentists in the UK is defined by the General Dental Council (GDC) as completing the required number of hours of Continuing Professional Development (CPD) and being able to produce certificates to prove this upon request.3 Currently, to maintain a dentist's name on the register requires participation in 250 hours of continuing professional development over five years. This means an average of 50 hours per year. Of the 50 hours, 15 (75 hours over five years), needs to be verified by an approved educational body.3,7,8

Revalidation is more demanding than recertification,8 requiring the individual to demonstrate that the activities undertaken in the revalidation period have had the effect of maintaining fitness to practise. Thus revalidation, in addition to participation in continuing education, encompasses participation in a range of activities such as: clinical audit, presentation of evidence of clinical performance, structured reflection upon practice and evidence of consequent change in practice (implicitly for the better). The medical profession has been moving towards revalidation since the year 2000.9

In 2002 the GDC committed itself in principle to the introduction of a scheme of revalidation.6 Accordingly a pilot scheme of revalidation for use by general dental practitioners was devised to explore the acceptability and practicalities of revalidation for the profession. Non-dental involvement in the assessment process was sought to engender wider ownership. Information on the required time implications was also collected.

The objectives of the study were to

  • Explore how revalidation might work in practice

  • Assess the acceptability of a scheme to the profession

  • Ensure that the scheme fits with clinical governance requirements

  • Assess the likely time required to produce the evidence required to facilitate revalidation.

Methods

A steering group for the project, comprising representative dentists from the profession, reached a consensus that a portfolio format was most likely to achieve a cost effective and acceptable result. Portfolios do have the potential to be used in the assessment of clinical performance over a period of time.10,11,12 Figure 1 shows a summary of the sections of the portfolio, together with the outcome measures used to assess that section. The measures for the three blank sections at part 5 were taken to be the research data, publications or lectures themselves.

Figure 1
figure 1

The GDP revalidation outcomes grid

Abbreviations: ID = identification, Nos = numbers, DPD = Dental Practice Division, DRO = Dental Reference Officer, CG = clinical governance, PR = Peer Review, CPD = Continuing Professional Development, GDC = General Dental Council, PG = Postgraduate, PLP = Personal Learning Plan

Notes:

Section I consisted of identity information that tied portfolio to the individual.

Scale used by DROs to assess practitioners work:

1 – Satisfactory, 2 – Adequate, with minor deficiencies, 3 – Inadequate, with more serious deficiencies that should trigger further assessments, 4 – So inadequate that other agencies should be informed – local Primary Care Trust, Police, GDC

Full size image

Practitioners were recruited to participate in the pilot scheme by inviting 20 vocational trainers in South East Scotland to participate. Ten dentists volunteered and all completed the scheme. Each dentist was provided with a ring binder portfolio, sub-divided by headed index cards with pro-forma sheets and plastic document pouches in each section to make recording of information and reflection straightforward. Portfolios were supplied at a half day induction course. This course introduced the concept of revalidation, showed how the scheme would fit with clinical governance requirements13 and explained how participation would be facilitated. A questionnaire was sent out after the induction course to allow post course evaluation (Fig. 2), which asked participants to rate how well the course had met its objectives, using a Likert scale.

Figure 2
figure 2

The questionnaire used to assess the induction course given to participants

Full size image

The scheme was run over the second six months of 2002. To identify any problems or difficulties, the scheme organiser made two pastoral contacts with all participants in the middle of the second and fourth months, but none were volunteered. Minor issues that arose during the operation of the scheme were dealt with directly — all were successfully resolved.

The eventuality of a participant not achieving a recommendation that revalidation was appropriate was addressed by creation of a remedial process. The scheme organiser proposed a draft Personal Development Plan (PDP) to address the deficiencies in their evidence to support revalidation. The dentist then submitted a codicil to the portfolio detailing actions taken, with documentary evidence where appropriate. Finally the assessors checked that the codicil met the requirements laid out in the PDP and gave a decision of 'still unable to recommend revalidation' or 'able to recommend revalidation after considering supplementary evidence submission'.

Assessment of the portfolios was carried out by a panel of three assessors. The panel was selected with a view to ensuring that the members were fully conversant with current trends and requirements in CPD for general dental practitioners, were aware of clinical governance requirements in general dental practice and that there was 'clinical governance informed' non-dental input. The assessors were given the following remit:

'To assess whether or not revalidation could be recommended and this decision was to be based on an assessment of whether the dentist had demonstrated (via documentary evidence) that they remain up to date in their practice of dentistry. Also, that they remain “fit to practise” according to the General Medical Council (GMC) definition of the term. The GMC definition is: “Fitness to practise — The level of performance, conduct, and health necessary in order to maintain registration”.'9

A valid assessment is one that measures what it claims to measure and what is important to measure.14 Reliable assessments are ones where the same marker reaches the same conclusions when presented with similar evidence.14 An assessment tool was devised in the form of a grid. National clinical audit criteria were used as a base for setting standards to assess the quality of audit activity.15 Where appropriate, supporting competencies derived for the assessment of Vocational and General Professional Training16 were included. This tool was piloted by the one of the authors (YM), prior to sending it out to the assessors for calibration.

Before carrying out the assessments, the assessors were asked to participate in a validation and calibration exercise. They were sent a portfolio completed with real data and deliberate flaws. This had three purposes: to familiarise them with the use of the assessment tool, its layout, and also to validate the assessment. It also facilitated a period of discussion and calibration before the 10 pilot portfolios were assessed. In this way a consistent approach was assured. The assessors were supplied with a questionnaire to determine their views on the revalidation pilot scheme process and the assessment process. They were asked to return these a week after the assessment day, leaving them some time for reflection.

All 10 participants were provided with grid sheets and instructions for recording how much time was spent compiling the various sections of the portfolio. They were asked to use these to record how much of their time and how much delegated time was spent compiling and reflecting upon the evidence.

Results

Eight of the 10 participants returned the post-induction course evaluation questionnaires. The results are summarised in Table 1

Table 1 The results from the Likert scale ratings for each of the questions in the post induction course questionnaire
Full size table

Initially, eight of the 10 portfolios were found to have included sufficient evidence to allow the assessors to form the view that recommending revalidation would be justified. Two were found to have insufficient documentary evidence to support such a recommendation. Consequently the remedial process was undertaken for these practitioners. The deficiencies noted were minor and simple remedial plans were soon fulfilled, allowing a view recommending revalidation to be formed by the assessors. Thus all 10 participants ultimately provided sufficient evidence to satisfy the assessors.

The time spent compiling the portfolio by dentists in the pilot is shown in the histogram below (Fig. 3) as averaged data. The measures of average show that the time spent on tasks was evenly spread between dentists and staff. However, the very low median for delegated tasks reflects a very large range, with many dentists not delegating any tasks at all in certain categories (eg Dental Reference Officer — DRO-reports) and sharing tasks equally in other categories (eg Audit).

Figure 3
figure 3

Time spent producing portfolio

Full size image

The panel of assessors completed their calibration task of assessing a trial portfolio in a consistent fashion. Difficulties and minor differences of interpretation were resolved at the pre-assessment meeting.

Discussion

The method used to select the mode of evidencing fitness to practise was a consultative one, involving a steering group. Using the same group, essentially a Delphi method was employed to devise what should be included in the portfolio. Some representatives of the profession suggested what should be included, then some volunteers from the profession tried it out and reported their experiences.

A portfolio is a collection of evidence maintained and presented for a specific purpose.17 Guidance on what to include in the portfolio was widely sought.18,19,20,21 GDC guidance on professional conduct Maintaining standards22 was employed to devise criteria against which to assess the portfolio. The first section was designed to collect identity information on the dentist to whom the portfolio relates and an affirmation by the dentist that the evidence contained in the portfolio relates to them. Subsection two consisted of volunteered information in this pilot, but would need to be strengthened in any substantive scheme — Chief Dental Officer (CDO) references would need to be checked, and the same with disciplinary and tribunal appearances. Numbers of NHS patients can be cross-referenced against Dental Practice Division (DPD) annual reports, but not very easily. This information relates only to NHS practice, so consideration would need to be given to alternative data sources for predominantly and wholly private practitioners. This also applies to the sub-sections in the performance assessment section relating to DPD, DRO reports and practice inspections. The section on assessment of performance was structured to include two different types of input from patients: unsolicited feedback in the form of complaints23 and compliments, but also an assessment by the patients via a randomly distributed questionnaire.24 The use of patients' views in assessing what constitutes good dental practice has been shown to be significantly at variance with a dentist's perspective.25 A section on peer review was included as it is suggested as a principle method of advancing clinical governance in general medical practice26 and has been shown to be effective in changing dentists' practice.27,28,29 A section on audit was included as it is an essential component of any clinical governance system. It can also be based on the assessment of patient outcomes — a direct reassurance to patients that dentists are performing well.30

Structured reflection is a key component of educational processes17 and of the development of professional judgement.31 Reflective practice has also been favourably reported in medical journals.32 A subsection specifically designated for reflection upon one or more events that had led to change in the practice was incorporated to encourage dentists to start recording this type of process. It was called 'critical event analysis' in the portfolio, but might be more readily recognised as 'significant event analysis'33,34 in future. The CPD section was created as a discrete component, but was positioned between the performance assessment section and reflection on practice section to try and indicate a flow of processes and activities through performance assessment to CPD and on through reflection on practice to a focus on the Personal Learning Plan. Thus reflection upon activity may lead to audit (which can count as CPD), which in turn may stimulate research or team training or initiate a further cycle of personal learning. This shows the difficulty in recognising and rewarding CPD in practice: 'Learning in the context of practice is a continuous experience, best acknowledged through processes rather than events.'35 Learning in practice has been recognised as a core process promoting quality.36,37 Approximately one quarter of the profession hold other registrable degrees38 so it seemed that this would be appropriate to record in the portfolio as a marker of CPD. In order to try and follow the current move towards multidisciplinary, team based learning39 a sub-section on training as part of the team was created in the 'reflection on practice' section. The completion of a personal learning plan has an educational role in appraisal and revalidation40 and two formats were reviewed40,41 before adopting as a template the one that was demonstrated to link with appraisal and revalidation in the medical profession.40

Patterns of prescription and treatment provision were assessed in the subsection on Dental Practice Division reports, while standards of treatment provision and probity were assessed simultaneously in the Dental Reference Officer (DRO) subsection. The standards employed by DROs are the judgements of established and experienced practitioners (Limen referencing).

The number of dentists participating in this pilot was small, at 10. However, it is in the nature of pilots to be small — ideally a larger study should follow. This requires waiting until the GDC has made some decisions about the format, nature and requirements for revalidation. All the participants were volunteers and so the results may not be fully representative of the profession, which may include some or many dentists who are less well disposed to the concept of revalidation. Persuading such dentists to participate in a substantive study of revalidation prior to its becoming mandatory would be challenging.

The responses to the post induction course questionnaire show that the participants were successfully introduced to the concept of revalidation. The results also show that the relationship of revalidation to clinical governance was well explained. The participants understood how to use the portfolio. The critical incident/significant event analysis exercise was well received, even though this is a new concept for most general dental practitioners. The process of assessing the completed portfolios was the area least well understood, but overall the induction day succeeded in meeting its educational objectives.

The average amount of time spent by dentists producing their portfolios was nine hours 16 minutes and the average amount of time delegated to carry out tasks was eight hours 17 minutes. Two practitioners had included all the time that they had spent conducting clinical audit and CPD in that time category rather than the time spent reflecting upon them and changes in practice as a result. These timings were reduced to the mean value of the other five. The adjusted values are seven hours 58 minutes for dentists' time and delegated time to six hours nine minutes. Thus the anticipated time commitment for a portfolio system would not seem to be too onerous at approximately eight hours of dentist time supported by six hours of delegated time per year.

Conclusions

Elements of clinical governance can be used to support the mechanism of revalidation. Allowing for the caveats made regarding the small number and the volunteer status of those involved in the pilot, the dentists in this pilot scheme were able to assemble the required documents and carry out suitable reflection. The assessors were adequately prepared and calibrated prior to assessing the portfolios. The time allowed for assessing the portfolios appeared to be about right.

Other important aims of the pilot were to gauge the acceptability of the scheme to the profession both as subjects and assessors. The results of a qualitative survey of attitudes is presented in a subsequent paper.