Main

In the recent past, dental education has been structured to 'teach' students increments of prescribed subject matter and formally assess this using mainly written tests and clinical long and short cases which were largely subjective.

Continuous assessment and quantitative analyses of attainment such as 'points systems' were the main methods of assessment of clinical operative skills. The essence of competency based education is learning 'in context' which in dentistry means orientation of the learning environment towards the clinical environment in which the student will eventually practice as a professional. This recognises a basic principle in education that 'learning is a personal event that results from sustained and meaningful engagement with one's environment'.1 In effect this means that learning results from interactions with the environment and the learner constructs new knowledge based on previous experience.2 This is at the core of the 'competency based curriculum' which must specify learning outcomes in the context of clinical competence and describe the expected level of attainment at defined stages in the progress of the student or trainee. Therefore new methods of assessment, of knowledge, skills (particularly clinical operative skills) and attitudes need to be developed.

Assessment methods for diagnostic and treatment planning scenarios using techniques such as Objective Structured Clinical Examinations (OSCEs) are becoming established in undergraduate curricula. The main advantages of the OSCE are the objectivity, flexibility and the clinical nature of the assessment. They do however have certain limitations, which has prompted the search for a complementary system for clinical assessment. To complement the OSCE methods for assessment of invasive clinical procedures are currently being devised using a form of one to one assessment with direct observation. These Structured Clinical Operative Tests (SCOTs) have a few unique features including a validated checklist based on the standard operative procedure for a particular task and an emphasis on encouraging self-evaluation by the student. These are currently being used as a formative assessment tool,3 but have been devised so that after validation it would be possible to use them in summative assessment.

Limitations

Attempts to set up clinical scenarios using 'phantom heads' and acrylic models in an effort to simulate certain clinical situation are apparently unsuccessful.4 The lack of authenticity from an interpersonal skills and behaviour management viewpoint are major drawbacks. The obvious alternative would be to consider bringing patients into OSCEs or dental OSCEs in particular, but these also present certain problems. Certain history taking/physical examination /diagnostic procedures can be carried out satisfactorily and objectively on a consenting patient or a simulated patient.5

For the sake of objectivity it is recommended that the same subject is used throughout and the repetitive nature of doing even a totally non-invasive procedure can be a disadvantage. There is also an element of 'training' in many patient simulation scenarios, and on occasion the lack of authenticity even with a well trained 'patient' can lead to problems, especially when the unexpected happens.6 However, these patient based scenarios are successfully used in medical OSCEs and with carefully chosen scenarios.7,8

On the other hand, invasive clinical or irreversible operative or surgical procedures that form a large part of the dental undergraduate training are by their very nature precluded from the OSCE. Therefore, in order to ensure that the OSCE scenarios will work we need to define very carefully what competencies we want to test, and defining the 'Assessment Objective' in OSCEs is of crucial importance for validity.

Testing of intra-operative skills

Operative technique or manual dexterity is not the only aspect to be assessed in a clinical procedure. It is often quoted that for any given clinical skill 25 per cent is operative skill and 75 per cent is knowing what is the appropriate operation, technique and approach to use for any given procedure.9 Assessment of competence needs to take account of both.

'Competence' can be regarded as a core value for safe, effective clinical intervention. Furthermore it is considered to be demonstratable behaviour and can be measured using criteria for performance standards. The concept of competence implies the capabilities to determine when it is appropriate to carry out a task as well as to be able to successfully complete it. This will involve performance of broader, more generic tasks such as planning, clinical reasoning, contingency management and performance of these with awareness of the psychosocial context and ethical framework.10

The skills employed are not just the technical ability to carry out clinical tasks but also the ability to apply them to new situations. The OSCE is extremely well suited to the assessment of competence in diagnosis and treatment planning for dental scenarios. However, it has severe limitations in its scope for testing both operative and intra-operative skills. These can be addressed in a more authentic clinical situation using the SCOT.

Table 1 Table 1
Full size table

Lack of student centred assessment

In contemporary medical and dental curricula continuous assessment is an important component of the overall assessment. This can take a variety of forms, all of which have two main aims: (a) they should record achievement of competency in as objective a manner as possible and (b) they should encourage continuous self-evaluation.

With the latter, the students reflect on their clinical performance and in consultation with their supervisors they record their plans to improve their competence in that skill or procedure. It is considered most appropriate to carry this out immediately on completion of a task whilst the experience is still fresh in their minds. The main advantage of this system is the encouragement of deep reflective learning as opposed to the superficial factual learning which is characteristic of the more traditional curriculum. Such supervisor-validated self-assessment10 learning experiences are designed to progressively improve future clinical performance and are described as 'formative'.

Introduction of the SCOT

If we aim to confer competence in clinical skills, it is important to be able to assess them, and to do so objectively. This will be necessary to underpin the introduction of a competency based curriculum.11 We therefore have to meet the need for a system of measuring deep learning of clinical skills and attitudes as well as knowledge that is amenable to external scrutiny. The basic ground rules from which the concept of the SCOTs has developed were as follows:

  • It would be desirable to assess manual and intra-operative skills.

  • Student feedback indicates that the multi-station OSCE is not a suitable forum.

  • A more structured chairside approach would be more objective than existing methods of assessment of these clinical operative skills.

  • The Structured Clinical Operative Test (SCOT) is perceived as being a possible way of achieving this.

The SCOT is an assessment system using a validated checklist for routine day to day clinical tasks. These can be applied at any stage during an undergraduate or postgraduate clinical course.

In the SCOT the student or trainee will perform authentic clinical procedures on real patients, under supervision and will be assessed using pre-agreed criteria listed on a check list. Immediately afterwards, the performance will be discussed between examiner and student/trainee in a formative manner. It is thus part of the continuous assessment designed to complement the end of course OSCE examinations. The SCOTs which have been devised for use in the formative assessment of core clinical skills in the introduction to clinics Year 2 term4 of the undergraduate curriculum are as follows:

  1. 1

    History taking and patient examination

  2. 2

    Patient management (communication and relaxation skills)

  3. 3

    Chairside infection control

  4. 4

    Administration of infiltration anaesthesia

  5. 5

    Taking an impression

  6. 6

    Safe use of instruments and handpieces

As a simple example, a student might wish to be assessed in the taking of an impression; a member of staff would supervise the procedure and assess it according to a predetermined and agreed set of criteria such as:

  • Preparation and positioning of the patient

  • Try-in of the impression tray and modification with the appropriate material

  • Loading of tray using appropriate impression material

  • Impression technique and cross infection precautions

  • Patient management and comfort during procedure

  • Contingency management

  • Quality of the final product – symmetry, extensions etc.

  • Instructions to the laboratory

At a later stage in their training the SCOTs devised will examine skills commensurate with the students' experience and training, and in the 4th year of the course the procedures that are examined by this method are:

  1. 1

    Fissure sealant placementFootnote 1

  2. 2

    Tooth extraction

  3. 3

    Orthodontic appliance adjustment

  4. 4

    Oral hygiene instruction

  5. 5

    Administration of ID block

  6. 6

    Cavity preparation

The SCOT conforms to the important principle in clinical learning that assessment drives learning and that the best environment for clinical teaching and assessment is to keep it in context. Among the principals for effective learning outlined by Schmidt et al., (1987)12 are that: (a) prior knowledge needs to be activated by signals in the context in which the information is being studied and (b) the ability to activate certain knowledge in the long term memory and to make it available for use depends on the context in which the knowledge was originally gained. SCOTs therefore make the most of the educational opportunities presented in the clinical environment, with the emphasis on formative assessment.

Clearly this system is not entirely objective because of the patient variation, but is an authentic clinical scenario and therefore, for the skills outlined, is a much more valid method of assessment. Therefore objectivity can be maximized by careful choice of tasks examined and meticulous design of the checklist used for marking.

Practicalities in the implementation of the SCOT.

The content of any examination of clinical skills (such as the OSCE and SCOT) should be analysed according to skills rather than disciplines. A skills matrix was devised with four broad categories broken down into smaller components and enumerating the skills individually. In any particular examination sitting all skills are being tested in appropriate proportions.

The four broad categories listed below may be used to categorise clinical skills.

1Inter-personal skills

Includes history taking, communication skills and establishment of rapport.

2Examination/Diagnostic skills

Includes extraoral and intraoral examination, decisions regarding appropriate investigations and differential diagnosis.

3Treatment planning skills

Includes interpretation of results, definition of treatment objectives and treatment planning, including clinical decision making and negotiation with patients (taking social, behavioural and ethical factors into account).

4Operative skills

This would include manual skills, intra-operative management, contingency management and intra-operative decision making. Also included are aspects of practical clinical chairside tasks such as appliance design and prescription writing as well as aspects of behaviour management (for example with regard to preventive dentistry)

Implementation of the SCOT raises some conceptual and many practical issues, some of which we can anticipate and address in advance. The following questions attempt to deal with some of these:

1Are there problems with objectivity or reliability of the SCOT:

In the past it was generally felt that manual operative skills could only be assessed subjectively in either continuous assessment or long and short clinical cases in end of year examinations. It is generally agreed that objective assessment of clinical skills is desirable and the use of the check-list means that the SCOT, while not being entirely objective as the patient scenarios differ between students, is structured and as objective as it is possible to achieve. The intra- and inter-examiner reliability of the SCOTs are being assessed and the results of this will be the subject of a future paper.

2How does this fit in with existing methods of assessment?

The SCOT system is used for formative assessment in the present system. A criticism of continuous assessment of clinical tasks is that they tend to be assessed subjectively by staff, and a grade awarded often with little or no explanation, a system which can be perceived as unfair by students. The SCOT introduces a greater level of objectivity to the assessment of operative clinical tasks by the use of the pre-determined checklist criteria. There will be no record of pass or fail, the student will either be signed up as having completed the procedure competently or not. In the case of the latter, they will be appropriately counseled by the supervising member of staff and advised to repeat it on another occasion.

3What will the students gain/how will it improve training?

Learning is predominantly assessment driven, and assessment must be seen to be fair. As a result, it must be carried out as objectively as possible. The SCOT will facilitate this. Also the students are able to measure their ability against a standard on a continuous basis throughout the course thus increasing self-awareness and encouraging learning by reflecting on experience. It should also facilitate early identification of individuals who are not coping well in the clinical situation, and will facilitate appropriate remedial action.

4.Will this affect the multi-station OSCE?

The multi-station OSCE and the SCOT are seen as being complementary rather than alternative methods of assessment. The SCOTs are a means of continuous formative clinical assessment concerned with operative/surgical skills, while the multi-station OSCEs are more suited to assessing a range of interpretation, diagnosis, treatment planning and perhaps inter-personal skills. Together, these examinations make up an element of clinical skills testing which is complementary to other methods of assessment such as essay questions, short answers, MCQs etc. It is envisaged that the SCOTs would run throughout the undergraduate clinical course with the content and complexity of the tasks tailored accordingly.

5How would these SCOTs fit in with existing manpower on the clinics?

To ensure the maximum degree of objectivity there must be inter-examiner agreement on the checklist criteria which are used for assessment of the SCOTs, and all supervising staff must be familiar with this. Staff and student feedback underlines their usefulness, but it is not possible to allocate a member of staff to the supervision of one student carrying out a time consuming procedure in its entirety. In order to address this it is possible to use the SCOT as a method of examining 'microskills' i.e. the breakdown of a complete procedure such as a restorative procedure into smaller component parts, each of which may be assessed separately, and in a number of different clinical areas.

6Will SCOTs have an application beyond undergraduate dentistry?

While the concept of the SCOT is being developed in undergraduate dentistry, it has applications beyond this setting. Being a one-to-one assessment it might also find application in the assessment of competency in postgraduate dentistry.

A Vocational Training (VT) pilot project in which various kinds of assessment methodology were tested reported a very favourable response from both the trainers and trainees.13 This project also suggested prior discussion of the SCOT check-list for a given procedure by trainer and trainee to ensure a better 'ownership' by both parties and therefore engender a greater degree of enthusiasm and co-operation with their use.

The OSCE/SCOT twin track approach

The OSCE may be used as a tool for summative assessment of clinical skills in professional or 'degree' examinations, but like all summative assessments may also (and should) be used formatively. The SCOT is primarily used for formative assessment of operative clinical tasks, perhaps as part of continuous assessment in the clinics, but can also be used summatively. The OSCE/ SCOT twin track system introduces a degree of objectivity to clinical examination which can be used to assess a range of interpretation, diagnosis, treatment planning, inter-personal,14,15 patient management16 and even attitudinal skills.17 Together they make up an element of clinical skills testing which is complementary to other methods of assessment such as essay questions and structured short answer questions designed to test knowledge.

The perceived advantages of the twin track approach are: a) it is seen by the students as being both valid and objective, and therefore a fair method of assessment, b) it facilitates contextual learning which improves the retention of knowledge, c) clinical progress can be more closely monitored and weak students can be identified early, d) it is student-based, with each student being assessed at their own pace, and e) formative assessment is designed to improve self evaluation, while under staff supervision.

The OSCE/SCOT twin track approach therefore can be regarded as an effective method of staff-validated self-assessment used for the evaluation of clinical competence.