Introduction

It has become apparent over the past 10 years that the traditional evaluation criteria in dental education, whereby assessment is carried out by means of number of tasks completed or their quality defined in objective terms, are probably insufficient to reliably distinguish the level of learning of emerging professionals.1 This has lead to a movement towards assessment by means of competencies which are based not only on the traditional evaluation of technical skills and theoretical knowledge, but also on the students' professional attitude.2,3,4,5,6,7,8

In the UK the General Dental Council states that the aim of the undergraduate curriculum is 'the production of a caring, knowledgeable, competent and skilful dentist who is able to accept professional responsibility for the effective and safe care of patients, who appreciates the need for continuing professional development, and who is able to utilise advances in relevant knowledge'.10

Study of the core competencies required for the practise of dentistry in the European Union, including directive 78/687/EEC,9 the work of the Association of Dental Educators in Europe and the Dental Education in Europe (DentEd) report8 confirm that the term clinical competence is applied to 'a combination of skill, attitude and knowledge, which provides the clinician with the capacity to undertake a specific clinical or administrative task'.

The 'Raamplan Artsopleiding 2001'11 (blueprint for the training of doctors) published in the Netherlands defines clinical competence as a combination of professional treatment and professional behaviour. In the recent report 'Consilium Abeundi',12 also published in the Netherlands, great emphasis is placed on the fact that professional behaviour should be an integral part of the education and examination of future doctors, dentists and veterinary surgeons. These reports use the term professional behaviour throughout because it is perceptible and assessable whilst attitude refers primarily to opinions and values. It would thus seem appropriate to use their terminology.

Professional behaviour is thus an integral part of the competencies required to practise dentistry, and whilst there are many methods of assessing technical skills and theoretical knowledge13,14 there is comparatively little published over concrete and acceptable methods of assessing professional behaviour.

Moreover van Luijk et al.15,16 state that it is important to have a system that can monitor the professional behaviour of students from the start of their studies. They point out that it is hardly fair to the students to tell them after several years of study that they are exhibiting unacceptable behaviour. The assessment of professional behaviour should therefore be carried out as early as possible in the study with appropriate feedback being given to allow students the time to develop this aspect of their education.

With the above in mind, it was decided that a mechanism was needed to assess the professional behaviour of first year dental students at the University of Groningen in the Netherlands, to enable appropriate feedback to be given to them. In order to place this assessment in context it was to be carried out during the preclinical course. A form for the assessment was to be used in conjunction with a specified system of giving feedback. A study was set up to assess both this form and the feedback procedure to see if they formed an acceptable means of carrying out the assessment of professional behaviour of first year dental students. The aim of the study was to see how both first year dental students and staff members would use the forms. Our initial expectations were that differences would exist, since it was to be expected that the students would develop professional behaviour during their study, whilst the staff were naturally much more experienced in this field. Secondly, we expected any initial differences to be reduced by the feedback given by the staff. Hopefully this study would also increase the awareness of the importance of development of professional behaviour by the students.

Our first objective was the development of an ordinal assessment scale to assess undergraduate professional behaviour. Using criteria for use in medical faculties introduced by van Luijk et al.,15 but adapting their criteria to make it more relevant to first year dental students, a form was developed with three main criteria (Fig. 1).

Figure 1
figure 1

English translation of the staff version of the form used to assess professional behaviour; a similar form was made for the students.

Full size image

Material and methods

The study was carried out during the first year preclinical course at the Department of Dentistry and Oral Hygiene, Faculty of Medical Sciences, University of Groningen, the Netherlands.

The students were asked to assess their own professional behaviour using the forms described above. This assessment was then compared with an assessment carried out by the staff and both the assessments were used to form the basis of an appointment during which the students could receive feedback from the staff with regard to their professional behaviour.

The assessment was carried out using the following criteria: Criterion 1: Carrying out assignments. Criterion 2: Behaviour in the pre-clinic. Criterion 3: Reflection upon own performance.

A five point scale was used on which to score the assessment for each of the criteria. A '1' on the scale denoted that the student's behaviour was not yet adequate for that criterion, whereas a '5' denoted that the student's behaviour was assessed as excellent.

The criteria were extended with a list of supplementary criteria, referred to as 'smileys'. These smileys were added to make the criteria more concrete and help define the feedback which was subsequently given. Criterion 1 was most concrete and there were more sub-criteria, criteria 2 and 3 were more abstract and there were fewer subcriteria (Fig. 1). Positive and negative smileys were used: a positive smiley meant that the student's performance was excellent and a negative smiley meant that the student's performance was a point for improvement. In this way it was possible to give not only negative but also positive feedback. The completion of the three criteria was compulsory, and the completion of the smileys was optional. In addition to the subcriteria room was left for any remarks, which were felt to be required, to enable the feedback to be as useful as possible.

The study included 56 first-year dental students and 8 staff members. Each lecturer coached a fixed group of 10-12 students during each term (some groups had two lecturers due to the constraints of the rosters, when this occurred the lecturers agreed on one assessment and thus used only one lecturer's version of the forms). The same member of staff coached the students over the first term (which was only seven sessions) and the second term (22 sessions). In the third term (again 22 sessions), another lecturer coached the same group of students. Before the start of the preclinical course, there were the usual meetings for calibration of staff members. The staff was made aware of the implementation of the assessment and could put their own views forward on the practical aspects of its implementation. Agreement was reached on the use of the forms and on the methods of giving feedback. This included a training session for the staff on this subject. For the first year students, a section was added to their manuals that specifically described the method of assessment and included copies of the forms, which were to be used for the assessment of the student's professional behaviour. The students were informed that this assessment would take place over the whole year, that a formal assessment using the forms would take place at the end of each term and that this would be followed by an individual appointment between the student and their group lecturer in order to discuss their findings and allow feedback over their behaviour to be given. Data from both student and lecturer versions of the forms were collected and processed to produce a database which enabled descriptive analyses (median, range) and non-parametric analyses of related samples (Wilcoxon) to be carried out. This was then repeated after each term to see if any variation remained after the feedback procedure had been carried out. Spearman rho tests were also carried out to assess any degree of correlation.

Results

Over the academic year each of the 56 students and eight staff members were issued with the appropriate number of forms. Thus 56 of each version were issued, at the end of each term 53 sets of forms were returned, a response rate of almost 95%. Unfortunately the forms were not always filled out very clearly; this resulted in between 45 and 47 matched forms (where both the student and the respective lecturer version were adequately filled in) being available for analysis. This is nevertheless a usable response rate of 80-85%. Verbal feedback relating to the perceived usefulness of the forms and the procedure, was also received from both the staff and the students. Although the staff sometimes stated that they found the assessment difficult they also stated that the use of the forms and the feedback procedure was very useful. The students also accepted the procedure. This was demonstrated by the fact that they not only attended the appointments held during preclinical time, but also made further appointments with the staff if both parties thought this would be useful.

Descriptive analysis of both groups gave a median of three with a range of two to five. Figure 2 shows the analysis of the differences between staff and students over the three criteria, over the three terms. The mean score from the staff was lower for all the three criteria.

Figure 2
figure 2

The mean, standard deviation, median and interquartile range (IQR) values of criteria 1-3 assessed by students and staff over the three terms.

Full size image

Wilcoxon Signed Ranks tests were carried out (Figure 3) and demonstrated that the differences in assessment after the first term were significant for criteria 2 (p=0.000) and 3 (p=0.000). These differences remained significantly different after the second term but by the third term the assessments under criteria 1 and 2 became insignificant (p=1.000, and p=0.808 respectively). Where the students' own assessment varied from that of the staff they tended to give themselves a more positive assessment, and this was especially marked under criteria 2 and 3.

Figure 3
figure 3

Wilcoxon Signed Ranks tests.

Full size image

Spearman rho tests (Figure 4) were also carried out to see if and where there was any correlation between the assessments by the students and those by the staff. The correlation coefficients between the assessment by the students and the staff were not significant. There were however some significant correlation coefficients found between the assessments of the three criteria by the students, which were more consistent than those seen in the assessments by the staff.

Figure 4
figure 4

Nonparametric correlations (Spearman's rho twotail).

Full size image

Descriptive analysis was also carried out with respect to the way the forms were used by the two groups. The frequency with which the subcriteria were used was markedly different (Fig. 5a). In this comparison it can be seen that the students gave themselves more positive smileys than the staff, that only a few negative smileys were scored, and that these were for time-management and assignments. The staff filled out the smileys section less often than the students. The students also gave themselves positive scores much more frequently than they gave themselves negative scores. This is particularly apparent with respect to criteria 2 and 3, where the vast majority of the students gave themselves positive smileys.

Figure 5a: The scores of the supplementary criteria 1-3.
figure 5

Here the results after the first term are shown for the students (left) and the staff (right). In each figure the score on the left is for negative smileys and that on the right is for positive smileys. The percentage of scores is shown for every supplementary criterion. When, for example all supplementary scores were filled out the score becomes 100%.

Full size image

After the second term (Fig. 5b) the staff made more use of the smileys. An interesting finding was that the scores for the smileys under the second criteria 'behaviour'given by the staff were balanced, ie as many positive as negative smileys were scored. This meant that the staff members gave both positive and negative feedback. The students were still very positive about themselves, and almost never used the negative smileys under criteria 2 and 3.

Figure 5b: The scores of the supplementary criteria.
figure 6

Here the results after the second term are shown.

Full size image

After the third term (Fig. 5c) the students were still using the smileys more often than the staff, although there was a great reduction in the tendency of the students to automatically fill in all the subcriteria. The assessment by staff members had changed dramatically particularly for the use of the smileys under the second criteria 'behaviour'. In contrast to the results seen after the second term when the staff used both positive and negative smileys, after the third term they only used negative smileys. The students, on the other hand were still very positive about themselves, particularly with reference to criteria 2 and 3. There was also a very large discrepancy between the students' description of themselves and the description by the staff under the specific subcriteria relating to attitude.

Figure 5c: The scores of the supplementary criteria.
figure 7

Here the results after the third term are shown.

Full size image

Discussion

The high response rate and the verbal feedback received from both staff and students support the ideal of acceptability of both the forms and the procedure. The system of assessment provides an opportunity for staff and students to discuss the meaning of professional behaviour, on an individual basis, during the feedback appointments. This, in turn, ensures that professional behaviour no longer remains an implicit part of the syllabus but becomes an explicit and integral part of the curriculum right from the start of the students' studies. The fact that this occurs in the preclinic, hopefully strengthens the link in the students' minds between the practise of dentistry and their professional behaviour. Thus the procedure not only states the importance of professional behaviour, it relates it directly to the learning of clinical skills and provides a planned time and place for students to ask questions and learn more about this issue.

The significant difference which existed between the scores of the students and the staff at the start of the study was reduced over two out of the three criteria, by the end of the year. Thus it would appear that there was less disagreement between the students and staff with respect to the students' professional behaviour after this procedure had been carried out. One of the major reasons for this may be that the students were given a chance to discuss the results and therefore discuss the meaning of professional behaviour with a member of staff at the end of each term. The feedback given by the staff has, presumably, played a role in increasing the students' awareness of the importance of behaving in a professional manner. Another important aspect is that the experience of both staff and students in the assessment of professional behaviour increases as the assessments are repeated. Thus there is a chance that the ability of both the staff and the students to assess professional behaviour changed and this produced the reduction in the differences between the two assessments. Finally there was perhaps a calibration effect due to the repeated assessment, perhaps the students were learning to assess the expectations of their staff members, and therefore adjusted their own assessments accordingly.

The attempt to show how the scores converged using a Spearman rho analysis shows that the students see a clear bond between the three criteria, frequently giving themselves the same score over all three criteria. This tendency is less often seen in the assessments by the staff. One explanation for this is that the staff have much wider experience and were using the forms in the knowledge that their main use was to enable them to help the students by providing concrete feedback. The staff therefore had to try and pinpoint the areas where each individual student could most benefit from advice, the students, on the other hand, had far less experience and it was therefore more difficult for them to be specific in their assessments. The significance of these results varied due to the variation in the number of matched pairs within the study which were low due to the limitation imposed by the number of staff and students available to take part in this study. There is no doubt that this needs further investigation.

The descriptive analysis produced a clear discrepancy between the use of the smileys by the staff and their use by the students. There are numerous reasons why this may have occurred; firstly the instructions were that this section was optional — a fact which the staff took on board to a greater extent than the students. Secondly each student had only themselves to assess whereas the staff had a group of 12 students and thus 12 assessments to make, as the staff were not given extra time to do this maybe the extra workload on the staff played a role in this difference. Thirdly, this difference was greatest during the first term, which was a very short term (there were only seven sessions in the preclinic), and it is possible that the staff did not have enough time to get to know their students well enough to make full use of the smileys. The fact that there was an increase in the use of the smileys by the staff in the second term seems to support this idea that the staff needed more sessions to get to know the students well enough to make full use of the forms. Finally the staff may have found the subcriteria inadequate and preferred to make their own remarks instead of using standard criteria. Further analysis of the forms including an analysis of the remarks made by both students and staff would be needed to clarify this point.

There was not only a difference in the frequency of the use of the smileys between the staff and the students, but also a difference in the way they were used; on the whole the students gave themselves more positive smileys than the staff. This tendency of the students to use the positive smileys was very marked under criteria 2 and 3, and it would appear that the more abstract the criteria the more positive the students scored themselves. It would seem that, where very little concrete evidence existed to support an alternative assessment, the students presumed that they were entitled to an excellent score. This pattern continued over all three terms, the largest discrepancy being seen for the subcriteria relating to the student's attitude to others (criterion 2) in the results after the third term. This may well be because this criterion covers both the student's attitude to their fellow students and to the staff. It could reasonably be argued that the students were assessing their attitude towards one (probably towards the staff) whilst the staff were referring to the other ie the student's attitude towards their fellow students. Unfortunately there were not enough remarks made on the forms to clarify this point, and it would be perhaps advisable in the future to make the students' attitude towards staff members and their attitude towards their fellow students two separate subcriteria.

The fact that the forms can be used differently (as can be seen from the use of the smileys) and yet produce the same overall assessment (as can be seen by the analysis of the main criteria) has a number of advantages when the forms are used to give feedback to the students. The differences provide a starting point for discussion whilst, at the same time, the similarity provides a basis for agreement and therefore helps the feedback to be both specific and to be given in a mutually acceptable context. Thus the discussions are most often perceived as helpful rather than critical. It is, therefore, this flexibility which increases their usefulness as a basis for giving feedback. This procedure does, however, takes up a reasonable amount of time and has a finite cost in terms of effort.

Additionally, there can be no doubt that the use of the forms produced a wider assessment of the individual students than the traditional assessment of their technical work and knowledge could have done. Thus the procedure certainly meets the criteria laid down in Raamplan Artsopleiding 2001,11 and goes some way towards meeting the wider assessment requested by DentEd,8 and inferred by the EU directive9 and GDC10 aims for the undergraduate curriculum. For our Dutch students it is also important that the procedure meets the criteria laid down in the report Consillium Abeundi.12 However our experience leads us to believe that the forms when combined with the system used in the study to give feedback to the students, are adaptable enough to be used more widely. There remains, however, a need for further assessment of the subcriteria and the way that the (sub)criteria are used, along with an analysis of the comments made by both the staff and the students on the open sections of the form in order to improve their specificity. Furthermore research needs to be done on the validity and reliability of the forms if they are to be used to produce a measurable outcome.

In summary, the following advantages and disadvantages were seen during the study. Firstly the filling in of the forms and the discussion and feedback procedure cost time and effort. The system does however have the advantage of being flexible, acceptable and does produce the wider assessment of each individual student that in turn has, hopefully, broader benefits for their education.

Conclusion

The study indicated that whilst there was an initial difference in assessment of professional behaviour between staff and first year dental students, this difference can be reduced by the forms and procedure used. It therefore formed a useful teaching aid. The main advantages of the forms are that they can be used to give feedback to the students and that they encourage the open discussion of professional behaviour between the staff and the students. This gave the staff the opportunity to discuss professional behaviour with the students and undoubtedly increased the awareness by the students of the importance of professional behaviour. The very high participation rate and the favourable feedback received from both staff members and students confirm that this is a very acceptable means of assessment of first year dental students' professional behaviour.