Main

The programmes were evaluated by two comprehensive questionnaires sent to all participating practitioners, the first during the development of the learning plans, and the second after completion of the one-year programme. This paper, part 2, presents the findings of the evaluation and assesses the effectiveness of this mode of delivery of continuing professional development. In a separate study, a number of delegates from both schemes were invited to take part in structured interviews with a social scientist, to evaluate their experiences from a qualitative perspective. These results are to be published as an independent paper. The results presented here were obtained from anonymous data, and from a quantitative evaluation. Whilst it is accepted that this evaluation cannot be viewed as scientifically accurate results, the findings must be of extreme interest to educationalists and those responsible for the provision of continuing professional development.

The methods used to collect the data necessary to evaluate the PLP pilot programme were described in part 1 of this paper. In an attempt to ensure open and frank evaluation of the programme, educational assessment was carried out by a research group (Working Minds Research, Edinburgh), and the responses remain anonymous to the PLP facilitators. The first paper considered the results of a brief pre-pilot questionnaire sent to all participants. This enquired about the reasons for getting involved in the project, present postgraduate activity and factors which may affect attendance at courses. It also included an 'attitudes to work' survey, adapted by Firth-Cozens.2

Shortly after completion of the programme the research group sent a second questionnaire to all participants. This document was more comprehensive and was used to evaluate the success of the programme by quantitatively analysing responses, and by seeking personal comments. The second questionnaire included a repeat of the 'attitudes to work' survey.

Although 80 dentists had originally expressed an interest in the PLP project, three dentists in the rural project declined to take part, and four in the urban project withdrew shortly after the start. Thus there were 73 participants in total. Three of these had either moved away or did not respond to the post-pilot questionnaire after two reminders. The evaluation of the concluding part of the pilot are therefore based on 70 completed questionnaires, 43 from the urban project (33 men and 10 women), and 27 from the rural project (21 men and 6 women), representing a 95.9% response.

Programme evaluation

1 Educational

Course attendance

The figures for course attendance are taken directly from the attendance registers and not from the participants responses. A table was produced listing all participants and comparing their previous postgraduate activity with their attendance at PLP sessions and additional 'routine' section 63 courses during the year of the investigation. These records were obtained from the national database, and do not include attendance at non-section 63 approved courses, or other educational activities. This data is summarised in Table 1.

Table 1 Average attendances at section 63 postgraduate courses
Full size table

Satisfaction of educational needs.

At the beginning of the PLP project, participants were asked to identify gaps from a list of 11 items. At the end of the scheme, they were reminded of the gaps they had identified and asked to indicate how well the gap had been filled by the PLP programme using a 5-point scale from 'not at all' to 'completely'.

No gap had been completely filled. Specialist clinical skills, which had been identified by most dentists in both areas (38 in the urban project and 20 in the rural project) were filled 'quite a bit' or 'a great deal' for 45% in the urban project and 70% in the rural project. The gap in specialist in-depth dental knowledge was filled well in the rural project (62%). In the urban project the gaps which were filled best were general clinical skills and acute care of dental patients.

In the urban project, the gap in research experience, which had been identified by 35 dentists, was not at all filled for 60%. In the rural project, the least satisfactorily filled were computing (not at all for 68%), financial management (67%) and research experience (65%). Dentists commented on the fact that a proposed computing course had not been organised in the rural area due to lack of suitable resources.

Home study on the practice of dentistry

Respondents were asked how much time they had spent each week over the PLP period reading about the practice of dentistry (disease, clinical skills etc). Nineteen percent in the urban project and 37% in the rural project had spent three or more hours. In the urban project, the percentage of respondents spending three or more hours per week rose from 10% prior to the project to 19% during the project. In the rural project the corresponding figures rose from 13% to 37%. The difference in the rural project between the two occasions reached statistical significance (Wilcoxon paired samples test), p = 0.008.

Although in the initial survey participants were asked only about their reading, in the second survey a question was asked regarding CAL programmes and educational videos. Over 85% of the respondents indicated that they spent less than one hour per week using these educational aids. Table 2 shows the results as a percentage of respondents.

Table 2 Educational activities outside the PLP programme
Full size table

2 Barriers to attendance

Participants were offered a list of seven possible reasons that may have prevented their attendance at postgraduate courses during the PLP programme. They were asked to score these on a 5-point scale from 'strongly disagree' to 'strongly agree'. The results are shown in Fig 1. Potential loss of income was ranked highest in both regions.

Figure 1
figure 1

Reported reasons for not attending postgraduate courses.

Full size image

Of least importance was the location of the training session. This was the case for both regions. However, when the group was divided into those who lived near the teaching centre (N = 34) and those who lived more remotely (N = 36), there was a statistically significant difference for the issue of the location (p = 0.035) with location being more of a barrier to people remote to the centre.

The dentists were invited to add their own reasons for not attending courses. Most common were personal circumstances, such as family commitments, illness and holidays, followed by practice commitments.

Influence of PGEA payments

Participants were specifically asked in a separate question whether the post graduate education allowance payments had influenced their willingness to attend. Significant difference was found between the two groups (p<0.005), as seen in Table 3.

Table 3 Percentages of dentists whose attendance at courses was encouraged by the PGEA payments.
Full size table

3 Critique of the programme

Questions were framed to ascertain participants' satisfaction with the personal learning programme firstly as a whole, and then from individual aspects; satisfaction with particular subjects; with particular learning styles; the difference made to clinical practice; and in comparison with a 'conventional' section 63 programme.

Participants were given 10 possible sources of satisfaction to rate on a 5-point scale from not satisfied to extremely satisfied. Both groups reported that the opportunities to interact with fellow professionals was the most satisfactory aspect of PLP, and the least satisfactory aspect was the study of computing. Several respondents remarked that their main dissatisfaction was that the programme had only run for one year. The results are presented in Table 4.

Table 4 Overall satisfaction with the personal learning plan initiative study
Full size table

Satisfaction with methods of training

Participants were given six options to rate on a 5-point scale from not satisfied to extremely satisfied: small group teaching, formal lectures, hands-on practice, peer study groups, 'away-days', practice meetings. Eighty two percent in the urban project and 70% in the rural project were extremely or very satisfied with small group teaching: 58% in the urban project and 55% in the rural project were extremely or very satisfied with hands-on practice. The results are presented in Fig. 2.

Figure 2: Satisfaction with various methods of training.
figure 2

(Non-respondents are not plotted in the graph)

Full size image

The difference made to clinical practice

Practitioners were asked to rate how the PLP programme had affected their actual clinical practice. The results are shown in Table 5.

Table 5 How would you rate the difference the programme has made to your clinical practice?
Full size table

Advantages of PLP over selecting courses from the 'section 63 programme'?

Ninety one percent in the urban project and 85% of respondents in the rural project considered PLP had advantages over section 63. The results are shown in Table 6.

Table 6 Do you feel that personal learning plans offer any advantage over selecting courses from the Section 63 programme?
Full size table

4 The attitudes to work questionnaire

The 'attitudes to work' questionnaire was completed at both stages of the project, and the results of the first questionnaire were reported in Part 1 of this paper. It is scored 0 = strongly disagree, 1 = mildly disagree, 2 = no opinion, 3 = mildly agree, 4 = strongly agree. The higher the mean, the greater the agreement with the statement

There were significant differences for the whole group between the two questionnaires in four of the items: 'I am developing new skills' (p = 0.040), 'I am being properly trained for my work' (p = 0.020), 'I am able to enjoy my personal life' (p =0.019) and 'I use my skills to the full in my job' (p =0.041). In the urban project, three items were statistically significant: 'I am being properly trained for my work' (p = 0.019), 'I am able to enjoy my personal life' (p =0.023) and 'I use my skills to the full in my job' (p = 0.041). For the rural project, there were no significant differences between the two phases of the course.

Respondents in the urban project expressed most agreement with the following five items, ranked in order

  • I am developing new skills.

  • I think most people in my position are suffering similar difficulties.

  • I am useful most of the time.

  • I am being properly trained for my work.

  • I can discuss work problems with other colleagues.

There was least agreement in the urban project with the statements

  • I regularly feel I am working beyond my capabilities.

  • I do not see myself continuing in dentistry.

  • I have sometimes been bullied

  • The responsibilities of my work are overwhelming.

  • I am worried about my future in dentistry.

In the rural project respondents expressed most agreement with the following five items ranked in order:

  • I am developing new skills.

  • I am under great pressure at work.

  • I have never experienced bias on account of race in opportunities at work.

  • I am useful most of the time.

  • I am being properly trained for my work.

They disagreed with the statements

  • I have sometimes been bullied.

  • I regularly feel I am working beyond my capabilities.

  • I do not see myself continuing in dentistry.

  • The responsibilities of my work are overwhelming.

  • I am worried about my future in dentistry.

Discussion

The premise underpinning the PLP programme was that if postgraduate dental education were tailored to the individual needs of the participant, it would result in a perceived improvement in patient care. It was also suggested that greater ownership of an educational programme, coupled with a sense of belonging to a study group, would stimulate and motivate practitioners to a greater extent. The results may be considered under a number of headings.

Did the programme lead to improved attendance?

Table 1 clearly answers this question. All the participants improved their attendance over their previous annual average, by an average factor of three or four times, although the most staggering was an increase by a factor of over 60. Thirty eight per cent of participants had attended additional section 63 or 'other approved' courses during their PLP year. Some of these may have been booked before the PLP programme was announced and the participants decided to attend despite the additional commitment of the project. If this was not the case, then it must be asked whether the project did indeed cater for all the needs of the participants when they themselves had identified their educational requirements. The majority, however, did not attend section 63 courses from the normal postgraduate calendar, even though they had been regular attenders in the past. The following are some of the comments received on the post-programme questionnaire.

'I tend to pick section 63 courses I like not the ones I need. PLP made me analyse my strengths and weaknesses — courses biased towards my weaknesses.'

'I tend to choose section 63 courses in subjects I enjoy and know something about. The PLP made me pick areas which I did not normally find interesting.'

'Helps you to focus and identify your weaknesses on a structured basis making course selection more targeted to my needs.'

However, the opportunity to select personal courses was not the only motivating factor. Further comments upon the advantages of PLP over section 63 programmes were:

'Location — tailor-made — suitable times – group therapy!'

'Giving more thought to structure of personal programme — convenience of locally held courses — some evening sessions — smaller groups.'

'Small group interaction and discussion resulted in change of practice. Evening-late afternoon format.'

'Structure and peer pressure made me do things I might otherwise not have bothered with.'

In addition to increased course attendance, participants also reported an increase in the amount of 'non-verifiable' educational activity, although this was mainly reading journals and textbooks. The use of CAL programmes and educational videos remains very low, in spite of their ready availability during the programme.

Overall, a picture emerges of a highly satisfied group of dentists, motivated through the personal learning programme to identify their learning needs, and to attend courses they might otherwise have ignored. This picture is further supported by the participants' personal comments.

Was the programme deemed successful by the participants?

The following comments are typical responses to the request for the most satisfying aspect of the programme.

'Small group teaching in a location easily accessible.'

'Practical teaching in paedodontics and one-to-one with the lecturers.'

'Interaction with other like-minded individuals.'

'Meeting on an informal basis with colleagues....realising that it is not all 'specialists' out there but general practitioners like myself.'

It is particularly interesting to note the importance given by the respondents under the most satisfying aspects of the programme to meeting, discussing and learning with colleagues. This would be especially relevant to those developing distance learning and CD-Rom format learning packages.3 Almost all responses focused on this small-group theme.

Some respondents specifically identified the opportunity to attend courses for which they had previously made unsuccessful applications centrally. However, the comments made in response to a request for the least satisfying aspect of the programme were more varied. Several dentists from the rural project focused on the two courses which had to be postponed, one on computing and one in preventive dentistry. Sadly both of these had been identified by a large number of the delegates, and their cancellation was regrettable but beyond the control of the facilitators, occurring too late to re-organise within the study period. Several of the rural delegates were also disappointed about the limited 'hands-on' facilities, and the lack of practical sessions.

As was reported in part 1, it was discovered during the planning that the number of permutations of educational requirements made it impossible to satisfy everyone during the programme. Thus various respondents commented that there were not enough small group sessions, that afternoon/evening sessions were unsuitable after a long day in the surgery, and that there were too many options to fit in to a busy schedule in one year. However, several delegates commented that the most unsatisfactory part of the programme was that it only lasted for one year!

Eighty nine percent of respondents considered the PLP to be better than a normal annual postgraduate calendar. No reasons for this were given by the 7% who did not, (4% did not express an opinion either way), but the following summarise the positive remarks.

'Tailoring the courses to my needs, rather than just picking one because it sounds good, is such an obvious advantage.'

'Section 63 is difficult to book and usually highly oversubscribed.... mostly placed on waiting lists.'

'No waiting lists for courses and it was taken out of your hands — someone else organised it all for you.'

'Being part of a programme led to more commitment on my part, and the facilitator made a difference.'

Barriers to attendance

In common with other workers as reported in part 1, it was found that loss of income was reported as the prime reason for not attending postgraduate courses. However, this finding relates to the post-programme questionnaire. In the rural, but not in the urban project, funding had been made available for the dentists to claim two additional sessions of post-graduate educational allowance. It is interesting that although significantly (p < 0.005) less than the urban project, loss of income was still the highest reported barrier to attendance on the PLP programme by the urban dentists as well.

Other relevant factors perhaps not always allowed for by educational providers were observed from the open comments:

'As a mother, it was quite a task trying to juggle work, school run, childminding, etc.'

'Other dentists in the practice were attending study sessions and it was not possible to leave the practice unmanned.'

'Not prepared to cancel patients already booked.'

Was there a perception by the practitioners of improved patient care?

Improvement in the quality of patient care must be the ultimate goal of post-graduate education, but how may this be quantified? Figure 2 shows the participants' own opinion, but that must be more subjective than objective. The 'attitudes to work' survey found only four significant changes following the programme, but three of these relate directly to the quality of patient care, and the fourth (I am able to enjoy my personal life) may show a more content dentist. The former three show that the dentists perceived that they were being well trained for their job, that they have developed new skills during the programme, and that following the programme they use their skills better than before. This could be either that new skills are being employed, or that an improved understanding has led to a reduction of previous inaccurate or wrong practices. It would appear from this that there is evidence of a perceived improvement in patient care. However, two personal observations are particularly relevant here.

'Realisation that the time and cost of many procedures are prohibitive to them being carried out under the NHS. Demoralisation from not being able to apply things learned on the programme in an NHS situation.'

'Good patient management/dentistry takes time. The more time and care you take, the more you lose.'

Lessons learned by the facilitators

The planning and implementation of the PLP pilot formed part of the continuing professional development of the facilitators themselves, and some valuable results were obtained in addition to those already reported. First, it was found that attempting to organise the individual learning plans into a cohesive group proved almost impossible. The initial aspirations of individually tailored PLPs were not found to be workable. It was felt that it may have been easier to produce personal learning plans, or practice development plans, for specific individuals rather than such a large group. This approach would also have accommodated the specialist practitioner, or those with limited practices, whose CPD requirements may be different to the general dentist.

It is certainly a pre-requisite of such an undertaking that the facilitators are highly committed and have sufficient time to devote to the project. However, it was interesting to learn that one of the main perceived benefits of the PLP pilot reported by participants related to their sense of ownership; not ownership of the PLP itself, but ownership of a new found enthusiasm for CPD. It became apparent that this extended to the group as an entity, with a distinct sense of supportive mutual development.

It was felt that the pilot may have benefited by the involvement of practice staff — the professionals complementary to dentistry. It was also felt that any available funding should be directed at the scheme overall, and not at encouraging participants to take part. Finally, whilst acknowledging the pilot nature of this programme, it was strongly felt that an extended period, or a continuous rolling programme of personal development, would have been more successful.

Conclusions

It is considered that the results support the original premises. If postgraduate dental education is tailored to the individual needs of the participant, it results in a perceived improvement in patient care. Greater ownership of an educational programme, coupled with a sense of belonging to a study group, stimulates and motivates practitioners.