Introduction

Clinical teaching has long been recognised as central to learning in dentistry.1 Dental students, faculty and administrators all acknowledge the importance of skilled, effective clinical teachers.2 Whilst dental educators have been concerned with assessing the quality of dental courses and educational material; unlike other clinical disciplines (such as medicine and nursing) there has been little consideration about the quality and effectiveness of the clinical teachers themselves.3,4 Dental faculties, like other clinical disciplines, require reliable documentation of teaching quality of individual staff for teaching quality control and accountability.5

Evaluating the effectiveness of dental teachers has implications on many fronts: The effectiveness of clinical teaching should be considered when promoting and/or offering tenure to faculty staff.6 In addition, students and other stakeholders involved in funding or consuming clinical training require standardised methods of evaluating the quality of clinical teachers to facilitate comparison of teaching quality within and between institutions.7 Individual clinical academic staff themselves also require feedback on their clinical teaching as a measure of identifying shortcomings that can be addressed in staff development training.8 Moreover, where investment in staff development occurs there is a need to demonstrate improvements in clinical teaching and returns from such investments objectively and reliably.

Thus, the aim of this project was to develop and test the psychometric properties (validity and reliability) of a questionnaire to evaluate clinical dental teachers (ECDT).

Methods

Developmental process

A review of the medical and dental literature was undertaken to identify appropriate conceptual frameworks for evaluating clinical teaching. One innovative and widely used framework within medical education has been the Stanford Faculty Development Program (SFDP) based on seven domains of clinical teaching: (i) learning climate, (ii) control of sessions, (iii) communication of goals, (iv) promoting understanding and retention, (v) evaluation, (vi) feedback, (vii) promotion of self-directed learning.9 The categories are based on educational and psychological theories of learning and empirical observation of clinical teaching.

Item selection

An item (question) pool was generated from a review of the literature, consultation with faculty staff (circulated at faculty board) and from focus group discussions with a 10% sample of the dental student population. Then item reduction took place, whereby a further 10% of the student population rated each item for clarity (yes-no) and relevance (low-moderate-high). Then following on, a factor analysis was conducted and the number of factors extracted was based on the theoretical framework (SFDP) and those domains with a cut off of eigenvalues greater than one. Individual items were selected where factor loading was greater than or equal to 0.40. These are accepted practices in designing scales or questionnaire design for selecting items (questions) and domains (sub-scales).10

Psychometric testing

The four years of dental students undergoing learning within the clinical environment were invited to participate in testing the questionnaire to evaluate clinical dental teachers (ECDT). The questionnaire is shown in Table 1. At the end of the academic year during resource sessions (review learning times), packages were distributed to students which contained the ECDT questionnaire and students were asked to self-complete the questionnaire for each principal clinical dental teacher responsible for the clinical modules (three clinics) they had undertaken that term. Completed ECDT questionnaires were then returned anonymously in the sealed envelope provided.

Table 1 Items selected based on factor loading and perceived relevance by students
Full size table

The data was entered into the statistical package SPSS for psychometric analysis. Where 10 or more ratings existed for an individual clinical dental teacher the validity and reliability of the ECDT questionnaire was explored. The criterion validity of the measure was assessed based on the hypothesis that a global rating of the clinical dental teacher on a five-point Likert scale (from 'very poor' to 'very good') would be associated with overall ECDT scores and individual domain or sub-scale scores. The construct validity of the measure was assessed based on the hypothesis that students would rate similarly individual clinical dental teachers. This was conducted by randomly allocating student's rating of individual clinical dental teachers to two groups and conducting a comparison analysis whereby overall and domain (sub-scale) scores were compared. In addition, evidence of bias of student ratings was explored by assessing mean directional difference in ECDT scores. The reliability of the ECDT measure was assessed by determining the internal consistency of the items overall within the scale and within individual domains employing Cronbach alpha statistics.

Results

Of the 74 items generated from a review of the literature, consultation with faculty staff and focus group discussion with students, only 25 items in the item reduction phase based on students' ratings of relevance (> 50% rated as moderate or highly relevant) and factor analysis (factor loading > or = 0.40), were chosen (Table 1).

In the psychometric testing phase 83% (158/191) of all the dental students who had undergone training in the clinics participated in the study. Overall 31 clinical dental teachers were evaluated with over 470 assessments collected. Analysis of data on 29 (94%) clinical dental teachers was conducted (two were excluded because less than 10 students assessments were available per staff member) and this represented 453 (96%) of all assessments collected. Range, mean and median ECDT scores (overall and domain) are presented in Table 2.

Table 2 Evaluation of clinical dental teachers [ECDT] scores
Full size table

ECDT scores and individual domain scores were associated with global rating of the clinical dental teachers (P < 0.001), Table 3. Those clinical teachers rated as 'very poor' or 'poor' had the lowest overall ECDT scores and lowest individual domain scores compared to those who were those who the students rated as 'good' or 'very good'.

Table 3 Criterion validity of clinical dental teachers [ECDT] scale
Full size table

In the comparison analysis of the ECDT scores between the two random groups of students, the mean directional difference of overall and domain scores were low and not significantly different from zero (P > 0.05), Table 4. The magnitude of the systematic difference (mean divided by standard deviation of directional difference) in overall and domain scores was less than 0.3, Table 4.

Table 4 Construct validity: agreement of the ECDT scores
Full size table

The mean correlation between items (Cronbach's alpha) for the ECDT questionnaire overall was 0.92, 0.92 for the 'learning climate' domain, 0.86 for 'control of session' domain, 0.92 for 'communication of goals' domain, 0.92 for the 'promoting and understanding and retention' domain, 0.92 for the 'evaluation' domain, 0.92 for the 'feedback' domain and 0.93 for the promoting self-directed learning domain. The inter-item correlation coefficient scores of the 25-item ECDT questionnaire overall socres overall ranged from 0.14 to 0.79, and no correlation was negative. The inter-item correlation ranges within each domain is presented in Table 5.

Table 5 Reliability (internal consistency) of the ECDT scale
Full size table

Discussion

Assessing abstract phenomena such as the effectiveness of clinical dental teaching is complex and requires appropriate conceptual and theoretical frameworks with which to guide the assessment process.11 The validity of the SFDP framework based on educational and psychological theories of clinical learning was extremely valuable in guiding the development process.9 In addition, the medical literature was valuable in generating an item pool which was further enhanced by focus groups discussion with clinical dental students and faculty staff. In the development process, although it is time consuming, it is imperative to include as wide a range of views as possible (consumers, stakeholders and providers of dental education) in ensuring that items tap relevant aspects of the attribute they are intended to measure. This approach also promotes the content validity of the scale.

For feasibility of use and to ensure that burden on students is kept to a minimum it is necessary to reduce the number of items (questions) within a scale (questionnaire), selecting a minimum number of items that are clear, relevant and explain most of the variance observed in a much larger item pool (factor analysis).10 Rating by a 10% random sample of dental students for clarity and relevance of items prompted face validity and ensured that items were unambiguous and reflected issues appropriate to clinical dental learning. Items (questions) were selected based on clarity and relevance (considered moderately to highly relevant by at least 50% of the students) and where the factor loading of the item was 0.4. This is accepted practice in questionnaire design.10

Although there is no 'gold standard' or criteria with which to evaluate the quality of a clinical dental teacher, a global rating of the clinical teacher from 'very poor to very good' was used to test the criterion validity of the ECDT scale, and was strongly associated with overall ECDT scores and individual domain scores demonstrating good criterion validity (Table 3). Construct validity was assessed based on the hypothesis that students' ratings of individual clinical dental teachers would be similar and when mean directional difference of ECDT scores between two random allocated groups of students was assessed the difference was minimal and not significantly different from zero. Furthermore, evidence of a systemic difference in ECDT scores between the student groups was minimal or small at less than 0.3, as the literature suggests.10 These findings support the construct validity of the measure.

Internal reliability assesses the internal consistency of a questionnaire or scale as it examines the extent to which a number of items assessing the same underlying concept are actually doing so, by measuring the mean correlation between items within domains (sub scales) and overall. In this respect, the data demonstrated strong internal reliability with Cronbach alpha values of > 0.80 in all domains. Various arbitrary levels of internal reliability are considered acceptable but most suggest internal consistency of above 0.69 to be appropriate.10

In conclusion, a valid and reliable measure to evaluate clinical dental teachers has been developed and under psychometric testing demonstrated acceptable construct and criterion validity, as well as strong internal reliability. An evaluation of the usefulness and performance of this instrument would be valuable to guide administrators, dental faculty and consumers of dental education.