Introduction

This second paper in a series of three articles explores the challenge of measuring quality in primary dental care, while cognisant of the fact that we currently have no agreed definition of quality in dentistry, an essential requirement as described in the first paper.1 In this paper we discuss the rationale for measuring quality, the fundamentals of measurement that need to be considered when designing and testing a measuring instrument, and the utility of such a measure used in the context of primary dental care. Finally, we speculate on what a measure of quality in primary dental care would look like and the practicalities of using it routinely in dental practice.

Purpose of measuring quality

Quality is a key organising principle of the NHS.2,3,4 However, as we saw in the first paper in this series, quality means different things to different people in different contexts.1 Moreover, an agreed definition based on a sound conceptual framework is necessary to ensure there is a common currency for describing and measuring quality.

The reasons for measuring quality are obvious: if you want to improve quality you need to be able to first measure baseline levels, introduce some kind of intervention and apply the same measure after the intervention has been made to see if there has been a change. Measurement is also important for the longitudinal monitoring of quality, separate to the evaluation of specific, planned interventions. It is particularly important to measure the outcomes of initiatives that aim to improve quality, as these interventions will affect health outcomes for patients, which could be harmful as well as beneficial. Incentives for improving quality are also increasingly used as part of systems for remuneration of healthcare professionals;5 measures of quality are often termed key performance indicators (KPIs) when used in contracts.6 We therefore need appropriate measures of quality to ensure that patient charges and public sector payments are actually paying for quality and that healthcare professionals are being appropriately remunerated for improving quality.

Appropriate measures of quality are also needed to support internal assessment of quality improvement programmes within practices; for objective external assessment of the care and service provided, for example in accreditation schemes; for performance management, to determine if contractually agreed targets for quality improvement are met; or to inform patients about the performance of services to enable benchmark comparison and thus inform patient choice. Therefore measures of quality are needed at different levels; the individual patient (micro) level, the organisation or practice (meso) level and at the system (macro) level, for example, NHS England.

Context

It is clear that there are some specific issues that need to be considered when trying to measure quality in dentistry and that quality means different things to different groups. The main stakeholder groups will have different needs and requirements and interests in quality; for example:

  • Patients and the public – there is the idea that providing information to patients and the public to enable them to compare the quality of different services will not only inform choice but, by bringing market forces to bear, drive up quality across the system. This was the thinking behind producing and publishing annual quality accounts, an idea of the previous government.7,8 It is also clear that perceptions of what constitutes high quality care will differ between different groups within society who have different needs; for example, healthy adults who regularly attend general dental practice will have a different view of what quality entails compared to older adults with a severe disability living in a nursing home. This does not mean that we should accept inequity in the quality of care provided; that is an individual or group in society receiving a different standard of care to other individuals or groups with similar health care needs. Rather, there needs to be a recognition that some particular elements of care are valued more than others by different groups in society and some elements, such as an expectation of being treated with dignity and respect, are common to all

  • Clinicians – can assess the outcomes of the care they provide to their patients and reflect on the outcomes to inform the planning of their continuing professional development

  • Practice owners – can use performance and quality indicators to measure the efficiency of the organisational management and customer satisfaction with the service provided to manage and improve their business and the performance of their workforce

  • Commissioners – if quality is to be contractually incentivised, measures of quality are a prerequisite to monitor performance against contractual targets

  • Policy makers – require appropriate measures to assess if policy goals, for example improving access and health outcomes for patients, are being met and also to establish if investment in quality is producing the anticipated effect.

These examples illustrate the difference in emphasis various stakeholder groups may have when measuring quality. Consequently any measuring instrument must be able to measure quality in its totality but also be flexible enough to assess separate elements of the overall construct.

Fundamentals of measurement

To measure any attribute, an appropriate measuring instrument needs to be developed and applied correctly. To assess if a measuring instrument is appropriate for the purpose it is being used for it should be judged against a set of characteristics9 and preferably using a testing protocol.10 Characteristics of any measuring instrument include:

Conceptually grounded

The measure must be based on a clear definition of the attribute (in this case quality) that one wishes to measure. This conceptual understanding and its distillation into a definition are absent for dentistry, due to an underdeveloped academic literature.1 A conceptually derived measure, grounded in the academic literature and, where necessary, supplemented by a collection of new evidence, provides a sound academic foundation for measuring quality. This offers a shared understanding of how to measure quality and promotes acceptance and utilisation of measures based on solid academic foundations.

Valid

Any measure must be valid, that is to say the measure must do the job expected of it; to measure the attribute that the measurer wants to measure, or that the measure claims to measure. This can be particularly difficult if we are seeking to measure a non-physical attribute such as an emotional response to a stimulus, or an abstract, multi-faceted concept such as quality. In every case validity can't be assessed using a single statistical test but only through a body of research to demonstrate the relationship between the measure and the attribute under investigation.11 An example of a measure with poor validity would be using a clinician's assessment to measure the pain experienced by a patient during a procedure.12 Face validity is particularly important for measures of quality, which can be described as a common sense test, that is, does the measure make sense to a layperson? Validity is often used synonymously with the term 'accuracy'.

Acceptable

A quality measure should be ideally acceptable to both assessors and assessed. If the measure is aligned to, for example, the needs and values of health professionals and their day-to-day priorities, it is more likely they will engage with them.

Reliable

For a measure to be useful it has to perform consistently each time it is used, so if a measure is reliable it will give the same reading each time it is applied, assuming that there has been no change in the attribute being measured. An example of a measure with poor reliability in the context of quality would be assessing patient satisfaction with the dentist occasionally asking his or her patient how satisfied they were with the treatment they had received and sometimes asking the same question anonymously; different results are likely to be obtained. A measure can be reliable but have poor validity, that is it measures something very consistently, but it turns out not to be the thing you thought you were measuring. However, poor reliability adversely affects the overall validity of a measure. Reliability is often used synonymously with the term 'precision'.

Responsive

Responsiveness or sensitivity to change can be defined as the ability of a measure to detect change when it has occurred; this is in part determined by the type of scale inherent in the measure. Measurement scales can be:

  • Nominal – simply assigning variables into a descriptive category for example, gender

  • Ordinal – where a variable can be graded into a descriptive rank order for example, tooth shades

  • Interval – in which the scale is made up of equal units for example, temperature measured by the Celsius scale

  • Ratio – in which the scale is made up of equal units and has an absolute zero for example, age.

Ratio and interval measures have greater power to detect change than measures that use nominal or ordinal scales, as they usually enable finer assessment of a variable. Most measures of human behaviour use ordinal or interval scales.

Complete

For complex concepts such as quality, a single measure is insufficient to capture all of the different facets that make up the concept. There are different domains of quality13 and we must be certain that a measure can capture all of these domains so that important elements are not omitted. This is usually addressed in the conceptual grounding of a measure through undertaking an extensive literature review supplemented where necessary with the collection of new information to produce a taxonomy or classification of quality measures, thereby systematically defining the components of quality and ensuring no important elements are missing.

Pragmatic/feasible

Measures must not only have the above technical qualities, they must also be easy to use, particularly in a general dental practice setting. If they take too much time to apply, healthcare staff and/or patients will not want to complete them. Time in general practice is synonymous with costs14 and a measure that takes a long time to complete by clinical staff, for example a complex clinical risk assessment, may be too costly to use. Similarly, if expensive equipment is required to support measurement or if very extensive training on using the measuring instrument is necessary, these factors can make measurement prohibitively expensive. Of course, there are costs associated with measuring quality but these costs needs to be outweighed by the benefits it produces.

The outputs of an ideal measure of quality must be easily understandable by healthcare and receptionist staff, as well as patients with varying levels of literacy and numeracy skills. The data produced by the measure should also be easy to turn into information. Ideally, analysis and presentation of the data should be undertaken automatically, in a timely fashion to give meaningful feedback to patients, clinicians and health service managers in easily accessible formats.15 This is linked to the concept of utility, the people who are the target audience for the information produced by the measure must value and make use of the information; if not the exercise of measurement will be futile.

A measure that demonstrably meets all of these requirements will have credibility and is likely to be used and its outputs trusted and acted upon.

What would a measure of quality in primary dental care look like?

A successful measure must do all of the things mentioned above. However, it must also be based within a recognised framework for assessing health care such as Donabedian's highly cited and used triad of structure-process-outcome:16,17

  • Structure (the context in which care is delivered)

  • Processes (how care and services are delivered)

  • Outcomes (the endpoints of care and services).

It is clear that quality is multi-dimensional and it is impossible for a single measure to capture all of these different elements in one summative score. It is also undesirable to have a single quality score; if the aim of measurement is to improve quality it is important to have the ability to disaggregate quality into its constituent elements and look at performance in each area.13 A comprehensive measure of quality in primary dental care will therefore include multiple components. What components should be included? The way a comprehensive measure would usually be compiled would be to undertake a comprehensive literature review to find all of the measures of different aspects of quality that currently exists. So, for example Ireland et al.18 undertook a review of the literature and obtained a national consensus for a set of quality indicators over ten years ago but these indicators were limited to clinical outcomes and important concepts such as access, service organisation, equity and patient experience were not considered.

Drawing on the medical literature is helpful, as some things are generic to all clinical care. For example, the Darzi review2 identified three domains of quality: patient safety, clinical effectiveness and patient experience. Also the General Medical Practitioner's Quality Outcome Framework (GPQOF) has evolved since its first appearance in 20045 with strong academic input and support from National Institute for Health and Clinical Excellence (NICE).10 The 2012-2013 GPQOF is comprised of four main components of domains and each domain contains a series of indicators that measure achievement. Achievement against these indicators is rewarded in terms of points with a maximum 1,000 points, with the pounds per point for 2012/13 for England being £133.76.19 The domains are:

  • Clinical: consisting of some 90 indicators organised by 20 disease categories. Largely focused on chronic disease management for example, hypertension, coronary heart disease, diabetes but in the dental context could apply to caries and periodontal disease

  • Patient experience: this domain remains comparatively underdeveloped. It consists of a small number of indicators that capture data on access to, and length of, consultations and patient satisfaction. In 2013-14 it consists of an indicator on the length of patient consultations. This area is just as relevant to dentistry, perhaps even more so due to provision of surgical procedures, levying of patient charges and marketing of cosmetic interventions1

  • Public health domain: this domain was introduced to QOF in April 2013. It contains nine indicators collecting data on cardiovascular disease (primary prevention), blood pressure, obesity, smoking and additional services indicators on cervical screening, child health surveillance, maternity services and contraceptive services. This again could apply to dental practice if specialised services are provided, for example oral surgery services under an 'any qualified provider' arrangement

  • Organisational: this domain will be removed from the 2013-14 GPQOF but had contained over 40 indicators measuring performance in five areas of practice organisation. These include records and information; information for patients; education and training; practice management, medicines management and quality and productivity. Again, these indicators are of direct relevance to dentistry

  • Quality and productivity (QP) domain: This domain contains nine indicators on reviewing data on secondary care outpatient referrals, emergency admissions, accident and emergency attendances, external peer review with other contractors, developing and following care pathways. Once again, some of these indicators are of direct relevance to dentistry.

All of the indicators in the GPQOF can be mapped to structure, process and outcome. Clearly all of the domains are relevant directly to dental practice but evidently many of the individual indicators within the domains are not applicable to dentistry. The GPQOF has been developed with strong academic support to test the concepts, and the validity and reliability of many of the indicators.10 However, relevant indicators in the GPQOF cannot be adopted by dentistry without first understanding how they work in the dental context.

The domains identified by Darzi2 and the GPQOF5 may also fail the completeness test for dentistry. For example, access to services is a major issue for dentistry and is not mentioned by Darzi or included in the GPQOF except in relation to quality and productivity and to review data on accident and emergency attendances. Access is not usually an issue for dental patients with a long history of regular attendance, but it is an issue for many members of the public who wish to join the NHS list of a local dental practice, or for individuals who wish to attend the dentist only when they feel they need to. Therefore, to develop a measure of quality for dentistry in addition to looking at what has already been developed in dentistry and in other areas of healthcare engagement with stakeholders is required. The views of the public, patients, dentists, dental care professionals and practice staff, plus commissioners of services are needed to check for completeness and to prioritise the inclusion and exclusion of indicators in a measure of quality or a quality framework containing multiple domains. An example is provided in Table 1 of mapping four quality domains against Donabedian's framework of structure-process-outcome16,17 to identify possible quality indicators for the dental assessment. The differences in the relative importance of each of the indicators for the regularly attending, asymptomatic patient and for the patient in pain who is seeking care become obvious.

Table 1 Example of a typology of quality indicators to assess care for a patient attending for an asymptomatic routine check-up or for urgent care due to pain
Full size table

In the UK work to measure quality in dentistry has started with the Clinical Effectiveness and Outcomes Group20 established by the Department of Health in response to the Darzi review.2 This group produced indicators based on the Darzi dimensions of quality: patient safety, clinical effectiveness and patient experience, some of which are represented in the preliminary Dental Quality Outcomes Framework (DQOF).21 Before publication of the DQOF, local DQOFs were established by some Primary Care Trusts22 but without rigorous academic foundation or testing. Dentistry is in many ways in a similar position to general medical practice in 2004; the GPQOF that was introduced in 2004 without piloting or robust academic review was refined in 2006.23 Since 2009 the GPQOF has progressed through research to review and retire existing QOF indicators24 and pilot new ones.10

Therefore, in the UK a start has been made to measuring quality in dentistry, academic rigour is now required to refine and support the evolution of the DQOF to ensure that the tools to assess quality in dentistry are valid, reliable, responsive, complete and pragmatically applicable for dentistry.

Mechanics of data collection and analysis

Measures of quality should be pragmatically applicable and their collection, analysis and presentation should have minimal intrusion into the lives of patients and clinicians. This is especially important in activity-driven remuneration systems. Data capture of quality indicators should if possible be a 'by-product' of clinical management of patients and service management processes through suitably designed IT systems.15 Innovative approaches are needed particularly in the collection of patient experience data; postal questionnaires are expensive, suffer from poor response rates and are probably outmoded in today's world. Other more current means of communicating with the public such as web-based surveys, SMS text messaging and email are likely to be more acceptable to patients in the future and may be cheaper to administer.

The process of measurement is not complete at data capture; the data needs to be turned into information through the processes of analysis, presentation and interpretation. Ideally analysis and presentation should be done automatically by IT systems producing information to predetermined templates or web-based enquiry systems, analogous to the e-reporting system developed by the Business Services Authority.25 Interpretation of data requires training and support; most data will be descriptive in nature and hypothesis testing and statistical tests will be of little interest to the majority of patients and clinicians. However, training and support are required to ensure that interpretation is valid and that appropriate decisions are made based on the data produced by the measuring system.

Careful thought is required on a number of issues, not least the unit of analysis. Measurement and analysis is possible and required (as we will see in the third paper of this series)26 at different levels: individual patient-level, practice-level and population-level. The latter is based on some pre-determined geographical unit for example, local authority boundary, local area team boundary and national level. There is a danger that data produced at an individual patient level could be simply and erroneously aggregated to produce practice level and population level data giving spurious results. Data at each level cannot be viewed as 'Russian dolls' fitting neatly one inside the other. Data at the level of the individual patient cannot simply be combined to produce values for practices and geographical units, this is because data on quality will have to be collected longitudinally and numerators and denominators will fluctuate over time. If data are to be aggregated it requires careful planning and analysis, which illustrates the importance of having pre-determined aims and objectives that clearly set out the reasons why the data is being collected and what it will be used for.

Conclusion

This paper sets out some of the requirements for measuring quality but also illustrates the challenges that need to be addressed if we are to measure quality in a valid, reliable and realistic way. Dentistry has a significant amount of work to do to ensure it meets this challenge and it needs to be approached with the same rigour that is being applied to the GPQOF. Once we have a sound conceptual basis for quality, and precise and accurate measures we are then in a position to systematically improve quality, something that will be discussed in the final paper of this series.