There are growing uses of health care information to improve the quality of delivered care and reduce health care expenditures in the United States. One such method is the public reporting of health care outcomes. The underlying theory for public reporting is that the consumer—for neonatology, a pregnant women or her physician—will choose to deliver at facilities with higher quality. Also, poorly performing facilities will voluntarily make improvements in their care so that they do not appear ‘at the bottom of the list’. Public reporting differs from benchmarking in that (1) the outcomes are disclosed to the public, and (2) reported outcomes may not be compared with other facilities, but rather to an expected rate of an outcome from an entire population that may have received care in settings different from the specific hospital to be measured.

As a result, it is critically important that a correct, valid measure of neonatal care is used in the public reporting process. In this case, neonatology lags significantly behind other specialties such as cardiothoracic surgery, which in some states has reported risk-adjusted mortality rates since the mid 1990s, and general medical hospitals, such as mortality rate for acute myocardial infarction and hospital infection rates. Papers such as the one by Kowalkowski, et al.1 are one critical step toward building this evidence base in neonatal medicine.

Quality measures can take one of three general forms based on a Donabedian model of quality health care.2 First, quality measures may be measured structures of care, such as the level of neonatal care as defined by the American Academy of Pediatrics. Second, process metrics can be a quality measure. A process of care is something ordered or performed by a health care provider, such as the administration of antenatal corticosteroids to women at risk for preterm delivery or performing timely ophthalmological examinations for retinopathy of prematurity. Finally, quality measures may be an outcome of care, such as mortality rates or the rate of specific complications such as health care infections or bronchopulmonary dysplasia. The Donabedian model would suggest that structures of care improve quality through changes in processes of care, which in turn result in improved outcomes of the patient. Thus, in theory, quality measures can take any one of these forms.

Quality measures may also differ in how well they measure the underlying, unmeasurable ‘quality of care’ delivered by a hospital. We can imagine a specific quality measure as a flashlight shining onto an unmeasurable black box that represents the quality of care at an institution. The stronger the light, the more accurate the measure is. The broader the beam, the more that changes in the measure may affect other measures. The ideal measure for neonatal public reporting would be bright and broad.

Data from the paper by Kowalkowski demonstrate some of the issues in defining and validating a quality measure for public reporting. Most clinicians prefer to be measured by what they do, or their process of care measures. Agreement on what processes of care should be reported, though, remains difficult. For example, the paper by Kowalkowski demonstrated good agreement between the clinician panel and the expert panel on the nine measures to include in the Baby-MONITOR metric. Only two of these measures, were process of care measures, and one of these process measures, the receive of antenatal corticosteroids, is a measure that typically measures the care provided by the obstetrician, not the neonatologist who receives the infant after delivery. Also, other process measures, such as the provision of surfactant within 2 h of delivery and the hypothermia at admission, were among the measures most likely to be rated by clinicians having ‘much too high’ or ‘slightly too high’ emphasis by the expert panel. Process measures suffer from difficult collection, usually requiring specific forms or surveys to collect the responses. Finally, it has been difficult to conclusively validate process of care measures, as seen in a prior randomized study of a package of process of care measures to improve the rates of bronchopulmonary dysplasia.3

For these reasons, most publicly reported measures of care are outcome measures. Outcomes measures are relatively easy to collect. These measures, though, are not without their problems. Outcome measures must be risk adjusted, or account for differences in hospital casemix that may affect the rate of a specific outcome. Although several ‘risk-adjustment models’ exist, few explore additional factors beyond birth weight and gestational age that alter the baseline risk of complications or death in a premature infant.4, 5 How these models are analyzed may also affect both the interpretation and transparency of the results, as seen by the concerns of the clinician panel in the Kowalkowski paper. Finally, if a risk-adjustment model is published, clinicians may alter or change their coding of these factors to make their casemix look sicker,6 or possibly change what patients they are willing to care for.7 Most risk adjustment models now have strict definitions to minimize this issue. Even with these concerns seven of the nine actors chosen for the Baby-MONITOR would be reasonably classified as an outcome measure.

Where does this leave neonatal medicine? There are still many unknowns. Although there are a large number of potential measures, there are no validation studies. Also, we do not know how changes in one measure will affect the overall ‘quality of care’ delivered by a neonatal intensive care unit. It is not surprising, then, that we have no information on whether publicly reported data even affects the choice of delivery hospitals by consumers at all, or whether public reporting results in improved quality of neonatal care. However, the current evidence base relies on data from pediatric or adult studies, which preferentially centers on mortality as the principal measure of care. We need more studies to identify the optimal measures to improve the overall care received by children. The alternative is that measures will be dictated or chosen by either state agencies or insurers that may result in worse outcomes of care.