Society for Pediatric Research Presidential Address 1996: Star Wars Empiricism-Will The Empire (SPR) Strike Back?

We are in an age of great technologic innovation that allows for a better understanding of the pathogenesis of disease, enhanced diagnostic capabilities, and improved treatment of patients. However, in many circumstances this technology is used in ways not documented to be medically effective. For example, high frequency ventilators are now often used in the neonatal and pediatric intensive care setting. These machines are substantially more costly then conventional ventilators, and their relative effectiveness remains a subject of considerable controversy. FDA approval for a particular technology is similar to that occurring with a drug. Once a technology is approved for one specific indication, there is little stopping its use in other circumstances. Thus, those intensivists whose personal experience suggests that high frequency ventilators are helpful are free to use them for any patient, even when this technology has not been formally studied in that particular setting.

Many nontechnologic things done in medicine are also of questionable efficacy. Indeed, the great majority of current medical practices are not based on data from well controlled studies. This frequent mismatch between factual knowledge base and the practice of medicine leads me to the main topic of today's talk, i.e. empiricism.

Empiricism greatly influences the way we deliver care. Modes of care and treatment of patients are often based on outside pressures, including parental and legal considerations rather than efficacy and cost-effective issues. For example, as the pressure to contain medical costs has become more predominant, substantial controversy has arisen around the insurance company and managed care driven practice of routine discharge of newborns from the hospital within 24 h after birth. Although some managed care organizations have put into place careful follow-up systems, for many newborns this type of close monitoring is not available. Currently, insufficient data exist on the safety and cost-effectiveness of this practice to justify its implementation throughout the United States.

The issues that I would like to consider today concern the effect of empiricism on the health of our children and what role do we in the pediatric research community have in combating empiricism.

Let me first state that in many ways it is harder to do a good clinical project than a basic research study. Although experiments done at the bench can be revised on a daily basis, this is much more problematic with clinical protocols. Even greater forethought and planning must accompany the clinical study, and still the unforeseen often comes back to plague you.

The days of ever-expanding amounts of government money for health care and research are over. Many feel this situation has reached the crisis stage. However, it is my belief that we have an opportunity rather than a crisis. Research itself provides the primary tool that can be used to overcome the constraints of a constricting budget. Research must be used to not only manage or cure disease, but also to decide how to most effectively spend health care dollars. If we have the foresight to put most of these cost savings back into additional research endeavors, two important but seemingly opposing goals can be achieved, i.e. better health for our children at a lower cost.

In pediatrics there are some spectacular illustrations of how well controlled multicenter trials can improve the health of our children in a cost-effective manner. In my own specialty of infectious diseases, a recent National Institutes of Health-sponsored study examined whether zidovudine, a medicine used to treat patients with AIDS, could be given to HIV-infected pregnant woman to prevent them from passing the virus on to their infants⁽¹⁾. Treatment of these mothers with zidovudine reduced the rate of HIV infection in infants by two-thirds, from 25% to 8%. This study showed for the first time that a medicine could be used to prevent HIV infection. Obviously, the ability to prevent rather than treat HIV infection allows for substantial cost savings. If the use of zidovudine in pregnancy was adopted universally, 650-1300 infants would avoid AIDS each year, saving over$85,000,000 to $170,000,000 annually.

Senator Hatfield, when introducing a new piece of legislation called the National Fund for Health Research Act, made the following point: “As we struggle in the coming months to achieve a balanced budget, we must embrace policies that enable us to make the most of our scarce federal dollars. Federal funding for medical research should be a top priority because, without new knowledge to develop new strategies to prevent disease, new treatment to delay the progression of disease, and new interventions to cure disease, health care costs will continue to spiral out of control. Disease drives the cost of health care”⁽²⁾.

I believe that the tentacles of research should be extended even further. Research should be part of almost every aspect of medicine. This includes not only the care of patients, but the administration of health care and how we train health care professionals. The Society for Pediatric Research and its individual members must lead the fight to make certain that research replaces empiricism.

PRINCIPLES OF RESEARCH

I would next like to discuss how empiricism becomes part of medical care. Some of my points will be highlighted using examples of medical practices within the domain of infectious diseases and neonatology. I apologize in advance to my colleagues in these two areas for the appearance of singling them out. No specialty is immune to empiricism.

J. B. McKinaly aptly described how empiricism creeps into medicine in a paper he wrote over a decade ago⁽³⁾. Five of the seven stages he describes in this process are as follows. First, a promising case report is published. Next, an uncontrolled study suggests benefit. Thereafter, the modality becomes the standard of care. Later, a controlled study is done, and no efficacy is noted. Finally, the modality slowly falls into disuse.

One example of creeping empiricism is the use of steroids and other adjunctive therapies for treatment of sepsis. The effectiveness of steroids as adjunctive therapy in the treatment of Gram-negative bacteria-induced shock had been debated for decades. Animal studies had shown protection against the deleterious effects of bacterial endotoxin when steroids were given in large doses. However, the amount of endotoxin used to induce shock in animals was much greater than that causing shock in humans. Furthermore, the animals were physiologically normal before induction of shock. Several small controlled human studies had not shown any benefit for steroids in the treatment of severe sepsis. However, proponents of steroid use argued that the patients in these studies were given steroids at doses that were too low and at a time point too late into their illness. Based on these animal data and case reports, some authorities advocated the use of industrial strength doses of steroids for severe sepsis. During the 1970s and 1980s many clinicians used steroids for this purpose.

During the past decade controlled clinical trials done in the United States reexamined this question by using selection criteria that allowed for potentially septic patients to be entered into the study at an early time point and using very large doses of steroids. (e.g. see the VA Systemic Sepsis Cooperatve Study⁽⁴⁾. These studies found that steroids were not beneficial. Thereafter, The Infectious Disease Society of America Working Group on Steroid Use concluded that steroids should not be used for Gram-negative sepsis associated with shock⁽⁵⁾.

Similar scenarios are currently being played out with other proposed panaceas for septic shock. These include receptors and their antagonists, MAb against endotoxin and various cytokines (Table 1). Many technology companies have watched their stock prices soar based on the initial excitement generated by case reports and preliminary efficacy trials, only to come tumbling down when the results of well controlled double-blinded studies were reviewed by the FDA.

Table 1 Some proposed treatments for septic shock

Richard Wenzel et al.⁽⁶⁾ in a recent review on our current understanding of sepsis highlighted the importance that double-blinded randomized controlled studies have in clarifying the usefulness of these agents. Initial studies with HA-1A and E-5, two MAb with activity against endotoxin, showed no benefit. Post hoc unplanned subset analysis suggested that there were subsets of septic patients that might be helped by these agents. However, it was puzzling that these subsets of patients were different. The risk of post hoc analysis was demonstrated when follow-up studies of both antibodies failed to show benefit.Post hoc analysis often has problems with bias or confounding. In the original HA-1A trial, at least part of the explanation for the apparent benefit was that the placebo group received inadequate antibiotic therapy more often than the treated group.

Analysis of the dose-response curve data obtained during a phase II trial of a IL-1 receptor antagonist provided great hope that this agent would benefit septic patients. However, although the study was randomized and controlled, it was not blinded, leaving open the possibility that bias occurred. A controlled randomized phase III study was done where the investigators and patients were blinded to the treatment and this agent was not found to be helpful. Once again a post hoc analysis suggested benefit in a subset of patients, but a subsequent study involving cases meeting this subset definition did not show the antagonist to be of value.

Multimillion dollar industries can arise from treatments with unproven benefits. For instance, home apnea monitors are now widely used in developed countries in an attempt to prevent sudden infant death syndrome (SIDS). Various companies are involved in the manufacturing, selling, and maintenance of these monitors. Furthermore, home health companies profit by providing services that allow parents to use this technology in their homes.

So what are the scientific data that justify these expenditures? A review by Keens and Davidson-Ward⁽⁷⁾ on this subject had the following to say about the use of apnea monitors in infants: “Although scientific studies have not been performed to prove the efficacy or lack of efficacy of home apnea-bradycardia monitoring in saving the lives of these infants, they have high risk for subsequent apnea, and home monitoring is used to detect these episodes.” The authors then conclude in their summary section that “Even if treatment of these infants does not have a large impact on the SIDS rate for the general population, thorough diagnostic evaluations and appropriate use of home apnea-bradycardia monitoring is indicated for this population and may reduce their risk of morbidity and mortality.”

With all due respect to these investigators, I come to a different conclusion. Although there are certainly psychosocial and legal issues that need to be considered, I can only surmise that the use of home apnea-bradycardia monitors begs for development and implementation of a large well controlled study, rather than unlimited continued use.

Once an empirical practice has gained a foothold in medicine, additional modalities often arise and flourish based solely on the original practice. This has been referred to as “spiraling empiricism.” For instance, the use of extracorporal membrane oxygenation (ECMO) originally started out as an intervention for use in neonates with severe pulmonary disease recalcitrant to treatment by mechanical ventilation. Although the efficacy of ECMO for this indication rests on one published randomized study and data from a national registry, the use of ECMO has expanded to include a host of other conditions in both infants and children.

There is also the phenomenon of “optimistic bias.” This involves instances where some studies suggest a benefit, but the conclusions extend well beyond what the data merit. For example, most physicians strongly advocate for prenatal care, believing it is a cost-effective means for promoting healthy infants. However, Huntington and Connell⁽⁸⁾ recently had the following to say regarding the effectiveness of prenatal care: “The evidence that prenatal care pays for itself is simply not strong enough to merit the virtual certainty with which this claim has been espoused. There is considerable risk in basing health programs and public policy on information that is more optimistic than scientific.”

One way to get around the problem of empiricism due to insufficient data is the use of meta-analysis. Although meta-analysis provides a useful tool in some instances, it has limitations that often go unheeded. All meta-analyses are hampered by the potential for publication bias; i.e. studies with negative findings may be both less likely to be written up for publication and accepted by a journal. Furthermore, the quality of meta-analysis can vary greatly. Edgar and Smith⁽⁹⁾ make the following point about the effect sample size: “...results of meta-analyses that are exclusively based on small trials should be distrusted-even if the combined effect is statistically highly significant. Several medium size trials of high quality seem necessary to render results trustworthy.”

Factors, in addition to study size, are also relevant for assessing whether the conclusions reached by meta-analysis are justified. The quality and heterogeneity of the studies included and excluded from the analysis must be examined. Just as the design of an individual study is critical to its quality, the caliber of the criteria used to do a meta-analysis are crucial to the validity of the conclusions reached.

CAN EMPIRICISM EVER BE JUSTIFIED?

The question I would next like to consider is whether empiricism is always wrong? One might think that after everything I said so far, I would not even raise this issue. However, a discussion on the use of the conjugated pneumococcal vaccine, initiated by Dr. Donna Ambrusino at last year's Pediatric Academic Societies Pneumococcal Theme Session, made me rethink this issue.

During the last four decades various vaccines have been developed and prospectively evaluated in well controlled studies. This has led to a remarkable decrease in the incidence of various diseases that previously had caused morbidity and mortality in many thousands of individuals(Table 2). For example, wild-type polio virus has been eliminated from this hemisphere and soon this disease will be eradicated worldwide.

Table 2 Reported cases of vaccine-preventable diseases in the United States⁽¹⁶⁾

Considerable progress continues to be made in improving and developing additional vaccines. For example, several recently completed double-blinded well controlled studies have shown that the acellular pertussis vaccine has a lower incidence of side effects and appears to be more efficacious for preventing pertussis in infants than the whole cell vaccine^{(10, 11)}. These types of studies have been instrumental in allowing us to demonstrate both the efficacy and cost-effectiveness of many of the vaccines used today. Many dollars are saved for every dollar spent on vaccinations (Table 3).

Table 3 Cost savings per dollar invested in various vaccines^*

For some other vaccines the capacity to prevent disease has been shown, but the question of cost-effectiveness is more complex. For example, varicella vaccine is not very cost-effective when evaluated by the cost per individual immunized. However, it is cost-effective from a societal standpoint if factors such as parental days missed at work are considered⁽¹²⁾. Disappointingly, some HMOs have argued that it is not cost-effective, or perhaps not profitable, for the HMO to give this vaccine.

The conjugated pneumococcal vaccine presents a tougher issue. The question posed by Dr. Ambrusino was whether the conjugated pneumococcal vaccine should be licensed and then examined postlicensure or studied prospectively in efficacy trials. This is not unprecedented. The nonconjugated polysaccharideHaemophilus influenzae vaccine was licensed without studies being done in the United States, and postmarketing studies showed the vaccine was only marginally effective. Similarly, the conjugate H. influenzae(Hib) vaccine was examined in the United States only postlicensure. However, in contrast, this vaccine has proven to be remarkably effective⁽¹³⁾. It is estimated that by preventing H. influenzae meningitis and the associated mental retardation the conjugated Hib vaccine saves $460,000,000 each year in the United States.

With this background the issue I would like to consider is whether the conjugated pneumococcal vaccine should be prospectively studied in well controlled trials before widespread use, or be approved and the effectiveness examined postlicensure. Currently little information is available regarding the efficacy of this vaccine. Some of the potential advantages of approving the conjugated pneumococcal vaccine without prospective efficacy include the following. 1) Some experts believe that serum levels of antibodies needed for protection can be predicted. If true, then use of the vaccine can be based on immune response and safety studies. This would allow more rapid incorporation of the vaccine into our immunization program. 2) Elimination of efficacy studies would decrease the cost of bringing the pneumococcal vaccine to the market. 3) If the vaccine is as effective as the Hib vaccine it could prevent approximately 3,000 cases of meningitis, 50,000 cases of bacteremia, 500,000 cases of pneumonia, and 7,000,000 cases of otitis media per year.

A number of potential disadvantages of approving the conjugated pneumococcal vaccine without prospective efficacy also exist, including the following. 1) Conclusions regarding vaccine effectiveness may be misleading. For example, if no decrease in pneumococcal disease occurs the year after licensure it could be due to vaccine ineffectiveness or another factor such as a bad influenza epidemic causing a marked increase in bacterial superinfections. 2) The prevalence of different strains ofStreptococcus pneumoniae and the recipient's immunologic response to these strains can vary between different ethnic groups. Thus, vaccine efficacy shown in one country may not predict what will happen elsewhere. 3) Findings that are both unexpected and useful for this and other vaccines may be overlooked. There is a DNA capsule coding gene that can cause the pneumococcus to shift from one type to another type. For example, S. pneumoniae 23F has been shown to change its capsule to type 14. Nesin and Tomasz⁽¹⁴⁾ have shown that pneumococcal capsular transformation may be a frequent mechanism by which the bacteria avoids serotype-specific opsonophagocytic antibodies. These studies raise the question of whether the vaccine might lead to an increase in this type of switching, thus resulting in decreased vaccine effectiveness.

The issue of doing efficacy trials versus postlicensure marketing is even more problematic for development of an AIDS vaccine. More than a dozen candidate vaccines have been tested in human phase I and II trials. The main obstacles to progress are a deficiency of definitive information on what constitutes a protective immune response, and the lack of a good animal model for HIV disease. Furthermore, consideration of testing one or more of these vaccines in large scale trials raises daunting ethical and social issues that need to be considered (Table 4).

Table 4 Some of the issues that need to be considered prior to implementing an HIV vaccine study

Empiricism also affects us in areas other than biomedical research including education and administration of health care. For example, the Pediatric Residency Review Committee puts in place a number of rules that attempt to achieve various education goals. Within the past year a number of us had the opportunity to review changes this committee was proposing in their upcoming regulations. One of these proposed rules suggested that pediatric programs have at least one general pediatrician for every six residents in the program. Their educational goal was to increase the general pediatric capabilities of residents when they left training. Although this is certainly a goal to aspire to, they could provide no data to support that this financially burdensome change would achieve their stated purpose. Rather, the number was justified by noting that this is the ratio that Family Practice uses.

There are several problems with this idea, including that the needs of a general pediatrician differ substantially from a family practitioner. Additionally, many pediatric subspecialties are able to provide residents with some of the aspects of training needed by general pediatricians. Determination of what modalities allow for effective education for students, residents, and practicing physicians can best be determined through research studies designed to evaluate various educational methods.

The question I started out this portion of the talk with was whether empiricism is ever okay. My answer is probably yes-on rare occasion. However, we must be very careful in choosing those times. The decision to use an empirical approach should never be based on what is fastest, easiest, or most expedient, but what is best for our patients.

POTENTIAL SOLUTIONS

During the last part of this talk I would like to ask three questions. I will address these queries by proposing possible ways that those of us involved in research can help develop the necessary tools that will allow for an increased number of well controlled studies and by that keep empiricism to a minimum.

What role should government have in research? In an editorial in The Lancet entitled “US Science on the Slide” Richard Horton⁽¹⁵⁾ notes that John Gibbons, President Clinton's assistant for science and technology, has undertaken a politically risky strategy by saying that scientists should reject the notion that they should stay out of politics. Gibbons states that avoidance of the political process is a luxury that we cannot afford, and he calls on scientists to be partisan advocates for their work. Horton notes that American biomedical science needs vocal leaders. He concludes by asking “where are they?”

The challenge is clear. We must continue to advocate (shout, scream, and cajole) the overwhelming benefits of research to government officials. Research is cost-effective for society when viewed on a long-term basis. The idea that research, especially biologic research, is extremely important to the health, social, and economic foundations of our society must be hammered home by forcefully and repetitively stating this message.

New initiatives can and should be forthcoming from government. A Research! America poll of 1,004 U.S. adults conducted in June 1995 showed that 65% oppose cuts in medical research spending, 73% would pay higher taxes to support medical research, and 60% are willing to designate tax refund dollars for medical research.

Two bills sponsored by Senators Hatfield and others address the issue of increased funds for research. The National Fund for Health Research Act is aimed at increasing the amount of money going to research by both an additional tax on cigarettes and a check-off box on the tax form. The Clinical Research Enhancement Act is designed to stabilize funding for the 75 National Institutes of Health-supported General Clinical Research Centers and should help both promote clinical research and train new investigators. These bills deserves our full-fledged support.

What role do health care institutions, managed care organizations and health care-related companies have in research? These entities are directly involved in patient care and have an ethical obligation to provide the best possible care for their patients. They should support research simply on this basis. Health care-related businesses, including pharmaceutical companies, also have an obligation to provide quality products to their patrons. The viability of for-profit medical care entities in the long run is predicated on delivering high quality cost-effective health care. Therefore, they have multiple reasons to provide substantial funding in support of clinical research.

A compelling case can be made that increased investment in research with a long-term versus the current short-term view to profitability is beneficial to both society and these health care entities. The June 1995 Research! America poll showed 61% of those polled urge congress to provide tax incentives for private industry to conduct medical research. The dollars saved by these entities from studies which compare cost-effectiveness should be shared with those doing the research. Most of the savings must be put back into further research. I recently was invited by Congress to testify on behalf of the pediatric research community and tried to make many of these points.

What initiatives can be promoted from within the research community? Some studies can and should be done without major infusions of money. Members of the pediatric research community must make it their credo to develop and participate in many multicentered studies that focus on outcomes research.

New initiatives such as the “Multicentered Trial Program” sponsored by the Society for Pediatric Research (SPR) and the American Pediatric Socity (APS) should be established to help stimulate multicentered trials. The SPR/APS initiative is designed to provide seed monies and a meeting place to support the initial implementation of well controlled multicentered studies that examine important clinical questions. Additionally, when needed the SPR/APS office will contact other investigators in the relevant subspecialty to help increase the number of institutions participating in a particular trial. During this first year of the initiative there were 11 grants submitted and these were judged competitively based on their science. Our funding allowed us to support the three that received the highest scores. These were multicentered trials to examine conventionalversus high frequency ventilation, management options for meconium-stained infants, and cost-effective utilization of selected antibiotics in the pediatric inpatient setting. Forty-five medical centers have agreed to participate in the antibiotic study.

There are many other issues that should be examined, and we encourage SPR/APS members to submit proposals. Although funding in addition to that provided by the SPR/APS is often necessary to complete the study, what is most needed is a group of investigators who have the discipline to adhere to a mutually agreed upon protocol. If you would like additional information regarding this initiative, please contact the SPR/APS office.

SUMMARY

In conclusion, there are many outside forces that we must deal with in our quest to provide the best possible medical care for our children. We must decrease empiricism in our everyday practice of medicine by doing more research and being active proponents for increased research funding. Let us not shrink from these responsibilities or be overwhelmed by them. Instead, let each of us pledge to deal with them head on while always keeping in sight what I believe is the ultimate aim of all of us involved in pediatric research,i.e. allowing every one of our children to reach adulthood functioning at their full potential. As Obe-Wan Canopi said to Luke Skywalker in the original Star Wars movie “May the force be with you.”