A three-year pilot project set up by the National Cancer Institute (NCI) offers many answers as to whether it is best to use external or internal reviewers to assess concepts ready to move to phase III clinical trials. The two experimental Concept Evaluation Panels (CEP) were disbanded when the project ended in mid-November. Although external evaluations worked, they were more expensive, time-consuming and sensitive to conflicts of interest than the NCI's internal review system.
"This was an expensive process," says Richard Schilsky, Chairman of the Cancer and Leukemia Group B, one of the NCI's cooperative groups that participated in the project. "It took time from a lot of professional people to do it."
Richard Schilsky
The NCI was concerned that internal reviewers were rejecting too many good ideas for research. Thus, two panels were created in October 1999—one in lung cancer and the other in genitourinary cancers—each with roughly equal representation from NCI staff, the cooperative groups of oncologists and external advisors. The NCI also set up a high-tech, web-based system that permitted monthly meetings of the panels over the Internet.
The panels evaluated a total of 36 concepts over the three years, 15 in lung cancer and 21 in genitourinary cancers, diseases that have historically had little inter-cooperative group collaboration. The panels ultimately approved about 60% of the lung cancer concepts and nearly half of the genitourinary cancer concepts.
"The panels actually disapproved more trials than the NCI disapproved historically," says Richard Kaplan, Chief of the NCI's Clinical Investigations Branch. "This endorsed the fact that previous NCI reviews were not eliminating studies that should have been approved."
Importantly, the CEPs also suffered from the potential for conflicts of interest, a sensitive subject for investigators. According to Schilsky, there is evidence that at least one concept reviewed and turned down by a CEP later resurfaced as a drug-company−sponsored trial. "People serving on one cooperative group might serve on a panel where another cooperative group's research is presented," says Schilsky. "We don't believe these concerns apply in the traditional NCI evaluation process."
And the CEP system was more expensive. "It cost somewhere on the order of $200,000 per year. If it were to be expanded to all the diseases in the cooperative group program, which would be a logical thing to do, it would cost over $1 million per year. That's money that could be better used to support other research," says Schilsky.
"It would have been worth continuing if it was better than the system it was intended to replace," agrees Kaplan. "The process wasn't a failure. It was, in an important sense, a success. It just proved to be overkill." The NCI still plans to consult external reviewers for controversial projects on a proposal-by-proposal basis.
