So-called 'pilot studies' often need more rigorous consideration before taking flight, according to a paper published in late October. The authors, a group of biostatisticians, call on researchers to exercise more care when designing and conducting these types of investigations (Clin. Transl. Sci. 4, 332–337, 2011).
When done right, pilot studies serve an important purpose; they usually involve small numbers of animals or participants and help researchers test the feasibility of methods that will be used in larger, more involved and expensive trials. Scientists can, for example, use pilot studies to test the tolerability of a given compound or the ease of trial subject recruitment. And often this information helps investigators net grant money to pursue larger projects.
But too often, the new paper argues, investigators see pilot studies as unimportant and fail to design them properly, and that attitude can harm investigators' careers and stymie scientific progress. That's an especially pertinent problem for the research community, given that pilot studies seem to be on the rise. An informal search of the Medline database by Paul Nietert, a biostatistician at the Medical University of South Carolina in Charleston and a co-author on the new study, found the number of publications mentioning 'pilot study' in the subheading or as a key term has grown about threefold over the past two decades.
Lehana Thabane, director of the biostatistics unit at the Centre for Evaluation of Medicines in Hamilton, Ontario who published a similar paper on pilot studies last year (BMC Med. Res. Methodol. 10, 1, 2010), says many researchers fail to outline their criteria for success at the outset. Consequently, they have no measure by which to interpret their results.
Alternatively, some investigators try to do too much in a pilot study. Nietert has seen proposals in which the investigator outlines plans to test a variety of hypotheses and conduct sophisticated statistical analyses on data from 15 people. “It's not going to be meaningful,” he says. “They're probably underpowered to test all those hypotheses.” Moreover, many pilot studies are never published. That means we can't learn from them, “and we keep repeating the same mistakes,” Thabane says.
Notably, the term 'pilot study' is not often used by the US Food and Drug Administration, according to Lisa Kubaska, a spokesperson at the agency's Center for Drug Evaluation and Research in Silver Spring, Maryland. The agency “often sees proof-of-concept studies, but these are often combined with the phase 1 safety studies as a dual-purpose study.” Nonetheless, background information can prove helpful, and “when such information does not already exist, well-designed pilot studies would be very useful.”
About this article
Cite this article
Willyard, C. Biostatisticians call for more scientifically rigorous pilot studies. Nat Med 17, 1531 (2011). https://doi.org/10.1038/nm1211-1531b