Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Biostatisticians call for more scientifically rigorous pilot studies

So-called 'pilot studies' often need more rigorous consideration before taking flight, according to a paper published in late October. The authors, a group of biostatisticians, call on researchers to exercise more care when designing and conducting these types of investigations (Clin. Transl. Sci. 4, 332–337, 2011).

When done right, pilot studies serve an important purpose; they usually involve small numbers of animals or participants and help researchers test the feasibility of methods that will be used in larger, more involved and expensive trials. Scientists can, for example, use pilot studies to test the tolerability of a given compound or the ease of trial subject recruitment. And often this information helps investigators net grant money to pursue larger projects.

But too often, the new paper argues, investigators see pilot studies as unimportant and fail to design them properly, and that attitude can harm investigators' careers and stymie scientific progress. That's an especially pertinent problem for the research community, given that pilot studies seem to be on the rise. An informal search of the Medline database by Paul Nietert, a biostatistician at the Medical University of South Carolina in Charleston and a co-author on the new study, found the number of publications mentioning 'pilot study' in the subheading or as a key term has grown about threefold over the past two decades.

Lehana Thabane, director of the biostatistics unit at the Centre for Evaluation of Medicines in Hamilton, Ontario who published a similar paper on pilot studies last year (BMC Med. Res. Methodol. 10, 1, 2010), says many researchers fail to outline their criteria for success at the outset. Consequently, they have no measure by which to interpret their results.

Alternatively, some investigators try to do too much in a pilot study. Nietert has seen proposals in which the investigator outlines plans to test a variety of hypotheses and conduct sophisticated statistical analyses on data from 15 people. “It's not going to be meaningful,” he says. “They're probably underpowered to test all those hypotheses.” Moreover, many pilot studies are never published. That means we can't learn from them, “and we keep repeating the same mistakes,” Thabane says.

Notably, the term 'pilot study' is not often used by the US Food and Drug Administration, according to Lisa Kubaska, a spokesperson at the agency's Center for Drug Evaluation and Research in Silver Spring, Maryland. The agency “often sees proof-of-concept studies, but these are often combined with the phase 1 safety studies as a dual-purpose study.” Nonetheless, background information can prove helpful, and “when such information does not already exist, well-designed pilot studies would be very useful.”

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Willyard, C. Biostatisticians call for more scientifically rigorous pilot studies. Nat Med 17, 1531 (2011). https://doi.org/10.1038/nm1211-1531b

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nm1211-1531b

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing