The Nonuse, Misuse, and Proper Use of Pilot Studies in Experimental Evaluation Research

Oct 15, 2026

This article examines how pilot studies are ignored, misapplied, or used appropriately in experimental evaluation research. It highlights a lack of clear theoretical and practical guidance for researchers integrating pilots into evaluation designs. Through statistical simulations, it shows how using pilot effect sizes can lead to incorrect decisions about whether to run full trials and to flawed power calculations. Drawing on literature and simulation results, the article offers actionable recommendations for proper pilot study use.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Abstract

Aims and key arguments

Lack of use guidance

Misuse risks and simulation approach

Practical recommendations for proper use

The Nonuse, Misuse, and Proper Use of Pilot Studies in Experimental Evaluation Research Erik Westlund 1 and Elizabeth A. Stuart2,3,4 American Journal of Evaluation 2017, Vol. 38(2) 246-261 ª The Author(s) 2016 Reprints and permission: [sagepub.com/journalsPermissions.nav](sagepub.com/journalsPermissions.nav)[ ](sagepub.com/journalsPermissions.nav)[DOI: 10.1177/1098214016651489](DOI: 10.1177/1098214016651489)[ ](DOI: 10.1177/1098214016651489)[journals.sagepub.com/home/aje](journals.sagepub.com/home/aje) Abstract This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot studies can be misused, using statistical simulations to illustrate the error that can result from using effect sizes from pilot studies to decide whether to conduct a full trial or using effect sizes from pilot studies as the basis of power calculations in a full trial. Informed by the review of the literature and these simulation results, the authors conclude by proposing practical suggestions to researchers and practitioners on how to properly use pilot studies in experimental evaluation research. Keywords evaluation practice, RCT, research design, evaluation theory When designing and conducting experimental research, pilot studies—small-scale studies conducted before a full trial—are considered a best practice. In evaluation research, where experiments are the gold standard of research design, pilot studies should be an integral part of the research design process. Yet, there is little explicit guidance in the literature on evaluation or research design to guide researchers who seek to incorporate pilot studies into experimental evaluation research. This article has three aims. First, we document the dearth of guidance available to researchers conducting pilot studies as part of experimental research. Using program evaluation in education research as a case study, we also show that although pilot studies are required by grant protocols of major education research grant-making bodies, methods for their appropriate use are rarely 1 Department of Sociology, University of Iowa, Iowa City, IA, USA 2 Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA 3 Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA 4 Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA Corresponding Author: Erik Westlund, Department of Sociology, University of Iowa, 140 Seashore Hall West, Iowa City, IA 52242, USA. Email: [erik-westlund@uiowa.edu](erik-westlund@uiowa.edu) Figure 1. ‘‘Pilot Power’’ web application. discussed or provided. As such, we show how researchers seeking guidance on how to use pilot studies in experimental evaluation research will find little guidance from either theory or practice. Second, we discuss how pilot studies can be misused in experimental evaluation research. Using the shiny framework for the R programming language, we built an application called ‘‘Pilot Power’’ (see Figure 1). We used this application to conduct simulations of pilot studies in experimental research designs to gather insight into how the uncertainty inherent to small studies can result in their misuse (R Core Team, 2015; RStudio, Inc., 2013; Westlund & Stuart, 2016) . We focus on two particular errors: using pilot studies to decide whether to conduct a full trial of an intervention and using pilot studies to determine how many cases to sample in such a full trial. We argue that researchers conducting randomized experiments shouldnever use pilot study effect sizes alone to determine whether a full trial should be conducted. We

FAQs

Why do the authors argue there is little guidance on using pilot studies?

They report scarce theoretical, practical, and empirical instruction in evaluation and research design literature for incorporating pilots into experimental evaluation designs, including limited direction from grant protocols.

What two misuse errors do the authors emphasize?

The article focuses on using pilot study effect sizes to decide whether to conduct a full trial and using pilot effect sizes as the basis for power calculations in that full trial.

How should pilot study results be used according to the article?

Pilot results should be treated as one data point among many when deciding on whether a full trial is worth conducting, and they can also support feasibility assessment of the intervention or study design.

Oct 15, 2026

Uploaded by supergirl