Estimating the Probability of Traditional Copying, Conditional on Answer-Copying Statistics - ACT Working Paper 2015-08

Oct 9, 2026

Statistics used to detect copying on multiple-choice tests typically report p-values for the probability of observing a value at least as large as the one obtained, assuming no copying. Posterior probability of copying is often more decision-relevant but requires knowledge of population copying rates and the sampling distribution of the answer-copying statistic under copying. This paper builds an estimator using estimable quantities, applies it to any answer-copying statistic, and evaluates performance through simulation with real-data examples, including uses, generalizability to other cheating, and limitations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

AccessFree to download

Abstract

Introduction

Traditional copying in standardized testing

Answer-copying statistics and p-values

Limitations of p-values for fraud probability

Estimation framework (posterior copying probability)

Estimator based on estimable quantities

Simulation evaluation and application to real data

Uses, generalizability, and limitations

ACT WORKING PAPER 2015-08 Estimating the Probability of Traditional Copying, Conditional on Answer-Copying Statistics Jeff Allen, PhD Andrew Ghattas December, 2015 ACT Working Paper Series ACT working papers document preliminary research. The papers are intended to promote discussion and feedback before formal publication. The research does not necessarily reflect the views of ACT. Jeff Allen is a statistician in the Research division at ACT. He specializes in longitudinal research linking test scores to educational outcomes and student growth models. Andrew Ghattas is a statistician finishing his PhD in Biostatistics at the University of Iowa. He works for the UI Policy Center on policy evaluation and does freelance Statistical Consulting through Logos Analytics LLC. His main interests are Bayesian hierarchical modeling, simulation experiments, and model selection. Acknowledgements: The authors would like to thank Ruitao Liu, Richard Sawyer, Jim Sconing of ACT, Inc., and two anonymous reviewers for their suggestions on previous versions of this paper. We also thank Jim Wollack of The University of Wisconsin-Madison for providing the item parameters used for simulation. A previous version of this paper was presented atthe 2013 National Council on Measurement in Education National Conference. The final version of this paper will appear in Applied Psychological Measurement. Estimating the Probability of Traditional Copying, Conditional on Answer-Copying Statistics Abstract Statistics for detecting copying on multiple-choice tests produce p-values measuring the probability of a value at least as large as that observed, under the null hypothesis of no copying. The posterior probability of copying is arguably more relevant than the p-value, but cannot be derived from Bayes’ Theorem unless the population probability of copying and probability distribution of the answer-copying statistic under copying are known. In this paper, we develop an estimator for the posterior probability of copying that is based on estimable quantities and can be used with any answer-copying statistic. The performance of the estimator is evaluated via simulation and we demonstrate how to apply the formula using actual data. Potential uses, generalizability to other types of cheating, and limitations of the approach are discussed. Introduction Cheating on standardized tests is a well-documented problem and there have been several high-profile incidents across the U.S. in recent years (NCES, 2013) . One of the most basic and oldest forms of cheating occurs when one examinee copies the responses of another. Traditional copying occurs when sight, sound, or touch (e.g., tapping) is used to obtain responses, without the aid of electronics.1 For traditional copying to occur, students must take the same test at the same time and in the same physical location. Efforts to prevent traditional copying include assigned seating at an appropriate distance (NCME, 2012), spiraling of test forms, physical barriers that prevent visual copying, and test proctors who look for wandering eyes and listen for unnecessary noises. Despite these preventive efforts, traditional copying can still occur for students in the same test center. When cases of suspected copying are reported, answer-copying statistics can be used to obtain additional evidence of copying. Or, the statistics can be used to trigger investigations of misconduct. These statistics are often designed for multiple-choice tests and measure level of item response similarity (Angoff, 1974) . Examples of answer-copying statistics include the K-Index (Holland, 1996), the Omega Index (􀁚; Wollack, 1996, 1997), the S-check (Weslowsky, 2000), the VM-Index (Belov, 2011), and the generalized binomial test (van der Linden & Sotaridona, 2006; Zopluoglu & Davenport, 2012) . Answer-copying statistics often yield a p-value representing the probability that the statistic could exceed a critical value, under t

FAQs

Why are p-values alone insufficient for deciding whether copying occurred?

p-values quantify the probability of observing a statistic as extreme as the one obtained under a no-cheating null hypothesis. They do not directly answer the decision question of the probability that fraud occurred given the evidence.

What problem does the paper address?

The paper addresses how to estimate posterior probability of copying when only estimable quantities are available, enabling use with any answer-copying statistic without requiring the population copying rate or the full distribution under copying.

How is the estimator evaluated and applied?

The paper evaluates the estimator’s performance via simulation and demonstrates how to apply the formula using actual data, while also discussing potential uses and limitations.

Oct 9, 2026

Upload by Mary Man