A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents

Jul 16, 2026

Large language models are increasingly used by crowdsourcing participants to answer online surveys, threatening the validity and trustworthiness of collected data. This study quantifies prevalence and evaluates detection and mitigation approaches using five survey conditions (N=250) across Prolific and Amazon Mechanical Turk. Distinct signatures of LLM-assisted responses are identified, with usage ranging from under 10% to over 80%. Mitigations reduce LLM use but do not guarantee improved data quality, motivating active screening and AI-targeted questioning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Introduction

Research questions and study design

Detection and mitigation evaluation

A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents Zane Xu New Jersey Institute of Technology Nathan Malkin New Jersey Institute of Technology arXiv :2607 .00403v 1 [ cs .HC] 1 Jul 2026 Abstract Large language models are increasingly used by participants on crowdsourcing platforms when responding to surveys, potentially undermining the validity of collected data. Our study aims to quantify the prevalence of this behavior and investigate methods to detect and prevent it. In a series of surveys (N = 250), we examined conditions such as platform choice, survey length, requests not to use AI, and disabling copy-paste functionality. We were able to identify distinct characteristics of LLM-assisted responses and found that their frequency varied widely, from under 10% on Prolific to over 80% on Mechanical Turk. Mitigation measures reduced LLM usage but did not necessarily improve data quality. No participants employed browser-use agents at the time of our survey, but we report on our own detection experiments. We recommend that researchers actively screen survey responses for LLM usage by recording and analyzing keystroke data and crafting instructions and questions aimed at AI. 1 Introduction Online surveys are an important instrument in scientific research, and for human-centered security in particular, but it is unclear to what extent they can remain trusted and useful going forward. The culprit is the widespread adoption of large language models (LLMs), which are highly adept at text generation and question answering, making responding to survey questions a relatively simple task for them. Many anecdotal reports and recent research [71, 78, 79] suggest that study participants are increasingly using LLMs for this purpose. Copyright is held by the author/owner. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee. USENIX Symposium on Usable Privacy and Security (SOUPS) 2026. August 23–26, 2026, Hannover, Germany. Respondents may turn to LLMs for a variety of reasons, ranging from improving their own writing to wholesale answer generation. Unfortunately, those on the receiving end have no way to tell apart these use cases. While editing assistance would not impact a study’s results, answers written by an LLM without human input are a serious issue for research validity. Survey-based research aims to capture the opinions of real people in the target population, not the confabulationsof a language model. In addition, LLMs’“opinions” have been shown to exhibit a lack of diversity and systematic biases [24, 25, 59, 69, 79] . This presents a particular problem for the human-centered security field, where surveys can be found in a large fraction of publications at venues like SOUPSand are used for determining privacy preferences and contextual norms, understanding the prevalence of and reasons for security behaviors, and driving interface design decisions. Respondents who fill out political, institutional, or product surveys may see their opinions effect change, but participants in scientific surveys accrue no direct benefits. With compensation as their only reward, some may be content to complete a survey as quickly as possible, without regard for the content of their answers. This is especially common on crowdsourcing platforms like Amazon Mechanical Turk, where data quality issues have long predated AI tools [21, 38, 50] . Existing data quality measures are largely not up to the task. LLMs can answer typical attention checks (“select disagree for the third option below”) and comprehension check questions, arguably better than humans can [43, 75] . LLM detectors have been repeatedly shown to be unreliable even for longer texts [44, 68], and survey responses are usually relatively short, making them even more difficult to detect [76] . Recent publications have started to raise the alarm about the

FAQs

Why does LLM usage in survey responses threaten research validity?

Answers generated by a language model without human input can replace the opinions of the target population, undermining validity. LLM outputs may also show limited diversity and systematic biases.

How prevalent is LLM-assisted survey responding in the study?

In surveys (N=250) the frequency of LLM-assisted responses varied widely by platform and condition, ranging from under 10% on Prolific to over 80% on Mechanical Turk.

What methods does the paper evaluate to detect or mitigate LLM use?

The study compares behavioral and procedural approaches, including keystroke-based patterns, attention check questions, and self-report questions. Mitigation measures reduce LLM usage, and the authors recommend active screening using recorded keystroke data and AI-targeted instructions and questions.

Jul 16, 2026

Upload by Lucas Martin