On the Risk of Coding Before Testing: An Empirical Study on LLM-Based Test Generation Workflow

Jul 16, 2026

Upload by Oliver

Large Language Models (LLMs) are increasingly used to generate both source code and corresponding test suites in software engineering workflows. This dual capability underpins test-first and agentic paradigms, yet it assumes that generated tests are independent, reliable oracles. The paper challenges this assumption by studying error propagation: faults in generated code are replicated in generated test artifacts, yielding mutually consistent but incorrect implementations and masking defects. Experiments quantify reduced fault detection effectiveness, show persistence across prompting strategies, and analyze effects in multi-step workflows.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Motivation and oracle independence

Error propagation mechanism

Implications for agentic workflows and validity threats

On the risk of coding before testing: An empirical study on LLM-based test generation workflow Michael Konstantinou SnT, University of Luxembourg Luxembourg [michael.konstantinou@uni.lu](michael.konstantinou@uni.lu) Florian Tambon SnT, University of Luxembourg Luxembourg [florian.tambon@uni.lu](florian.tambon@uni.lu) Mike Papadakis SnT, University of Luxembourg Luxembourg [michail.papadakis@uni.lu](michail.papadakis@uni.lu) arXiv :2607 .05 139v 1 [ cs . SE] 6 Jul 2026 Abstract—Large Language Models (LLMs) are increasingly used in software engineering workflows to automatically generate both source code and corresponding test suites. This dual capability has enabled emerging development paradigms, including test-first and agentic workflows, where a single model is responsible for producing and validating implementations. However, these approaches implicitly assume that generated tests act as independent and reliable oracles—a fundamental requirement for effective software testing. In this paper, we challenge this assumption and investigate whether LLM-generated code biases the generation of subsequent tests. We introduce and empirically study the phenomenon of error propagation, where faults present in generated code are systematically replicated in the associated test artifacts. This leads to cases where incorrect implementationsand tests are mutually consistent, thereby masking defects rather than revealing them. We evaluate this effect across a range of programming tasks and agentic workflows, analyzing the consistency between generated code and test assertions, with particular focus on scenarios of aligned failures. Our study examines (i) whether erroneous code artifacts bias test generation,(ii) whether such bias persists under different prompting strategies, including chain-of-thought reasoning, and (iii) how errors propagate across multi-step workflows in which intermediate outputs are reused as context. The results show that error propagation is both prevalent and impactful: generating tests after faulty code significantly reduces fault detection effectiveness compared to generating tests independently (14% vs. 25%). These findings highlight a fundamental limitation of current workflows, where lack of independence between generated artifacts undermines the reliability of automated testing. Furthermore, our results exposea previously underexplored threat to validity in empirical studies that rely on coupled generation pipelines. Index Terms—LLM-based test generation, error propagation I. INTRODUCTION Large Language Models (LLMs) are increasingly integrated into software engineering workflows, where they are used not only to generate production code but also to synthesize accompanying test suites. This dual capability has fueled optimistic visions of highly automated development pipelines, in which both implementation and validation are delegated to the same generative system. In particular, recent toolchains promote test-first or co-evolutionary workflows, where LLMs generate candidate tests and subsequently produce code that satisfies them, potentially accelerating development while reducing human effort. However, this paradigm implicitly assumes that LLM-generated tests provide an independent and reliable oracle for assessing correctness. In traditional software testing, effectiveness critically depends on the independence between the system under test and the test oracle [1], [2] . When both artifacts are produced by the same underlying model, this assumption may no longer hold. In this paper, we investigate this fundamental assumption by examining whether the generation of faulty code by an LLM biases the generation of tests. When tasked with producing both artifacts for the same problem, an LLM is likely to rely on similar internal representations, assumptions, and reasoning trajectories. This shared generative process increases the likelihood that errors are not independent but systematica

FAQs

What is the core assumption about LLM-generated tests that the paper challenges?

The paper challenges the assumption that LLM-generated tests function as independent and reliable oracles when code and tests are produced through a coupled generative process.

What does the paper mean by error propagation?

Error propagation refers to the systematic replication of faults from LLM-generated code into associated generated test artifacts, causing tests and implementations to agree on incorrect behavior.

How does generating tests after faulty code affect fault detection effectiveness?

The results show that generating tests after faulty code significantly reduces fault detection effectiveness compared with generating tests independently (14% vs. 25%).

Jul 16, 2026

Upload by Oliver