Knowledge Guided Synthetic Bug Feedback for LLM Based Unit Test Generation

Jul 16, 2026

Large language models enable automatic unit test generation, yet executable tests often fail to expose real defects. This paper proposes a framework that converts historical real-bug mechanisms into executable feedback targets. It builds structural and semantic representations of real-bug records, retrieves mechanisms relevant to a focal method, and instantiates them as synthetic bugs for iterative test enhancement. Experiments on Defects4J method-level real-bug detection show mechanism-guided synthetic-bug feedback improves detection over execution, coverage, mutation, knowledge, and search baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Motivation: limitations of proxy feedback signals

Framework idea: transforming real-bug mechanisms into executable feedback

Challenges: representation, generalization, and injection

Knowledge-Guided Synthetic Bug Feedback for LLM-Based Unit Test Generation Ziheng Wang, Maike Li, Chen Zhi Zhejiang University Email: {zihwang,22451156,[zjuzhichen](zjuzhichen}@zju.edu.cn)[}](zjuzhichen}@zju.edu.cn)[@zju.edu.cn](zjuzhichen}@zju.edu.cn) arXiv :2607 . 1 1573v 1 [ cs . SE] 13 Jul 2026 Abstract—Large language models (LLMs) have opened new opportunities for unit test generation, but executable tests do not necessarily reveal real defects. This paper studies how historical real-bug mechanisms can be transformed into executable feedback targets for LLM-based unit test generation. The proposed framework constructs structural and semantic representations of real-bug records, retrieves mechanisms applicable to a focal method, and instantiates them as synthetic bugs that guide iterative test enhancement. We evaluate the approach on methodlevel real-bug detection tasks from Defects4J and show that mechanism-guided synthetic-bug feedback improves real-bug detection over execution-, coverage-, mutation-, knowledge-, and search-based baselines. The results suggest that organizing realbug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles. Index Terms—LLM-based unit test generation, synthetic bug injection, real-bug detection, repository knowledge I. INTRODUCTION Unit testing is a fundamental technique for detecting regression defects and validating program behavior. Automated test generation has long explored random testing [1], searchbased testing [2]–[4], and symbolic execution and its hybrid variants [5]–[8] . Recent LLM-based methods further synthesize test code from a focal method, repository context, or natural-language intent [9], [10], and improve compilability and project adaptation by leveraging repository knowledge or language-specific constraints [11]–[13] . However, compilable and executable tests do not necessarily reveal defects. For regression detection, generated tests must not only execute the target code, but also trigger bug-relevant behavior, include effective oracles, and distinguish correct behavior from buggy behavior [14] . To improve generated tests beyond one-shot prompting, existing LLM-based methods introduce feedback into the generation loop. Execution-feedback methods repair candidate tests using compilation errors, runtime failures, or invalid assertions [15], [16] . Coverage-guided methods prompt the model to exercise uncovered statements or branches [17], [18], while mutation-guided methods use surviving mutants as stronger feedback targets [19]–[22] . These signals are measurable and automatable, but they remain proxy objectives for real-bug detection: a test may cover target code without checking the critical semantic boundary, or kill generic mutants without triggering the input conditions on which a real bug depends. Fig. 1 and Fig. 2 illustrate this mismatch using CSVFormat\#printAndQuote(...) in Apache Commons CSV. Under the MINIMAL quote mode, the method checks whether characters in a field fall within the RFC4180 TEXTDATA range and decides whether the field should be quoted. The real bug concerns the safe range of the first character: for a single backslash \ (0x5C), the correct version should emit the character without quotes, whereas the buggy version emits a quoted result. Detecting this bug requires entering the quote-decision logic, constructing an input on the safe-character boundary, and asserting the exact serialized output. In Fig. 1, CoverUp enters the MINIMAL branch and covers the quote-decision logic, but it does not generate an oracle for the safe-character boundary. In Fig. 2, MutGen kills several local mutants and improves mutation score from 0.40 to 0.75, but these mutants mainly encode local perturbations rather than the bug mechanism. Coverage and generic mutation can strengthen tests, but they do not necessarily provide mechanism-level infor

FAQs

Why do compilation- and execution-successful generated tests fail to reliably find real defects?

Because tests may execute the target code and even satisfy measurable proxy objectives, while not triggering the semantic boundary and behavioral oracle required for the real bug to manifest.

What does the proposed framework do with historical real-bug records?

It abstracts them into transferable bug mechanisms by capturing applicability conditions and observable behavioral differences, then instantiates the retrieved mechanisms as synthetic bugs to guide iterative test improvement.

How is mechanism-guided synthetic-bug feedback evaluated and what is the result?

The approach is evaluated on method-level real-bug detection tasks from Defects4J and shows improved real-bug detection compared with execution-, coverage-, mutation-, knowledge-, and search-based baselines.

Jul 16, 2026

Upload by Theodora