How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection

Jul 16, 2026

Flaky tests pass and fail on the same code version, weakening the signal of test outcomes and disrupting continuous integration pipelines. Although code-based flakiness detectors achieve strong benchmark scores, practical adoption remains limited. The work argues that flakiness is not a static property of test source code and that many benchmarks inflate F1 via shortcuts and evaluation choices. It builds controlled datasets (including CI-mined cases) and reframes prediction around whether an observed failure is flaky and how environment affects failure probability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Introduction

Background and problem motivation

Limits of code-based detection

Study approach and contributions

How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection ¨Omer Oktay G¨ultekin 1 , Alexander Berndt 1,3 , Jonathan Bell 2 , Thomas Bach 3 , and Sebastian Baltes 1 1Heidelberg University, Germany 2Northeastern University, USA 3 SAP, Germany oemer.gueltekin@stud.uni-heidelberg.de, alexander.berndt@uni-heidelberg.de, [j.bell@northeastern.edu](j.bell@northeastern.edu), [thomas.bach03@sap.com](thomas.bach03@sap.com), sebastian.baltes@uni-heidelberg.de arXiv :2607 .09345v 1 [ cs . SE] 10 Jul 2026 Abstract—Flaky tests pass and fail on the same code version, weakening the signal of test results and disrupting continuous integration (CI) pipelines. Code-based flakiness detectors report strong benchmark results, yet their use in practice remains limited. We argue that the field is studying the wrong problem: Flakiness is not a static property of test code, which often lacks the information needed to decide whether a test is flaky. Analyzing three code-based detectors operating on test code, we found that widely used benchmarks contain shortcuts that inflate reported F1 scores and that evaluation protocols overstate generalizability. To control for these shortcuts, we curated two datasets. The first, C-IDoFT (54468 unit tests from 57 GitHub projects), keeps a developer-confirmed subset of IDoFT’s flaky tests and rebuilds only the non-flaky class from repeated executions instead of fixed versions of flaky tests. C-IDoFT is a controlled counterfactual, not a benchmark for reuse. Our CodeBERT reimplementations of two published detectors scored far above its constant baselines under the published crossvalidation protocol but no better than them once projects were separated. The high scores rested on the labeling shortcut and the evaluation protocol, not on the test code. On FlakeBench, a benchmark restricted to flakiness types typically recognizable from test code, and the same project-disjoint protocol, the models identified nearly all flaky tests. The second dataset, mined from CI logs, contains 86 flaky end-to-end tests that passed and failed on the same commit. The test code and CI log yielded a cause for 42% of them; the other 58% required further execution evidence. Rather than abandoning flakiness prediction, we reframe it around whether an observed failure is flaky and how likely a test is to fail given its execution environment. Our datasets and CI-mining method support this direction. I. INTRODUCTION Flaky tests impede continuous integration (CI) . Flaky tests pass and fail on the same version of the code, so their failures no longer reliably signal a software defect [1] . Thus, flaky tests waste developer time spent debugging spurious failuresand erode trust in test results [2] . The common mitigation is to repeatedly re-execute failing tests in the CI pipeline to separate flaky failures from real ones. Companies such as Google rely on this to keep flaky failures from blocking their pipelines [3], and Meta reports that flakiness reduces test effectiveness and increases the cost of testing [4]. However, repeatedly executing flaky tests is expensive at scale, and at Google flaky testseven undermine rerun-based test selection heuristics [5] . For example, practitioners at SAP report that 500 hours of compute per day go to rerun-based handling of flaky failures during SAP HANA pre-submit testing on the main branch alone [6] . To avoid the cost of repeatedly executing flaky tests, a large body of work tries to detect flaky tests statically, without reruns, using signals such as the test execution history [3, 7, 8], code metrics [9], or combinations of metrics and the test code [10] . The most actively studied approach is code-based detection, where a model is trained on a test’s source code to predict whether it is flaky, ranging from Pinto et al.’s lexical approach and follow-up vocabulary studies [11–14] through feature-based and fine-tuned code models [15–17] to recent methods based on Large L

FAQs

Why are flaky tests harmful to continuous integration?

Flaky tests pass and fail on the same code version, so failures no longer reliably indicate real defects. This wastes developer time and erodes trust in test results.

What is the main critique of code-based flakiness detection?

Flakiness is not a static property of test source code; it emerges from how the test interacts with the system and its execution environment. Therefore, deciding whether a test is flaky from code alone is often ill-posed.

How do the authors evaluate and control the shortcomings of existing benchmarks?

They reproduce multiple code-based detectors and find benchmark shortcuts and overly optimistic evaluation protocols. They then curate controlled counterfactual datasets (e.g., C-IDoFT) and also use a project-disjoint protocol on flake-focused benchmarks and CI-mined end-to-end failures.

Jul 16, 2026

Upload by Clementine