WHO&WHEN PRO: Can LLMs Really Attribute Failures in AI Agents?

Jul 16, 2026

Upload by Aria

Automated failure attribution uses LLMs to locate where and why agentic systems fail, especially as failures become subtler with stronger models. WHO&WHEN PRO introduces a large-scale benchmark built with a controlled pipeline that injects failures only after replaying an exact successful prefix. The benchmark creates 12,326 failed trajectories with golden labels across 3 modalities and 26 scenario benchmarks. Extensive experiments analyze attribution patterns across modalities, protocols, and model families, enabling empirical guidance for future automated systems.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

Introduction

Automated Failure Attribution and Motivation

WHO&WHEN to WHO&WHEN PRO

Controlled Failure-Injection Pipeline

Benchmark Design and Experiments

WHO&WHEN PRO: Can LLMs Really Attribute Failures in AI Agents? Jiale Liu 1 2 Huajun Xi 3 Shaokun Zhang 1 Yifan Zeng 4 Tianwei Yue 5 Chi Wang 2 Jian Kang 3 Qingyun Wu 1 2 Huazheng Wang 2 4 Project page: [https://whowhenpro.github.io](https://whowhenpro.github.io) arXiv :2607 .09996v 1 [ cs .AI] 10 Jul 2026 Abstract Automated failure attribution (Zhang et al., 2025c) uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce WHO&WHEN PRO, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injectsa failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems. 1. Introduction Recent works show that model scaling (OpenAI, 2026 ; Anthropic, 2026 ; Google DeepMind, 2025), together with careful harness engineering (Lou et al., 2026 ; Lin et al., 2026 ; Pan et al., 2026 ; Lee et al., 2026), can produce AI agents capable of operating in diverse digital environments, with demonstrated potential across domains such as coding (Gao et al., 2025b ; Yang et al., 2024 ; Xia et al., 2024), scientific discovery (Ren et al., 2025 ; Bran et al., 2023 ; Boiko et al., 2023), and complex real-world problem solving (Mialonet al., 2024 ; Liu et al., 2025b ; Zhang et al., 2025a ; Liu et al., 2026 ; Song et al., 2024) . However, advancing agentic systems becomes increasingly challenging as their capabilities grow: failures in more capa- 1Penn State University 2AG2ai, Inc. 3Mohamed bin Zayed University of Artificial Intelligence 4 Oregon State University 5Mathos AI. Correspondence to: Jiale Liu <[jiale.liu@psu.edu](jiale.liu@psu.edu)>, Shaokun Zhang <[svz5418@psu.edu](svz5418@psu.edu)>, Huazheng Wang <[huazheng.wang@oregonstate.edu](huazheng.wang@oregonstate.edu)>. ble agents are more subtle and harder to detect, yet these failures provide crucial signals for further improvement. This motivates automated failure attribution, a research direction that uses LLMs to identify where and why agentic systems fail and convert failures into actionable feedback (Zhang et al., 2025c ; Cemri et al., 2025) . The goal is to make failure attribution benefit from model scaling in the same way agent capability does: stronger models should support not only more capable agents, but also better detection of their increasingly subtle failures. More broadly, such human-free feedback potentially provides active signals for building self-evolving agentic systems, thereby enabling agents to improve from their own failures (Gao et al., 2025a ; Fanget al., 2025a) . As an early effort in this direction, Zhang et al. (2025c) introduced WHO&WHEN, a benchmark of 184 failure traces collected from 127 LLM agentic systems. Each trace identifies the failure agent, the error step, and the failure cause in natural language, defining a task of identifying who failed and when. However, WHO&WHEN narrows to text tasks, whereas real-world agentic deployments span much broader modalities, such as image/video reasoning. Its small scale of 184 instances is also insufficient to reflect the breadth and complexity of real-world agentic scenarios. At the core, both limitations arise from the difficulty of obtaining highquality failure annotations without human intervention, a challenge that becomes even greater once moves outside the text domain. To address these limitations, we introduce a scalable pipeline for automatically constructing failure attribution data. Specifically, we rollout agent traject

FAQs

What problem does WHO&WHEN PRO address in AI agents?

It targets automated failure attribution—using LLMs to identify where and why agentic systems fail as those failures become increasingly subtle and difficult to detect.

How are failure trajectories constructed in WHO&WHEN PRO?

The pipeline replays an exactly matched successful prefix, injects an error at a chosen step, and then warm-starts execution to generate a failed continuation so labels follow the decisive-error definition.

What scope and coverage does the WHO&WHEN PRO benchmark provide?

It includes 12,326 failed trajectories across 3 modalities and covers 26 benchmark scenarios, enabling comparisons of attribution behavior across different settings and model families.

Jul 16, 2026

Upload by Aria