WILDTRACE Benchmarking Natural Evidence Trails in Long-Context Reasoning

Jul 16, 2026

Answering complex questions over long documents depends on integrating evidence dispersed across distant passages within the same source. Existing benchmarks often rely on needle probes, planted facts, or reverse-engineered chains that may differ in distribution or placement, obscuring whether success reflects real source reasoning or artifacts. WILDTRACE introduces 481 tasks over 214 naturally occurring long-form sources, defining seven source-internal evidence geometries and validating trail necessity, groundedness, rubric fidelity, contamination resistance, and answerability. Evaluations show best models reach 75.3% yet struggle on counterfactual and causal attribution settings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

Abstract

Introduction

arXiv :2607 .09328v 1 [ cs .CL] 10 Jul 2026 WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning Zixin Chen 1 ,2 ∗Peng Liu2 Haobo Li 1 Rui Sheng 1 Jianhong Tu2 Xiaodong Deng2 Fei Huang2 Kashun Shum 1 ,2 Dayiheng Liu2 Huamin Qu 1 1 Hong Kong University of Science and Technology 2 Qwen Team, Alibaba Group [zchendf@connect.ust.hk](zchendf@connect.ust.hk) Abstract Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages. In an incident report, the operating condition, design flaw, and missed safety check that jointly explain a disaster may appear dozens of sections apart; in a novel, a character’s true motive may surface only through scenes far removed from the moment it becomes relevant. This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts. We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document’sown causal, temporal, and narrative logic. Drawing on Pearl’s causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents. A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multistage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance and answerability. Across 18 frontier systems evaluated under evidence-withheld conditions, the strongest reaches 75.3%, with pronounced weaknesses on reasoning-intensive geometries such as counterfactual branching and causal attribution. These failures persist despite sufficient context capacity. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research. 1 Introduction Long-document reasoning is not merely retrieval over more tokens. For a given analytic question, a model must decide which ordinary source fragments are load-bearing evidence, which sameregister details are distractors, and which causal, temporal, comparative, abductive, or counterfactual dependency makes the selected fragments jointly warrant an answer. In literary analysis, a character’s motive may not reside in the scene where the decisive action occurs; it may be warranted only by linking an early promise, a later contradiction, and a delayed consequence that changes how earlier scenes should be read. Structured technical documents pose the same burden in a more procedural register. Even when a report contains findings, headings, tables, and cross-references, a follow-up ∗Work done during internship at Qwen Team Preprint. question may require linking an operating condition, a design constraint, and a review lapse whose dependency is not packaged as the answer to that query. The relevant passages may all be locatable, but the task is to recover why they jointly support the conclusion. Many existing evaluations measure important adjacent capabilities, but often control the evidence environment in ways that reduce the burden of evidence discovery and relation recovery. Needle-style and controlled long-context probes measure access, position sensitivity, aggregation, and robustness, yet the relevant facts are inserted or otherwise benchm

FAQs

What challenge does WILDTRACE target in long-context reasoning?

WILDTRACE targets the difficulty of reasoning with source-internal evidence that is naturally dispersed across distant passages and must be integrated to jointly warrant an answer.

How does WILDTRACE design its evaluation tasks and evidence trails?

It defines seven source-internal evidence geometries, constructs tasks by mining candidate trails from document structure before question writing, and applies multistage validation for necessity, answer groundedness, rubric fidelity, contamination resistance, and answerability.

What performance and failure patterns are reported for frontier models?

Under evidence-withheld conditions, the strongest system reaches 75.3%, with notable weaknesses on reasoning-intensive geometries such as counterfactual branching and causal attribution.

Jul 16, 2026

Upload by Clementine