Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA

Jul 16, 2026

Omni-modal evidence-seeking question answering requires an agent to answer queries whose supporting evidence is sparsely distributed across videos, audio, images, web pages, and computation outputs. Existing agentic multimodal systems often lose traceability of what is grounded, what is missing, and when evidence is sufficient. Omni-Decision introduces a training-free evidence-state framework that converts each query into an evidence-closure process with structured state tracking confirmed facts, conflicts, dependencies, and open needs, enabling inspectable acquisition, validation, repair, and stopping.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Introduction

Motivation: evidence sparsity and partial observability

Control of answer readiness and stopping

State-maintenance view and cognitive-control analogy

Benchmarks and task variants

Omni-Decision overview: query-scoped evidence state

arXiv :2607 . 1 1433v 1 [ cs .AI] 13 Jul 2026 Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA Ming Ma 1 ,2 Yi Zhu3 ,∗ Yiran Zhong3 ,∗ Feida Zhu3 Weigao Sun3 Junhan Shi4 Lingrui Mei5 Tianming Yang 1 Steven Hoi3 1 Institute of Neuroscience, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Tongyi Lab, Alibaba Group 4 Tsinghua University 5 Institute of Computing Technology, Chinese Academy of Sciences [mam2022@ion.ac.cn](mam2022@ion.ac.cn), [zhu.yee@outlook.com](zhu.yee@outlook.com), [zhongyiran@gmail.com](zhongyiran@gmail.com) ∗ Corresponding authors. Abstract Omni-modal evidence-seeking QA requires agents to answer questions whose evidence is sparsely distributed across videos, audio, images, web pages, and computation results. Existing agentic multimodal systems often leave evidence in scratchpads, tool trajectories, or free-form histories, making it difficult to track what has been grounded, what remains missing, and when the evidence is sufficient to answer. We propose Omni-Decision, a training-free evidence-state system that turns omni-modal QA into a query-scoped evidence-closure process. For each query, Omni-Decision maintains a structured evidence state containing confirmed evidence, unresolved conflicts, fact and computation dependencies, and open evidence needs. A shared state view conditions planning, evidence acquisition, validation, repair, and finalization. Heterogeneous observations from media, web, computation, and verification modules are normalized, judged, and committed through deterministic state updates. This design enables targeted evidence acquisition, preserves sparse cross-modal cues, and provides inspectable control over repair and stopping. Omni-Decision achieves 45.6% accuracy on OmniGAIA and 58.3% on WorldSense, improving over the baselines by +27.3 and +30.2 percentage points, respectively. No-state ablations and trajectory audits further support the role of explicit evidence-state control in multi-step omni-modal evidence seeking. 1 Introduction Omni-modal question answering is moving beyond closed-form perceptual understanding toward evidence-seeking QA [Fu et al., 2024, Wu et al., 2024, Hong et al., 2025, Li et al., 2026] . This setting is closer to practical agentic problem solving: a user asks a question grounded in heterogeneous media, and the answer may require the system to identify relevant visual or acoustic evidence, complete missing attributes from external sources, check consistency across modalities, and sometimes compute a derived value before responding [Nakano et al., 2022, Schick et al., 2023, Yu et al., 2026] . This setting is difficult for three reasons. First, evidence is sparse and distributed [Zhong et al., 2022, Ranasinghe et al., 2025, Ren et al., 2025, Tang et al., 2025, Zhang et al., 2025, Yu et al., 2026] . A relevant clue may appear in a short video segment, a subtitle span, an audio event, an image region, a web page, or a computation result. Second, the evidence chain is partially observable. The system usually does not know in advance which entity attribute, relation, or intermediate value Preprint. will be needed until earlier evidence has been grounded. Third, answer readiness is itself a control problem. A system must decide not only what to acquire next, but also whether a candidate answer is supported, whether a conflict should trigger repair, and whether the remaining gap is unfillable under the available actions [Shinn et al., 2023, Yao et al., 2023, Han et al., 2025, Wang et al., 2026a, Zhang et al., 2026] . These properties make omni-modal evidence-seeking QA a state-maintenance problem rather than merely a context-scaling problem [Wu et al., 2024, Chen et al., 2025, He et al., 2025, Li et al., 2025, Yang et al., 2025, Wang et al., 2026b] . A longer context window or a larger number of frames can expose more raw observations, but it does not by itself specify which entity h

FAQs

What problem does Omni-Decision address in omni-modal question answering?

Omni-Decision targets evidence-seeking QA where supporting information is sparsely distributed across multiple modalities and sources, making it hard to know what has been grounded, what is missing, and when to stop.

How does Omni-Decision represent and manage evidence during a query?

For each query, it maintains a structured evidence state that tracks confirmed evidence, unresolved conflicts, fact/computation dependencies, and open evidence needs.

What makes Omni-Decision effective compared with existing approaches?

By using deterministic state updates that normalize and commit heterogeneous observations into the same evidence state, it enables targeted evidence acquisition, explicit repair control, and improved stopping behavior, leading to higher accuracy on OmniGAIA and WorldSense.

Jul 16, 2026

Upload by Genevieve