Empowering Long-form Omni-modal Understanding with Robust Audio Perception

Q: Why is omni-modal understanding still under-explored despite progress in multimodal models?

Because comprehensive datasets with rich, explicitly aligned auditory cues are scarce, which limits learning robust cross-modal reasoning beyond vision-driven cues.

Q: What is AVDC and how does it address deficiencies in existing datasets?

AVDC is a large-scale audio-visual decoupled caption dataset. It generates aligned captions for audio-only, visual-only, and joint audio-visual views, capturing non-speech events and complex cross-modal interactions.

Q: How is AVDC-QA-CoT used to improve audio-visual reasoning in models?

AVDC-QA-CoT extends the resources with question-answer pairs augmented by chain-of-thought rationales, enabling instruction tuning that supports reasoning about both auditory and visual evidence.

Jul 16, 2026

Upload by Asher

Recent advances in large-scale multimodal models have improved vision-language performance, yet robust omni-modal understanding remains limited by datasets that lack rich, explicitly aligned auditory cues. This work introduces AVDC (Audio-Visual Decoupled Captions), an automated pipeline producing tripartite captions: visual-only, audio-only, and joint audio-visual to disentangle semantics and model cross-modal interactions. It further proposes AVDC-QA-CoT for chain-of-thought audio-visual reasoning, supported by a two-stage training strategy. Extensive experiments show consistent gains across video captioning, audio-centric analysis, and omni-modal benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages31

Reading Time1 h 18 min

Abstract

Dataset Construction and Modal Decoupling

AVDC: Audio-Visual Decoupled Captions

AVDC-QA-CoT: Chain-of-Thought QA

[ cs .LG] 1 1 Jul 2026 Empowering Long-form Omni-modal Understanding with Robust Audio Perception Kaiying Yan 1 , Luoyi Sun2 ,3 , Xiao Zhou 1 , and Weidi Xie 1 1 SAI, Shanghai Jiao Tong University, shanghai, China 2 Zhejiang University, Hangzhou, Zhejiang, china, 3 Shanghai AI Lab, shanghai, China Abstract. Recent advances in large-scale multimodal models have driven remarkable progress in vision-language tasks; however, comprehensive omni-modal understanding remains under-explored, largely due to the scarcity of datasets with rich, explicitly aligned auditory cues. To bridge this gap, we present AVDC (Audio-Visual Decoupled Captions), a large-scale dataset designed to disentangle visual and auditory semantics. Specifically, we propose an automated pipeline that leverages off-the-shelf models to annotate videos with tripartite captions: visual-only (V), audioonly (A), and joint audio-visual (AV) . This decoupled structure explicitly captures both modality-specific nuances and complex cross-modal interactions. Building upon this, we introduce AVDC-QA-CoT, a Chainof-Thought augmented question-answering dataset to foster audio-visual reasoning. To fully exploit these resources, we employ a two-stage training paradigm: omni-modal caption generation pre-training on AVDC, followed by instruction tuning on AVDC-QA-CoT. Extensive experiments across diverse downstream tasks, spanning video captioning, audio-centric analysis, and omni-modal benchmarks, demonstrate consistent and significant performance gains, showing the efficacy of our proposed datasets and training strategy in advancing omni-modal perception. Code and to comof view. 2 | | | (b) Question: Which event occurs LAST in the video? Choices: A.Crinkling sounds from plastic wrappers B.A muted thud as bagged toys are set down C.The woman mentions the missing checklist D.A hollow clatter from placing the Trash Pack container Answer: D | Reasoning: Question decomposition: order of events Temporal grounding: Crinkling at 0:02… Visual perception: Trash Pack container… Audio perception: Hollow clatter at 0:14… Multimodal reasoning: The clatter (D) aligns with the final visual placement …. | | --- | --- | --- | --- | | | | | | | (c) Q: What changes occurred in the state of the ring-shaped drones in the video? A. transitioned from a state of motion to rest and then back to motion. B. changed from a state of rest to motion and then remained in motion. C. transitioned from a state of motion to rest and then remained at rest. D. changed from a state of rest to motion and then returned to rest. Caption: First, two men are in a room, discussing a drone…The scene then shifts outdoors, where a man in a red hoodie operates a drone with a remote control, performing various flight maneuvers... [MISSING DETAILS] Answer: B. Caption: Two men are in a workshop, discussing and examining a drone...The video transitions to an outdoor scene where the device is flying, [MISSING DETAILS] . Finally, the men return to discussing with satisfaction. [HALLUCINATION] . Answer: <think>...Initially, the drone is hovering...The drone continues to fly around... the man appears to have landed the drone and made it fly...shows transition from motion to rest and then back to motion </think><answer>B. </answer> Caption: Two men examine a large drone in a workshop with lively background music. Outdoors, a man in red flies the drone performing stunts to cheerful music and occasional laughter until it crashes into a building, and the men discuss and laugh while walking toward it. Answer: <think> [QUESTION DECOMPOSITION] ... initial state? transition to?…[TEMPORAL GROUNDING]...began at 1:06 and ended at the end. [VISUAL]The drones are seen hovering, flying, landing …[AUDIO] Mechanical sounds align with hovering and flight phases,…. [MULTIMODAL REASONING]...confirms that the drones changed... </think> <answer>D. </answer> Qwen2 .5-omni <b

FAQs

Why is omni-modal understanding still under-explored despite progress in multimodal models?

Because comprehensive datasets with rich, explicitly aligned auditory cues are scarce, which limits learning robust cross-modal reasoning beyond vision-driven cues.

What is AVDC and how does it address deficiencies in existing datasets?

AVDC is a large-scale audio-visual decoupled caption dataset. It generates aligned captions for audio-only, visual-only, and joint audio-visual views, capturing non-speech events and complex cross-modal interactions.

How is AVDC-QA-CoT used to improve audio-visual reasoning in models?

AVDC-QA-CoT extends the resources with question-answer pairs augmented by chain-of-thought rationales, enabling instruction tuning that supports reasoning about both auditory and visual evidence.

Jul 16, 2026

Upload by Asher