Q-TriM - Question-Guided Tri-Modal Attention for Audio-Visual Question Answering

Sep 21, 2026

Uploaded by Asher

Audio-Visual Question Answering (AVQA) requires joint reasoning over synchronized video and audio in response to a natural-language question. Many AVQA methods stack multiple self- and cross-attention layers across text, video, and audio, risking information loss of subtle inter-modal cues and accumulating errors across sequential fusion stages. Q-TriM introduces shallow, parallel multi-modal fusion conditioned on text, yielding Tri-Modal Attention representations and question-guided token filtering. Results achieve state-of-the-art performance on AVQA benchmarks, including strong MUSIC-AVQA-R robustness and out-of-distribution generalization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

AccessFree to download

Introduction

Audio-Visual Question Answering background

Problem with deep stacked attention

Proposed approach: Q-TriM

Q-TriM: Question-Guided Tri-Modal Attention for Audio–Visual Question Answering arXiv :2607 .03825v 1 [ cs .CV] 4 Jul 2026 SungHun Kim Dept. of Computer Science and Engineering Korea University Seoul, South Korea [sainthun12@korea.ac.kr](sainthun12@korea.ac.kr) SeungJun Baek⋆ Dept. of Computer Science and Engineering Korea University Seoul, South Korea [sjbaek@korea.ac.kr](sjbaek@korea.ac.kr) Abstract. Audio-Visual Question Answering (AVQA) extends classical VQA by requiring joint reasoning over video and synchronized audio. However, many AVQA systems rely on deeply stacked layers of self-and cross attention across text, video, and audio. Such sequential stacking may incur loss of information such as subtle inter-modal cues over the layers, causing errors to accumulate across sequential attention layers during the fusion. We introduce Q-TriM which performs multi-modal fusion in a shallow and parallel manner instead of a deep and sequential manner. For Q-TriM, we propose a novel framework for attention operation incorporating video and audio conditioned on text. As a result, we obtain not only standard cross attention outputs but also Tri-Modal Attention representations in which Query, Key, and Value come from distinct modalities. These attention representations are combined in parallel at a single stage, thus avoiding the multi-modal fusion with deep stacksin order to mitigate error accumulation and depth-induced issues. QTriM achieves state-of-the-art performance on three AVQA benchmarks, including substantial gains on MUSIC-AVQA-R, which demonstrates its robustness and out-of-distribution generalization. Code is available at [https://github.com/Sunghun95/Q-TriM](https://github.com/Sunghun95/Q-TriM) Keywords: Audio-Visual Question Answering · Attention Model · Multimodal Fusion 1 Introduction AVQA(Audio-Visual Question Answering) is the task of answering a naturallanguage question given a video with synchronized audio, extending classical VQA [1] by jointly leveraging acoustic and visual streams [42] . AVQA spans many use cases: (1) scene-aware dialog/assistive agents that reason over live audio–video streams [37],(2) long-form egocentric “memory” systems that find evidence moments for open-ended questions [7], and (3) first-person action settings where EPIC-Fusion aligns sounds with egocentric visuals before answering [15] . ⋆ Corresponding author. 2 S. Kim and S. Baek Fig. 1. Overview of Q-TriM. Given input video, audio, and text, Q-TriM combines representations from Tri-Modal Attention, together with cross attention, to perform a shallow, parallel fusion of the modalities for reasoning of the answer. In AVQA, extracting salient evidence from both video and audio is essential. Most AVQA systems rely on stacked attention layers across text, video, and audio [16, 20, 25, 44] and deepen the attention stack to capture inter-modal dependencies. In general, neural layers perform the abstraction and compression of information. On the one hand, these operations are necessary to selectively extract essential information, e.g., Information Bottleneck (IB) principle [40] . On the other hand, they accompany the loss of information [13, 28, 36], which may be problematic for fusing information from many modalities. For example, a modality processed in earlier layers may lose important inter-modal cues, making them unavailable for fusion at deeper layers. A simple remedy is to design architectures that achieve similar effects with shallow attention layers. However, since AVQA must integrate three modalities, stacking attention is often unavoidable. In standard cross-attention [41], only two modalities interact at a time; a Query attends to a single Key–Value source, and thus at most two modalities interact simultaneously. For example, prior methods [20, 21] first process text and video with cross-attention, and then combine its output with audio representation via another cross-attention. Thus, coordinating tex

FAQs

What task does Q-TriM address?

Q-TriM targets Audio-Visual Question Answering (AVQA), which answers natural-language questions using video with synchronized audio while reasoning over both streams together with text.

Why do existing AVQA systems often fail under deep attention stacking?

Deep stacked self- and cross-attention can lose subtle inter-modal cues in earlier layers due to information compression, making them unavailable for later fusion and causing errors to accumulate across sequential stages.

How does Q-TriM change the fusion design to improve robustness?

Q-TriM uses shallow and parallel multi-modal fusion instead of deep and sequential stacking, including a Tri-Modal Attention operator where queries, keys, and values come from distinct modalities, combined in a single stage and reinforced with question-guided token filtering.

Sep 21, 2026

Uploaded by Asher