OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

Jul 16, 2026

Upload by Aurora

Omni-modal large language models (OmniLLMs) can process audio and video jointly, but they produce large token sequences under audio-visual inputs, driving high inference cost. Existing token compression for audio-visual data often uses unimodal guidance, ignoring temporal locality of query-relevant evidence and assuming shared temporally aligned information density. OmniFocus introduces training-free, query-guided compression that independently estimates importance for video and audio, achieving modality-symmetric compression and preserving modality-specific salient evidence with alignment. Experiments on Qwen2.5-Omni across four benchmarks validate strong compressed performance, including DailyOmni 25% retention.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

Introduction

Token compression for OmniLLMs

Limitations of unimodal-guided compression

Proposed method: OmniFocus

Experimental evaluation (overview)

arXiv :2607 .03050v 1 [ cs .LG] 3 Jul 2026 OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models Shijie Cao1,2 , Qingyu Zhang2 , Boxi Yu3 , Yuzhong Zhang4 , Boxi Cao2 , Yaojie Lu2 , Hongyu Lin2 , Xianpei Han2 , Le Sun2 , 1 School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, 2 Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences 3University of Limerick 4 CUHK, Shenzhen {boxi2020, luyaojie, hongyu, xianpei, [sunle}@iscas.ac.cn](sunle}@iscas.ac.cn)[ ](sunle}@iscas.ac.cn)[caoshijie22@mails.ucas.ac.cn boxiyu@link.cuhk.edu.cn](caoshijie22@mails.ucas.ac.cn boxiyu@link.cuhk.edu.cn)[ ](caoshijie22@mails.ucas.ac.cn boxiyu@link.cuhk.edu.cn)[123090848@link.cuhk.edu.cn](123090848@link.cuhk.edu.cn) Abstract Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost. Existing audio-visual token compression methods often rely on unimodal guidance, overlooking the temporal locality of query-relevant evidence in audio-visual inputs and implicitly assuming that the two modalities share a temporally aligned information density distribution. We propose OmniFocus, a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio, enabling a modality-symmetric compression design that preserves modality-specific salient evidence while maintaining audio-visual alignment, thereby mitigating the modality bias issue that can arise from unimodalguided compression. Experiments on the Qwen2.5-Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios and outperforms existing baselines on several major benchmark scores at 25% token retention. On DailyOmni with Qwen2.5-Omni-7B at 25% token retention, OmniFocus maintains 59.40 accuracy while delivering up to 1.38 × prefill speedup relative to the full-token baseline, highlighting a favorable practical accuracy-efficiency trade-off. 1 Introduction Omni-modal large language models (OmniLLMs) extend language models to joint audio-visual understanding, showing broad potential across tasks such as audio-visual understanding [19], social and empathetic understanding [23, 33, 31], and audio captioning [36, 4] . However, audio-visual inputs generate large token sequences, which limits the applicability of OmniLLMs in long-form audio-visual settings and resource-constrained environments. Token compression as a method to reduce inference cost has been widely used in vision, audio, and long-text settings [39, 24, 28, 15, 22, 26] . By removing redundant tokens while preserving key evidence, token compression can improve inference efficiency and reduce GPU memory usage without sacrificing much performance. Unlike unimodal compression, token compression in OmniLLMs must preserve audio evidence, visual evidence, and cross-modal alignment simultaneously, which poses unique challenges for compression design. Preprint. LLM Compression Module Vision Encoder Audio Encoder Figure 1: Illustration of the limitation of unimodal-guided audio-visual token compression. Audioguided compression preserves audio-centric performance relatively well, but degrades more noticeably on visual and joint audio-visual understanding tasks, motivating a query-guided and modalitybalanced compression strategy. The modality-type labeling protocol is described in Appendix B. In previous work, researchers typically use one modality as a guiding modality to assess the density of event information, and design asymmetric compression strategies for the two modalities based on this signal [29, 7, 18] . This unimodal-guided asymmetric compression strategy has two limitations: (1)

FAQs

Why do OmniLLMs have high inference cost for audio-visual inputs?

Audio-visual inputs generate large token sequences for OmniLLMs, increasing inference cost and limiting use in long-form or resource-constrained settings.

What limitations arise from existing unimodal-guided audio-visual token compression?

Unimodal guidance overlooks temporal locality of query-relevant evidence and implicitly assumes temporally aligned information density across modalities, which can cause modality bias.

How does OmniFocus address modality bias in token compression?

OmniFocus is training-free and query-guided: it performs independent importance estimation for video and audio, producing modality-symmetric compression that preserves modality-specific salient evidence while maintaining audio-visual alignment.

Jul 16, 2026

Upload by Aurora