Do All Visual Tokens Matter Equally Object-Evidence Preserving Token Merging for Vision-Language Retrieval

Jul 16, 2026

Upload by Maeve

Multi-vector vision-language retrieval preserves fine-grained visual evidence via maximum-similarity late interaction, yet dense image-side tokens cause costly storage and scoring. Existing token compression can discard or collapse object- and region-level evidence needed by future query tokens. SaMer proposes object-aware token merging that compresses image-side post-projector tokens into K representative centroids while keeping the late-interaction interface unchanged. Using object annotations only as a training merge prior, SaMer avoids detector needs at inference and adapts only the shared projection layer; with K=64 it removes over 93% of tokens, reduces ColPali storage by 16.09×, and improves R@1 while enhancing phrase-level grounding.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Abstract

Introduction

Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval Suhyeong Park 1,3 , Junha Jung2,3 , Jungwoo Park2,3 , Jaewoo Kang2,3∗ 1The Catholic University of Korea 2 Korea University 3AIGEN Sciences Inc. [pshpulip22@catholic.ac.kr](pshpulip22@catholic.ac.kr) {goodjungjun,jungwoo-park,[kangj}@korea.ac.kr](kangj}@korea.ac.kr) arXiv :2607 .04605v2 [ cs .IR] 13 Jul 2026 Abstract Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object-and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into K representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With K = 64, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by 16.09 ×, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select. Code—[https://github.com/dmis-lab/SaMer](https://github.com/dmis-lab/SaMer) 1 Introduction Efficient multi-vector vision-language retrieval requires preserving the object-, attribute-, and relation-level evidence that query tokens can select under late interaction. Multivector retrieval has been widely adopted in text retrieval since ColBERT (Khattab and Zaharia 2020; Santhanam et al. 2022b), which performs maximum-similarity (MaxSim) matching between contextualized query and document token embeddings. Recent vision-language retrievers such as ColPali (Faysse et al. 2024) extend this paradigm to visual inputs by storing image-side patch embeddings and comparing them with query tokens through MaxSim. Similarly, ColQwen2 (Faysse et al. 2024) builds on Qwen2-VL (Wanget al. 2024), whose visual tokenization supports flexible image representation. These models use MaxSim to compare each query token with all image-side tokens and select the ∗Corresponding author. most similar visual token as evidence, making retrieval sensitive to objects, attributes, and relations rather than only global image-text similarity (Li et al. 2019; Diao et al. 2021) . However, preserving token-level image representations makes retrieval expensive in both storage and scoring. A multi-vector retriever stores hundreds to over a thousand image-side token embeddings per image, and retrieval requires MaxSim comparisons between query tokens and all stored image tokens. As the index grows, both memory footprint and scoring latency increase rapidly, limiting largescale image search and retrieval-augmented visual question answering. The issue is especially pronounced in patch-based visual representations, where each image produces many local tokens that must be stored and compared during MaxSim scoring, making patch-level embeddings a major bottleneck for multi-vector VLM retrievers (Dosovitskiy et al. 2021; Tang et al. 2022; Ma et al. 2025a) . One strategy is to compress visual tokens through pruning or merging (Ryoo et al. 2021; Rao et al. 2021; Lianget al. 2022; Marin et al. 2023; Bolya et al. 2023) . Recent multimodal acceleration methods reduce visual tokens for large vision-lan

FAQs

Why can’t generic token compression be safely applied to multi-vector vision-language retrieval?

Late interaction depends on object-, attribute-, and relation-level evidence selected by individual query tokens. If compression prunes or merges phraserelevant tokens, future MaxSim matching may lose the evidence a query requires.

What is SaMer and how does it preserve object-evidence for retrieval?

SaMer performs object-aware token merging on image-side post-projector tokens, forming K representative centroids with a feature-spatial soft assignment. A merge prior guided by object annotations during training discourages cross-instance mixing so the merged tokens remain selectable for later MaxSim queries.

What does SaMer require during inference, and what is trained?

At inference, SaMer needs no ground-truth bounding boxes and no object detector. Training freezes the vision and language backbones and adapts only the shared projection layer to keep compatibility with the late-interaction MaxSim scoring interface.

Jul 16, 2026

Upload by Maeve