Modality Relevance is not Modality Utility Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

Jul 16, 2026

Upload by Noah

Multimodal retrieval-augmented generation (RAG) grounds generation in evidence from heterogeneous modalities such as text, tables, and images, yet real deployments face a cost asymmetry between cheap text/table processing and expensive vision–language model (VLM) image understanding. Prior adaptive routing decides modality or fidelity pre-retrieval, but modality relevance poorly predicts true answer utility. Oracle analysis on MultiModalQA shows many image-supported questions are solvable without images. The proposed post-hoc selective escalation answers first with text+tables, verifies and localizes missing evidence by modality, and invokes VLM only when value justifies visual cost, recovering always-on accuracy with fewer visual calls.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Introduction

Cost asymmetry in multimodal RAG

Relevance–utility gap for modality

Post-hoc selective modality escalation

arXiv :2607 .05438v 1 [ cs .IR] 3 Jul 2026 Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG Xue Li, Yiming Gai Hangzhou International Innovation Institute, Beihang University Hangzhou, China Abstract Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities—text, tables, and images. The dominant deployment choice is binary and made before the model has tried to answer: either run a cheap text(+table) pipeline, or pay for an expensive vision–language model (VLM) over every image. Recent adaptive systems improve on this by selecting the modality or fidelity pre-retrieval, from a question-conditioned predictor of which modality will be needed. We show that this is the wrong decision point. Through an oracle headroom analysis on MultiModalQA, we find that the relevance of a modality to a question is a weak predictor of whether that modality is actually needed to answer correctly: a large fraction of questions whose gold support includes an image are nonetheless answerable from text and tables alone, and a pre-retrieval router that escalates on apparent visual relevance over-escalates substantially relative to an oracle. We propose post-hoc selective modality escalation: answer cheaply from text and tables, run a verifier on the (query, draft answer, evidence) tuple that localizes which modality is missing, and pay for VLM evidence only there. A calibrated value-of-escalation router then decides whether the expected accuracy gain justifies the visual cost. On MultiModalQA, our router recovers the accuracy of an always-on VLM pipeline while issuing far fewer visual calls, and closes most of the gap to the oracle escalation rate. The result extends a routing-signal hierarchy established for retrieval depth and reasoning hops to a third axis—modality—under a single cost-aware selective-escalation view. 1 Introduction Retrieval-augmented generation (RAG) grounds large language models in external evidence by retrieving content and conditioning generation on it. In the multimodal setting, that evidence is heterogeneous: a question may be answerable from a passage, from a table cell, from the content of an image, or from a combination. Production multimodal RAG systems face a basic cost asymmetry. Reading text and tables is cheap, but turning an image into something a language model can use requires a vision–language model (VLM), which is markedly more expensive in latency and compute. The default deployment pattern resolves this asymmetry rigidly: either restrict the pipeline to text(+table) and accept failures on visually grounded questions, or invoke a VLM over every candidate image and pay that cost on every query—including the majority that never needed it. Recent adaptive multimodal RAG aims to spend the visual budget selectively. A prominent line decides which modality or fidelity to retrieve from a question-conditioned success predictor, before any evidence is read or any answer is drafted [Anonymous, 2026]. Agentic pipelines go further, planning modality-specific sub-queries and aggregating across retrieval branches [Anonymous, 2025a,d], but at the cost of many model calls per question. The first family is attractive because it is cheap, but it commits to the visual decision at the earliest and least-informed point in the pipeline: before the system has observed any evidence or attempted an answer. This paper argues, and shows empirically, that the modality decision should be made after a cheap attempt, not before. Our central observation is a modality analogue of a relevance–utility gap that has been documented for retrieval scores in single-hop RAG: the relevance of a modality to a question is not the same as its utility for answering. A question can be about an entity that has an associated image and still be answerable entirely from the surrounding text and table evidence; conversel

FAQs

Why does modality relevance fail to predict modality utility in multimodal RAG?

The paper shows that a modality can be relevant to the question while still being unnecessary for correct answering. Oracle headroom analysis on MultiModalQA finds many questions whose gold support includes an image are answered correctly using only text and tables.

What is the proposed post-hoc selective modality escalation method?

The system first generates a draft answer from cheap text+table evidence, then runs a verifier on the (query, draft answer, evidence) tuple to localize which modality’s information is missing. It escalates to VLM evidence only when the missing gap is localized to the image modality, then regenerates using compact textual sidecars.

How does the system decide whether to pay the VLM cost?

A calibrated value-of-escalation router compares the expected accuracy gain from escalation against the visual cost. This yields a tunable operating point on the accuracy–cost frontier and reduces unnecessary visual calls.

Jul 16, 2026

Upload by Noah