Trust Before Fusion QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

Jul 16, 2026

Multimodal retrieval-augmented generation (RAG) is evaluated with clean evidence, but real retrieval often returns topically relevant yet unreliable material, including false text and misleading images caused by corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, or style transfer. The paper presents QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes. Across four generator/gate stacks, naive fusion proves brittle; source-aware trust resolution (SATR) selects among Parametric, Text-only, and Full-MM candidates using source reliability, achieving a best balanced score of 0.816.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Introduction

Related Work

Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG Saadeldine Eletter Owais Aijaz Preslav Nakov Mohamed bin Zayed University of Artificial Intelligence {saadeldine.eletter,owais.aijaz,[preslav.nakov}@mbzuai.ac.ae](preslav.nakov}@mbzuai.ac.ae) arXiv :2607 . 10798v 1 [ cs .CL] 12 Jul 2026 Abstract Multimodal retrieval-augmented generation (RAG) is often evaluated with clean evidence, yet real retrieval can return topically relevant but unreliable content: false text and misleading images from corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, blends, or style transfer. We introduce QIMG-7, a controlled benchmark for multimodal retrieval pollution in multisentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes, for 1,760 evaluation rows per method. Across four generator/ -gate stacks, naive multimodal fusion is brittle: in the main gpt-4o-mini stack, Full-MM support drops from 0.908 with clean text to 0.490 with polluted text, often making Parametric fallback safer than retrieval. We propose source-aware trust resolution (SATR), a training-free approach that compares Parametric, Text-only, and Full-MM candidate answers and selects among candidate answers or falls back based on source reliability. The Field-Selector variant achieves the best balanced score, 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points. Ablations show that, in this text-first setting, explicit text-reliability modeling is the dominant driver of these gains. Overall, in text-first factual QA with multimodal retrieval conflict, our results support selective trust rather than unconditional fusion. Artifacts are available at [https://github.com/](https://github.com/)[ ](https://github.com/)SaadElDine/Trust_Before_Fusion. 1 Introduction Retrieval-augmented generation (RAG) is widely used to ground large language models in external evidence, improving factuality for knowledgeintensive QA and generation (Lewis et al., 2020 ; Izacard and Grave, 2021) . However, relevant retrieved content is not necessarily reliable: misinformation in retrieval corpora can steer models toward Retrieved evidence is topical but mixed Retriever Figure 1: Motivating failure case for multimodal retrieval pollution. The retrieved evidence is topically relevant but mixed: clean text supports the correct Tokyo Disneyland answer, while polluted text and image evidence suggest the false claim that it opened in 1979 near Paris. A naïve multimodal RAG system may fuse these conflicting signals and produce a confident wrong answer, motivating source-aware trust resolution. confident but false answers (Pan et al., 2023 ; Zenget al., 2025) . In long-form settings, this problem is amplified because an early factual mistake can propagate through a multi-paragraph answer. The multimodal setting makes the problem harder. Modern systems retrieve images alongside text, and multimodal models can use both. Images can help by providing grounding, but they can also mislead via false captions, typographic overlays, out-of-context crops, swapped entities, or visually edited evidence. While robustness research has focused mostly on text-only retrieval, and multimodal RAG benchmarks usually assume clean evidence, the combined problem of multimodal polluted retrieval remains underexplored (Yu et al., 2025 ; Hu et al., 2025 ; Mortaheb et al., 2025) . Figure 1 illustrates the core failure mode: the retrieved multimodal evidence can be relevant but unreliable, and naïve fusion can amplify polluted evidence into a confident hallucination. In this paper, we study multimodal retrieval pollution for multi-sentence factual QA, making the following four contributions: • Benchmark. We construct QIMG-7, a paired clean/polluted multimodal RAG benchmark for multi-sentence factual QA, spanning four datasets, seven image-attack famili

FAQs

What problem does the paper address in multimodal RAG systems?

Retrieved multimodal evidence can be topically relevant but unreliable due to corrupted metadata, misleading images, and edited or adversarial content. Naive multimodal fusion may then amplify pollution into confident false answers in multi-sentence factual QA.

What is QIMG-7 and how is it constructed?

QIMG-7 is a controlled paired clean/polluted multimodal retrieval benchmark for multi-sentence factual QA. It spans four datasets, seven image-attack families (metadata, pixel, and style perturbations), and 16 evaluation regimes with 1,760 rows per method.

How does SATR work and what advantage does it provide over full multimodal fusion?

SATR is a training-free source-aware trust-resolution approach that compares Parametric, Text-only, and Full-MM candidate answers and chooses, composes, or falls back based on structured source reliability fields. The Field-Selector variant achieves the best balanced score (0.816) and improves over Full-MM by 11.7 points, indicating selective trust beats unconditional fusion.

Jul 16, 2026

Upload by River Wang