MERGEHEIR - Mitigating Multimodal Hallucinations as the Tax of Model Merging

Oct 12, 2026

Upload by Putri

Model merging consolidates task-specialized experts into a single deployable model, but it introduces a merging tax manifested as increased hallucinations. Across eight model-merging methods, every merged checkpoint shows a higher hallucination rate than the average of its constituent experts. Simply adapting existing hallucination-mitigation methods to the merged model can break inherited capabilities. MergeHEIR proposes a post-merge adaptation framework that reduces hallucination while preserving inherited expertise via calibration-based layer-wise null-space projection, with theoretical distortion and dimensionality guarantees.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages31

Reading Time11 min

AccessFree to download

Introduction

Merging tax and visual hallucination

MergeHEIR framework

Theoretical guarantees and trade-off

arXiv : 2609 .32422v 1 [cs .AI] 26 Sep 2026 MERGEHEIR: MITIGATING MULTIMODAL HALLUCINATIONS AS THE TAX OF MODEL MERGING Jinyu Li 1,* Hao Fang2,* Zhiming Zhang 1 Jiawei Kong2 Bin Chen 1 Shu-Tao Xia2 1Harbin Institute of Technology, Shenzhen 2Tsinghua Shenzhen International Graduate School, Tsinghua University *Equal contribution ABSTRACT Model merging consolidates task-specialized experts into a single deployable model. However, we show that such capability consolidation incurs a merging tax of increased hallucination: across 8 model-merging methods, every merged checkpoint exhibits a higher hallucination rate than the average of its constituent experts. An intuitive approach is to adapt existing hallucination-mitigation methods to the post-merge model, yet this unconstrained adaptation disrupts inherited capabilities, creating a tension between hallucination mitigation and expertise retention. To tackle this challenge, we introduce MergeHEIR, a postmerge adaptation framework designed to reduce this merging tax while preserving expertise inherited from initial experts. Using small expert-task calibration sets, MergeHEIR constructs layer-wise null-space projectors via SVD from task-specific activations collected from the merged checkpoint, and periodically projects the accumulated post-merge displacement onto the resulting null spaces to preserve inherited expertise. Theoretically, we establish minimum-distortion and maximum-dimensionality guarantees, characterize the threshold-controlled adaptation-retention trade-off, and extend perturbation guarantees beyond finite calibration data. Across 24 paired comparisons spanning three MLLM configurations and 8 model-merging methods, MergeHEIR consistently mitigates hallucination while largely preserving inherited expertise, demonstrating a more favorable hallucination-retention trade-off. 1 INTRODUCTION Model merging consolidates independently specialized models into a single deployable checkpoint, providing a practical way to combine complementary capabilities without jointly retraining a unified model on all constituent data (Wortsman et al., 2022; Ilharco et al., 2023) . This paradigm is particularly appealing for multimodal large language models (MLLMs), for which capabilities such as chart understanding, geometry, optical character recognition, and visual question answering are often distributed across specialized checkpoints (Wei et al., 2025) . However, capability consolidation alone does not guarantee deployment reliability. We first investigate visual hallucination after model merging and reveal a systematic merging tax: across eight merging methods and two Qwen2.5-VL (Bai et al., 2025) scales, every evaluated merged checkpoint exhibits a higher AMBER hallucination rate than the average of its constituent experts, as shown in Figure 1 . The mean rate increases from 33.48 to 54.35 for Qwen2.5-VL-3B and from 20.83 to 25.41 for Qwen2.5-VL-7B. Notably, all constituent experts share the same frozen vision encoder and differ only through language-side LoRA specialization. We attribute the resulting tax to visual-evidence utilization collision: expert specialization forms co-adapted language-side computations for using visual evidence, while model merging can perturb the cross-layer interactions through which this evidence influences generation. As a result, the encoded visual representation may remain intact while its downstream use becomes less effective, increasing susceptibility to visually unsupported generation. Appendix B provides a mechanistic interpretation of this effect. These results show that consolidating complementary capabilities does not guarantee reliable visual grounding, making hallucination correction a necessary post-merge stage. Hal Rate Qwen2.5-VL-3B 70 60 50 40 30 | 59.2 59.6 59.5 | | | | | | | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -

FAQs

What is the “merging tax” described in the paper?

After model merging, hallucination rates increase relative to the average of the constituent experts, indicating a systematic “merging tax.” This effect appears across multiple merging methods and evaluated scales.

Why do naive hallucination-mitigation adaptations cause problems?

Unconstrained adaptation to the post-merge model can disrupt inherited capabilities, creating a tension between reducing hallucinations and retaining expertise.

How does MergeHEIR mitigate hallucinations while preserving expertise?

MergeHEIR uses small expert-task calibration sets to build layer-wise null-space projectors (via SVD on task-specific activations) and periodically projects post-merge displacement onto these null spaces to preserve inherited expertise.

Oct 12, 2026

Upload by Putri