[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-564811-105":53,"doc-detail-564811-en":126},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":119,"head_meta":121,"extra_data":123,"updated_unix":125},105,"en","mergeheir-mitigating-multimodal-hallucinations-as-the-tax-of-model-merging","MERGEHEIR - Mitigating Multimodal Hallucinations as the Tax of Model Merging","","Model merging consolidates task-specialized experts into a single deployable model, but it introduces a merging tax manifested as increased hallucinations. Across eight model-merging methods, every merged checkpoint shows a higher hallucination rate than the average of its constituent experts. Simply adapting existing hallucination-mitigation methods to the merged model can break inherited capabilities. MergeHEIR proposes a post-merge adaptation framework that reduces hallucination while preserving inherited expertise via calibration-based layer-wise null-space projection, with theoretical distortion and dimensionality guarantees.",{"@graph":63,"@context":118},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":11,"@type":70,"position":76},"https://docshare.wps.com/template/presentations/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/mergeheir-mitigating-multimodal-hallucinations-as-the-tax-of-model-merging/564811/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/mergeheir-mitigating-multimodal-hallucinations-as-the-tax-of-model-merging/564811.png","ImageObject",442,249,{"name":88,"@type":89},"Putri","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-10-19","2026-10-12",true,{"@type":98,"interactionType":99,"userInteractionCount":47},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104,110,114],{"name":105,"@type":106,"acceptedAnswer":107},"What is the “merging tax” described in the paper?","Question",{"text":108,"@type":109},"After model merging, hallucination rates increase relative to the average of the constituent experts, indicating a systematic “merging tax.” This effect appears across multiple merging methods and evaluated scales.","Answer",{"name":111,"@type":106,"acceptedAnswer":112},"Why do naive hallucination-mitigation adaptations cause problems?",{"text":113,"@type":109},"Unconstrained adaptation to the post-merge model can disrupt inherited capabilities, creating a tension between reducing hallucinations and retaining expertise.",{"name":115,"@type":106,"acceptedAnswer":116},"How does MergeHEIR mitigate hallucinations while preserving expertise?",{"text":117,"@type":109},"MergeHEIR uses small expert-task calibration sets to build layer-wise null-space projectors (via SVD on task-specific activations) and periodically projects post-merge displacement onto these null spaces to preserve inherited expertise.","https://schema.org",{"og:url":78,"og:type":120,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":122,"canonical":78},"index,follow",{"doc_id":124,"site_id":56},564811,1792171490,{"code":4,"msg":5,"data":127},{"doc_id":124,"user_id":128,"nickname":88,"user_avatar":129,"doc_module":9,"category_id":8,"category_name":11,"doc_title":59,"doc_description":61,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":47,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":135,"language":136,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":61,"update_tm":140,"read_time":8},962085571259,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","arXiv : 2609 .32422v 1 [cs .AI] 26 Sep 2026  \nMERGEHEIR: MITIGATING MULTIMODAL HALLUCINATIONS AS THE TAX OF MODEL MERGING  \nJinyu Li 1,* Hao Fang2,* Zhiming Zhang 1 Jiawei Kong2 Bin Chen 1 Shu-Tao Xia2  \n1Harbin Institute of Technology, Shenzhen  \n2Tsinghua Shenzhen International Graduate School, Tsinghua University  \n*Equal contribution  \nABSTRACT  \nModel merging consolidates task-specialized experts into a single deployable model. However, we show that such capability consolidation incurs a merging tax of increased hallucination: across 8 model-merging methods, every merged checkpoint exhibits a higher hallucination rate than the average of its constituent experts. An intuitive approach is to adapt existing hallucination-mitigation methods to the post-merge model, yet this unconstrained adaptation disrupts inherited capabilities, creating a tension between hallucination mitigation and expertise retention. To tackle this challenge, we introduce MergeHEIR, a postmerge adaptation framework designed to reduce this merging tax while preserving expertise inherited from initial experts. Using small expert-task calibration sets, MergeHEIR constructs layer-wise null-space projectors via SVD from task-specific activations collected from the merged checkpoint, and periodically projects the accumulated post-merge displacement onto the resulting null spaces to preserve inherited expertise. Theoretically, we establish minimum-distortion and maximum-dimensionality guarantees, characterize the threshold-controlled adaptation-retention trade-off, and extend perturbation guarantees beyond finite calibration data. Across 24 paired comparisons spanning three MLLM configurations and 8 model-merging methods, MergeHEIR consistently mitigates hallucination while largely preserving inherited expertise, demonstrating a more favorable hallucination-retention trade-off.  \n1 INTRODUCTION  \nModel merging consolidates independently specialized models into a single deployable checkpoint, providing a practical way to combine complementary capabilities without jointly retraining a unified model on all constituent data (Wortsman et al., 2022; Ilharco et al., 2023) . This paradigm is particularly appealing for multimodal large language models (MLLMs), for which capabilities such as chart understanding, geometry, optical character recognition, and visual question answering are often distributed across specialized checkpoints (Wei et al., 2025) . However, capability consolidation alone does not guarantee deployment reliability.  \nWe first investigate visual hallucination after model merging and reveal a systematic merging tax: across eight merging methods and two Qwen2.5-VL (Bai et al., 2025) scales, every evaluated merged checkpoint exhibits a higher AMBER hallucination rate than the average of its constituent experts, as shown in Figure 1 . The mean rate increases from 33.48 to 54.35 for Qwen2.5-VL-3B and from 20.83 to 25.41 for Qwen2.5-VL-7B. Notably, all constituent experts share the same frozen vision encoder and differ only through language-side LoRA specialization. We attribute the resulting tax to visual-evidence utilization collision: expert specialization forms co-adapted language-side computations for using visual evidence, while model merging can perturb the cross-layer interactions through which this evidence influences generation. As a result, the encoded visual representation may remain intact while its downstream use becomes less effective, increasing susceptibility to visually unsupported generation. Appendix B provides a mechanistic interpretation of this effect. These results show that consolidating complementary capabilities does not guarantee reliable visual grounding, making hallucination correction a necessary post-merge stage.  \nHal Rate  \nQwen2.5-VL-3B  \n70  \n60  \n50  \n40  \n30  \n\n| 59.2 59.6 59.5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -","cbCaij4rTFStfkiT","https://ap.wps.com/l/cbCaij4rTFStfkiT","pdf",1737817,31,"English","# Introduction\n## Merging tax and visual hallucination\n## MergeHEIR framework\n## Theoretical guarantees and trade-off","[{\"question\":\"What is the “merging tax” described in the paper?\",\"answer\":\"After model merging, hallucination rates increase relative to the average of the constituent experts, indicating a systematic “merging tax.” This effect appears across multiple merging methods and evaluated scales.\"},{\"question\":\"Why do naive hallucination-mitigation adaptations cause problems?\",\"answer\":\"Unconstrained adaptation to the post-merge model can disrupt inherited capabilities, creating a tension between reducing hallucinations and retaining expertise.\"},{\"question\":\"How does MergeHEIR mitigate hallucinations while preserving expertise?\",\"answer\":\"MergeHEIR uses small expert-task calibration sets to build layer-wise null-space projectors (via SVD on task-specific activations) and periodically projects post-merge displacement onto these null spaces to preserve inherited expertise.\"}]","MERGEHEIR - Mitigating Multimodal Hallucinations as the Tax of Model Merging | PDF",1791844315]