Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

Jul 16, 2026

Upload by Jiven

Multimodal large language models (MLLMs) enable clinical VQA and radiology report generation, yet inference-time hallucinations still produce fluent conclusions that can conflict with imaging evidence. Existing fixes often add training, external retrieval, or complex post-hoc verification, harming cost and generalization. This work proposes a holistic, training-free evidence-injection framework using dual-side verifiable anatomical evidence: ROI-guided activation modulation and coordinate-to-token external memory, plus a task-aware dynamic router. Evaluations across multiple tasks, datasets, and models show up to ~6% higher closed-ended accuracy and ~35% fewer open-ended hallucinations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Hallucinations in medical MLLMs

Existing inference-time mitigation methods

Proposed training-free evidence-injection framework

Proposed approach

ROI priors and evidence extraction

Perception-reasoning intervention mechanism

Task-aware dynamic router

Evaluation

arXiv :2607 .00060v1 [ cs .CV] 30 Jun 2026 Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence Rui Hao 1 , Qiankun Li2 ,3†, Junyuan Mao4 , Linghao Meng4 , Dirui Xie 1 , Dayu Tan5 , and Zhigang Zeng 1† 1 Huazhong University of Science and Technology, Wuhan, China 2 Imperial Global Singapore, Imperial College London, London, United Kingdom 3 Nanyang Technological University, Singapore, Singapore 4 National University of Singapore, Singapore, Singapore 5 Anhui University, Hefei, China [ruihao@hust.edu.cn](ruihao@hust.edu.cn) ; [q.li2@imperial.ac.uk](q.li2@imperial.ac.uk) ; [zgzeng@hust.edu.cn](zgzeng@hust.edu.cn) †Corresponding author. Abstract. Multimodal large language models (MLLMs) show strong promise for clinical VQA and radiology report generation, yet inferencetime hallucinations still undermine trustworthy use: models can produce fluent conclusions that conflict with imaging evidence. Existing mitigation strategies typically rely on additional training, external retrieval/-knowledge bases, or multi-stage post-hoc verification, which increases cost and pipeline complexity and often generalizes poorly across models and tasks. To address this, we propose a holistic, training-free evidenceinjection framework that systematically mitigates hallucinations through dual-side evidence injection. By leveraging ROI priors acquired using MedSAM in our implementation, we recalibrate the visual perception trajectory via ROI-guided activation modulation while anchoring the textual reasoning trajectory by mapping anatomical coordinates into discrete semantic tokens as verifiable external memory. Then we introduce a task-aware dynamic router to select modality-specific interventions based on task semantics, balancing perceptual grounding and linguistic fluency. We conduct systematic evaluations on 2 tasks and 5 datasets using LLaVA-1 .5-7B, LLaVA-Med-1 .5-7B, Qwen3-VL-8B/32B, and InternVL-3 .5-8B/38B. Controlled ablations and visualizations further validate the framework, which consistently outperforms baselines across medical benchmarks, improving close-ended accuracy by up to ∼ 6% ↑ and reducing open-ended hallucinations by ∼ 35% ↓ . The code has been made available on GitHub: [https://github.com/Henry991115/SPRG](https://github.com/Henry991115/SPRG). Keywords: Multimodal large language models · Visual misinterpretation hallucination · Hallucination mitigation · Evidence injection. 1 Introduction MLLMs combine visual perception and language generation with emerging clinical reasoning, enabling medical VQA and radiology report generation [1–5] . 2 Rui Hao et al. However, hallucinations remain a key barrier to trustworthy deployment [6], as models may generate fluent outputs that are unsupported by or even contradict imaging evidence [7, 8] . Recent benchmarks categorize medical hallucinations into visual misinterpretation, knowledge deficiency, and context misalignment [9]; in high-stakes settings, such evidence-inconsistent outputs can lead to incorrect lesion judgments and unsafe clinical attributions [10, 11] . To mitigate these risks, many training-free, inference-time methods have emerged. VCD [12], DoLa [13], and OPERA [14]adjust decoding by contrasting perturbed inputs, contrasting layer logits, or penalizing over-trusted tokens. AVISC [15], DAMRO [16], and PAI [17]calibrate attention or visual tokens by correcting outliers, filtering vision-side noise, and reducing text-dominant bias. M3ID [18] strengthens image grounding through visually dominant constraints. However, most existing mitigations rely on internal statistical calibration without explicit anatomical grounding, which limits generalization across clinical tasks with different evidence requirements. In this work, we propose a holistic, training-free inference-time evidenceinjection framework to mitigate visual misinterpretation hallucinations in medical MLLMs. Using ROI priors as v

FAQs

Why do medical MLLMs still produce unreliable outputs?

Inference-time hallucinations can generate fluent conclusions that are unsupported by or contradict imaging evidence, undermining trustworthy clinical use.

What limitation do many existing mitigation strategies share?

They often rely on additional training, external retrieval/knowledge bases, or multi-stage post-hoc verification, which increases cost, pipeline complexity, and can generalize poorly across models and tasks.

How does the proposed framework inject verifiable anatomical evidence?

It uses ROI priors to recalibrate the visual perception trajectory via ROI-guided activation modulation and anchors textual reasoning by mapping anatomical coordinates into discrete semantic tokens as verifiable external memory.

Jul 16, 2026

Upload by Jiven