Super-Generalist Towards Comprehensive and Accurate Medical Image Understanding via Generalist–Specialist Synergy

Jul 16, 2026

Upload by Eden

Medical imaging demands comprehensive, precise interpretation to support diagnosis across diverse clinical conditions. Vision-language generalist models provide broad coverage and zero-shot ability, yet often miss fine-grained anatomical and lesion awareness needed for reliable decisions and spatial interpretability. Supervised specialist models excel on specific tasks but generalize poorly across diseases and anatomies. SuG unifies generalist learning with specialist objectives using spatial priors from multiple segmentation experts and lesion-mask guided text-conditioned attention calibration. Evaluations on chest and abdominal CT benchmarks show state-of-the-art performance and robust lesion grounding, including generalization to lesion types without class-specific supervision.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages31

Reading Time1 h 18 min

Abstract

Introduction

Challenge: Unifying scope, performance, and grounding

Existing paradigms: Specialist vs. Generalist

Proposed method: Super-Generalist (SuG)

Synergistic components for specialist-competitive performance

arXiv :2607 .09135v1 [ cs .CV] 10 Jul 2026 Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist–Specialist Synergy Shaoteng Zhang 1 ,3 Weiwei Cao 1 ,2 Wanxing Chang 1 ,2 Yutong Xie8 , Kai Cao5 , Zaiyi Liu4 , Yu Shi6 , Tingbo Liang7 , Qi Zhang7 , Ling Zhang 1 ,2 , Yong Xia3†,Jianpeng Zhang 1 ,2† 1DAMO Academy, Alibaba Group 2Hupan Lab 3Department of Radiology, Ningbo No. 2 Hospital, Ningbo, Zhejiang, China 4Department of Radiology, Guangdong Provincial People’s Hospital, Guangzhou, China 5Department of Radiology, Shanghai Institution of Pancreatic Disease, Shanghai, China 6Department of Radiology, Shengjing Hospital of China Medical University, Shenyang, China 7The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China 8Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates Abstract Medical images require comprehensive and accurate interpretation to support the diagnosis of diverse clincial conditions. Recent vision–language generalist mod els offer broad task coverage and promising zero-shot capabilities, yet often lack fine-grained anatomical and lesion awareness for reliable diagnosis and spatial in terpretability. In contrast, supervised specialist models achieve strong performance on specific tasks but typically lack generalization across diseases and anatomies. In this work, we present SuG, a Super-Generalist framework that unifies generalist vision–language learning with specialist objectives, enabling both broad general ization and specialist-level diagnostic capability. We perform specialist-enhanced vision-language alignment in SuG by incorporating spatial priors from multiple segmentation experts, including anatomy, class-specific lesion and class-agnostic lesion segmentors that captures lesions beyond anatomies annotated during training. To improve lesion grounding capability, we leverage lesion masks as spatial priors to calibrate text-conditioned visual attention, encouraging disease-related seman tics to focus on clinically relevant regions. We evaluate SuG on extensive chest and abdominal CT benchmarks, including CT-RATE, Merlin, MedVL-CT69K, and several in-house tumor datasets. SuG achieves state-of-the-art performance across a wide range of disease diagnosis tasks and surpasses specialist models on several critical tumor diagnosis benchmarks. Furthermore, SuG demonstrates strong lesion grounding capability, including robust generalization to lesion types lacking class-specific supervision. 1 Introduction Toward comprehensive and accurate medical image understanding, a core challenge in modern diagnostic AI is to unify three foundational pillars: broad disease scope, specialist-competitive diagnostic performance, and interpretable lesion grounding [1–4] . A system possessing all three would combine the clinical breadth to handle diverse diseases, the precision to rival human specialists, ∗ Equal contribution. †Corresponding author: [yxia@nwpu.edu.cn](yxia@nwpu.edu.cn), [jianpeng.zhang0@gmail.com](jianpeng.zhang0@gmail.com) Preprint. Specialist (a) (b) Generalist (c) Super Generalist (SuG) Disease scope Performance Grounding Figure 1: Comparison of medical AI paradigms: Specialist, Generalist, and the proposed Super Generalist (SuG). We evaluate these paradigms across three critical dimensions: disease scope (range of tasks), performance (diagnostic accuracy), and grounding (lesion localization) . (a) Specialists excel in performance and grounding but are limited to a narrow disease scope. (b) Conversely, Generalists support a wide scope but suffer from inferior performance and a lack of precise grounding.(c) Our proposed SuG overcomes these trade-offs, achieving high diagnostic accuracy and precise lesion localization across diverse diseases. and the transparency required for clinical decision-making. However,

FAQs

Which datasets and evaluation settings are used to assess SuG’s performance?

SuG is evaluated on extensive chest and abdominal CT benchmarks including CT-RATE, Merlin, MedVL-CT69K, and several in-house tumor datasets, showing state-of-the-art results across disease diagnosis tasks.

Jul 16, 2026

Upload by Eden