Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

Jul 16, 2026

Upload by Riley

Accurate breast cancer classification from mammography depends on effectively integrating complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views. Existing multi-view methods often aggregate features or use single-stage cross-attention, which can entangle view-specific and shared representations and limit cross-view interaction depth. A token-centric dual-view framework is proposed to unify prompt-based adaptation and cross-view fusion using a frozen vision transformer backbone. Dedicated fusion tokens enable bidirectional token-level communication via cross-attention, and multi-depth fusion modules insert interaction across transformer layers. Experiments on VinDr-Mammo and CMMD show consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines, including 50.40% F1 and 0.8090 AUC on VinDr-Mammo BI-RADS.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages13

Reading Time33 min

Introduction

Motivation: multi-view mammography and diagnosis challenges

Prior work: single-view and fusion strategies

Transformer-based cross-view attention and its limitations

Medical foundation models and parameter-efficient adaptation

Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification Aysan Ghayouri Pirsoltana , Shima Babakordia and Mohammad Reza Mohammadia,∗ a School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran ARTICLE INFO Keywords: Breast cancer classification BI-RADS classification Mammography Multi-view fusion Vision foundation models Prompt learning Cross-attention 7 Jul 2026 AB STRACT Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address these limitations, we propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone. The framework reformulates inter-view interaction as structured tokenlevel communication, where dedicated fusion tokens explicitly encode bidirectional information exchange between CC and MLO views via cross-attention, serving as intermediate carriers of cross-view dependencies rather than relying on direct feature fusion. Unlike conventional methods that apply fusion at a single layer, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Fusion tokens are reintegrated into the token sequence and refined by subsequent transformer layers, facilitating hierarchical propagation of complementary information while preserving viewspecific structure. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines. On the VinDr-Mammo BI-RADS classification task, the framework achieves 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting. Ablation studies further validate the effectiveness of token-based fusion and multi- depth interaction design. 1. Introduction Breast cancer remains one of the leading causes of cancer-related mortality worldwide, where early detection through mammography screening plays a critical role in improving survival rates. Despite its clinical importance, automatic interpretation of mammograms remains challenging due to low lesion contrast, complex tissue structures, and subtle inter-class variations. These challenges are further compounded by the need to integrate complementary information across multiple mammographic views. Importantly, clinical diagnosis is inherently multi-view, where radiologists jointly analyze craniocaudal (CC) and mediolateral oblique (MLO) views to exploit complementary anatomical information. However, most existing deep learning approaches do not fully capture this diagnostic process, either relying on single-view analysis or adopting fusion strategies that provide limited cross-view reasoning. Early deep learning methods primarily focused on single-view mammogram classification using convolutional neural networks, achieving promising results but inherently discarding complementary cross-view information [25, 3, 7, 14] . To address this limitation, multi-view learning strategies have been introduced, typically based on early, intermediate, or late fusion of view-specific representations [12, 23, 17] . Early fusion enables information sharing at shallow representation levels, whereas late fusion preserves view-specific feature learning until high-level semantic representations are formed. Intermediate fusion offers a compromise by integrating information during feature extraction. Despite their respective adva

FAQs

Why is integrating CC and MLO mammography views important for breast cancer classification?

CC and MLO views provide complementary characterization of breast abnormalities. Integrating both helps capture more complete information for classification.

What limitation do many existing multi-view learning approaches have?

They often perform feature fusion or cross-attention at a single stage, which can mix view-specific and shared representations and restrict cross-view reasoning across network depth.

How does the proposed token-centric dual-view framework perform cross-view interaction?

It uses dedicated fusion tokens that exchange bidirectional information between CC and MLO views through cross-attention, inserted at multiple transformer depths for progressive, repeated interaction.

Jul 16, 2026

Upload by Riley