MMA-Former Multi-Window Mixture-of-Head Attention Transformer for Adaptive PNI Prediction in 3D MRI

Jul 16, 2026

Perineural invasion (PNI) is a key prognostic factor in cholangiocarcinoma, and accurate non-invasive prediction from 3D MRI is difficult due to subtle imaging cues and the need to jointly model fine-grained details and global context. MMA-Former introduces a novel end-to-end 3D Coarse-Fine Transformer (CFT) for parallel multi-scale feature extraction, enhanced by Window-Specific Mixture-of-Head attention (WS-MoH) that routes each 3D window to specialized or shared heads. On 168 retrospective T1-weighted MRI scans, it reaches an AUC of 0.752, outperforming CNN baselines (AUC 0.708) and Transformer baselines (AUC 0.681) without increasing parameters.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages5

Reading Time13 min

Abstract

Introduction

Methodology

Datasets and Preprocessing

MMA-FORMER: MULTI-WINDOW MIXTURE-OF-HEAD ATTENTION TRANSFORMER FOR ADAPTIVE PNI PREDICTION IN 3D MRI Youngung Han 1 ,2, Induk Um3, Kyeonghun Kim2, Junga Kim 1, Hyunsu Go 1, Jaewon Jung 1, Woo Kyoung Jeong4, Won Jae Lee5, Pa Hong5, Ken Ying-Kai Liao6, Hyuk-Jae Lee 1, Nam-Joon Kim 1 ,† 1 Seoul National University, Seoul, Republic of Korea 2 OUTTA, Seoul, Republic of Korea 3 Chung-Ang University, Seoul, Republic of Korea 4 Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea 5 Samsung Changwon Hospital, Changwon, Republic of Korea 6NVIDIA AI Technology Center, Taipei, Taiwan †Corresponding author: [knj01@snu.ac.kr](knj01@snu.ac.kr) arXiv :2607 . 10988v1 [ cs .CV] 13 Jul 2026 ABSTRACT Perineural invasion (PNI) is a critical prognostic factor in cholangiocarcinoma. Non-invasive prediction from 3D MRI is challenging, demanding models that efficiently capture both fine-grained details and global context. We propose the Multi-window Mixture-of-Head Attention Transformer (MMA-Former), a novel end-to-end 3D architecture featuring a Coarse-Fine Transformer (CFT) structure for parallel multi-scale feature extraction. We advance this structure by integrating a novel Window-Specific Mixture-of-Head attention (WS-MoH) mechanism. Unlike standard Multi-Head Self Attention (MSA), WS-MoH generates a representation for each 3D window and dynamically routes the entire window to specialized or common attention heads. This enables spatially adaptive feature extraction tailored to the local context of each window, enhancing specialization and reducing redundancy without increasing parameters. Evaluated on a retrospective dataset of 168 T1-weighted MRI scans, MMA-Former achieved an AUC of 0.752, outperforming other 3D architectures, including the best CNN (AUC of 0.708) and Transformer baselines (AUC of 0 .681) . Index Terms— Vision Transformer, Mixture-of-Head attention, Adaptive Feature Extraction, Window-level Routing 1. INTRODUCTION Perineural invasion (PNI), the insidious infiltration of cancer cells along nerve sheaths, is a critical route of metastasis in cholangiocarcinoma. Its presence significantly escalates the risk of recurrence, correlates with poor survival, and dictates surgical planning [1, 2, 3] . Accurate preoperative identification of PNI is therefore paramount for personalized treatment. Despite its clinical urgency, non-invasive PNI prediction remains challenging due to subtle MRI features [4, 5] . Furthermore, most PNI studies are hampered by small cohorts, often numbering in the low hundreds [2, 6], which frequently leads to class imbalance. Methodologically, existing approaches often rely on radiomic features [7], which may fail to capture complex 3D spatial patterns. While end-to-end deep learning offers potential, standard architectures struggle. CNNs [8, 9] are limited in modeling long-range dependencies, while Vision Transformers (ViT) [10, 11] and their 3D adaptations [12] incur prohibitive computational costs in 3D MRI. Axial Coronal Sagittal Fig. 1. Grad-CAM visualizations, highlighting critical regions at the tumor interface. (a) PNI-positive and (b) PNI-negative cases localized by MMA-Former. Hierarchical approaches like Swin Transformer [13] improve efficiency by computing attention within local windows. Building on this, architectures incorporating parallel multi-scale window processing, such as MViT [14] or Focal Transformers [15], have emerged to explicitly capture features at different scales simultaneously. However, the uniform processing of standard Multi-Head Self Attention (MSA) in these methods limits adaptive feature selection and causes attention head redundancy.[16, 17, 18] . The Mixture-of-Head attention (MoH) mechanism [16], inspired by Mixture-of-Experts (MoE) principles [19, 20], addresses this redundancy by treating attention heads as experts. MoH employs arouter to dynamically select a subset of specialized heads for ea

FAQs

What clinical problem does MMA-Former target?

MMA-Former targets non-invasive prediction of perineural invasion (PNI) in cholangiocarcinoma using 3D MRI.

How does WS-MoH differ from standard multi-head self-attention?

Instead of attending at the token level, WS-MoH generates representations per 3D window and dynamically routes the entire window to specialized or common attention heads, improving spatial adaptivity.

What performance did MMA-Former achieve on the retrospective dataset?

On 168 retrospective T1-weighted MRI scans, MMA-Former achieved an AUC of 0.752, outperforming a best CNN baseline (AUC 0.708) and Transformer baselines (AUC 0.681).

Jul 16, 2026

Upload by Genevieve