Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification

Jul 16, 2026

Upload by Sage

Medical image classification models should identify diagnostically relevant regions, yet standard classification losses provide little spatial supervision. Explicit anatomical guidance using segmentation masks can help but requires expensive manual annotation and additional training cost. Leveraging segmentation foundation models for strong cross-modality localization, this work proposes Locus, an anatomical attention regularization framework. Locus guides classifier attention toward meaningful anatomical structures using an adaptive regularization term that balances foreground and background attention. Experiments on eight diverse datasets show consistent classification gains and anatomically grounded, interpretable attention.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Introduction

Motivation and Challenge

Related Work and Limitations

Proposed Approach: Locus

arXiv :2607 . 10851v1 [ cs .CV] 12 Jul 2026 Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification Tonmoy Hossain 1 , Atiqur Rahman* 2 , Farhana Hossain Swarnali* 3 , and Miaomiao Zhang 1 ,4 1 Department of Computer Science, University of Virginia, Virginia, USA 2 Ahsanullah University of Science and Technology, Dhaka, Bangladesh 3 Kahlert School of Computing, University of Utah, Utah, USA 4 Department of Electrical and Computer Engineering, University of Virginia, USA * These authors contributed equally. Abstract. Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated segmentation model. In this paper, we propose a new framework Locus, an anatomical attention regularization framework that leverages pretrained segmentation foundation models to guide a classifier’s attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixelwise alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances attention between anatomical (foreground) and background regions, penalizing the classifier when background attention dominates. We validate Locus on eight diverse medical imaging datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded attention. Our code is available at [https://anonymous.4open.science/r/miccai26_LearnFocus-F3EC](https://anonymous.4open.science/r/miccai26_LearnFocus-F3EC) . Keywords: Image Classification · Attention Regularization · Anatomical Shapes. 1 Introduction Reliable image classification is central to many high-stakes medical applications, from cancer screening to disease diagnosis, yet medical image classifiers are typically trained with only image-level labels, leaving the network to implicitly determine which regions are diagnostically relevant without any direct spatial 2 Hossain et al. guidance [7, 8, 20, 24, 37] . In the absence of such explicit spatial supervision, models learn attention patterns purely from statistical correlations, distributing focus randomly across anatomically irrelevant regions such as surrounding tissue, imaging artifacts, or dataset-specific contextual cues, limiting interpretability and anatomical grounding [18, 19, 37] . Fig. 1: Visualization of overlaid attention maps from SOTA classifiers vs. our anatomy-guided model (Locus), which reweights foundation-model features toward the anatomical regions (e.g., lung and optic disc/cup) for improved performance with better interpretability. Related works. To address this, prior works have explored auxiliary anatomical shape information, for example, encoded as segmentations of relevant structures, to guide classifier attention toward relevant anatomy [5, 10] . To provide the spatial guidance that existing networks lack, a line of work has investigated different ways of directly supervising the classifier’s attention with anatomical structure. First, manually delineated anatomical masks directly constrain the classifier during training [20, 24, 26], but require expert clinicians to annotate the entire training set, making them costly and difficult to scale to large datasets. To

FAQs

Why do standard medical image classification losses often fail to provide anatomically grounded attention?

Because training usually uses only image-level labels, the model learns attention from statistical correlations rather than explicit spatial supervision. This can cause focus to spread to irrelevant regions such as surrounding tissue or artifacts, reducing interpretability and anatomical alignment.

How does Locus use segmentation foundation models to guide attention without training a dedicated segmentation network?

Locus leverages pretrained segmentation foundation models to extract anatomical shape priors across imaging modalities. It then integrates these priors into a regularization strategy for the classifier, avoiding the need for pixel-level annotation and dedicated segmentation training.

What is the key idea of Locus’s regularization term compared with directly enforcing pixelwise mask alignment?

Instead of forcing pixelwise alignment to foundation-model-derived masks, Locus introduces a regularization term that adaptively balances attention between anatomical foreground and background regions. It penalizes the classifier when background attention dominates.

Jul 16, 2026

Upload by Sage