What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks

Jul 16, 2026

Upload by Eden

Saliency maps identify image regions that preserve a model’s behaviour, enabling explanations that can be validated as meaningful visual evidence. SEAMS introduces a sufficiency-based saliency method that optimises a learnable soft mask with a preservation objective over a frozen differentiable model output, such as class probability, CLS embedding, or token representation. The framework uses a compact optimisation pipeline with a learnable budget and a three-way image composite, requiring no distractor dataset, architecture-specific attribution, or differentiable top-k relaxation. Experiments show compact, stable, competitive masks across ViT and ConvNeXt targets and reveal architecture-dependent sufficient evidence.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Abstract

Introduction

Motivation: sensitivity vs sufficiency

SEAMS overview and key idea

Visualisation and targets (class, CLS, token)

What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks Magdalena Tr˛edowicz Jagiellonian University [magdalena.tredowicz@doctoral.uj.edu.pl](magdalena.tredowicz@doctoral.uj.edu.pl) Łukasz Struski Jagiellonian University Arkadiusz Lewicki University of Information Technology and Management in Rzeszów Karolina Pachota Jagiellonian University Medical College Andrzej Grudzie´n Jagiellonian University Medical College arXiv :2607 .09164v1 [ cs .CV] 10 Jul 2026 Mateusz Jagła Jagiellonian University Medical College Jacek Tabor Jagiellonian University Abstract Saliency maps are most useful when they identify the image regions that are sufficient to preserve a model’s behaviour. We introduce SEAMS, a sufficiency-based saliency method that directly optimises a soft mask using a preservation objective. Given a frozen differentiable model output, such as a class probability, CLS embedding, or token representation, SEAMS searches for a compact mask that preserves the selected output. The approach relies on a simple optimisation framework based on soft masks, a learnable budget, anda three-way image composite generated entirely from the query image. As a result, it requires no auxiliary distractor dataset, architecture-specific attribution mechanism, or differentiable top-k relaxation. Experiments with frozen ViT-S/16 and ConvNeXt models show that the same optimisation pipeline can generate objectlevel, class-conditioned, and token-level explanations by changing only the preserved target. The resulting masks are compact, interpretable, stable across random initialisations, and competitive on insertion and deletion benchmarks. Our results also indicate that different architectures often rely on different sufficient evidence while achieving similar preservation fidelity, highlighting the architecture-dependent nature of visual explanations. 1. Introduction Understanding which image regions are responsible fora neural network prediction remains a central problem in DINOv2 ViT-S/ 14 @ 5 18 ViT-S/16 @ 224 Input Mask Mask on Grey Figure 1 . Sufficiency-based saliency across scales and architectures. SEAMS identifies compact image regions that are sufficient to preserve a selected model representation. Top: ImageNet image analysed with a supervised ViT-S/16 encoder. Bottom: DIV2K image analysed with a self-supervised DINOv2 ViT-S/14 encoder. From left to right: original image, optimised soft mask, and the corresponding saliency overlay. Despite differences in image resolution, training paradigm, and architecture, the same optimisation framework consistently identifies a small subset of pixels sufficient to preserve the target representation. explainable artificial intelligence. In computer vision, explanations should be spatially precise enough to distinguish relevant object parts from background content, allowing researchers to verify whether a model relies on meaningful visual evidence. This challenge has become particularly important with the growing adoption of deep Input Class Probability CLS Embedding Patch-Token Violin Lycaenid Ant Figure 2 . One optimisation pipeline, multiple explanatory targets. Each row corresponds to a single ImageNet image, while each column preserves a different target output: a class probability, the CLS embedding, or the full patch-token representation. The optimisation procedure, regularisation, and image composite remain unchanged. Different targets produce different saliency patterns, showing that distinct components of a visual representation rely on different subsets of image pixels. convolutional networks and Vision Transformers, whose internal representations are difficult to interpret directly. Most existing visual explanation methods are based on gradient attribution. Representative examples include vanilla saliency maps [19], SmoothGrad [20], DeepLIFT [18], Integrated Gradients [21], and Layer-wise Relevance Propagation [1] . More

FAQs

What problem does SEAMS address in visual explanation methods?

SEAMS targets the mismatch between sensitivity-based attribution and the question of which pixels are sufficient to preserve a model representation or prediction.

How does SEAMS generate explanations?

SEAMS optimises a continuous soft mask using a preservation objective so that the selected frozen model output remains unchanged while only a compact region of the image is retained.

What are the main practical advantages claimed for SEAMS?

SEAMS is fully post hoc and model agnostic, requiring no auxiliary distractor dataset, no architecture-specific attribution mechanism, and no differentiable top-k relaxation.

Jul 16, 2026

Upload by Eden