Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformers

Jul 16, 2026

Uploaded by Theodore

Vision Transformers (ViTs) are challenging to interpret because relevance propagation and attention-flow methods often ignore architectural realities like heterogeneous attention-head importance and residual (skip) connections. Existing techniques frequently assume uniform head contributions and treat skip paths as identities, which causes misassigned relevance. GradSkip introduces adaptive head weighting and skip-aware propagation, redistributing relevance between attention and residual pathways. Experiments on ImageNet1K and BloodMNIST yield state-of-the-art faithfulness with over 14× fewer GFLOPs, and segmentation evaluations improve localization against ground-truth regions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages29

Reading Time1 h 13 min

AccessFree to download

Introduction

Explainability approaches for ViTs

Limitations from attention-head uniformity and residual paths

GradSkip contributions

arXiv :2607 . 10365v1 [ cs .CV] 11 Jul 2026 Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformers Christopher Buratti 1 , Michele Marchetti 1 ,Federica Parlapiano 1 , Davide Traini 1 ,2 , Domenico Ursino 1 , and Luca Virgili 1∗ 1 DII, Polytechnic University of Marche 2 CHIMOMO, University of Modena and Reggio Emilia ∗ Contact Author [c.buratti@pm.univpm.it](c.buratti@pm.univpm.it) ; [michele.marchetti@univpm.it](michele.marchetti@univpm.it) ; [f.parlapiano@pm.univpm.it](f.parlapiano@pm.univpm.it) ; [davide.traini@unimore.it](davide.traini@unimore.it) ; [d.ursino@univpm.it](d.ursino@univpm.it) ; [luca.virgili@univpm.it](luca.virgili@univpm.it) Abstract Vision Transformers (ViTs) are difficult to interpret because current methods of relevance propagation and attention flow do not fully consider some key architectural features, such as the uneven importance of attention heads and residual connections. Prior approaches typically assume uniform importance across attention heads; furthermore, they model skip connections as identity paths, leading to inaccurate relevance attribution. To address these issues, we introduce GradSkip, a novel relevance propagation method for ViTs based on adaptive head weighting and skip-aware propagation. GradSkip models the different importance of the attention heads and dynamically distributes relevance between the attention and residual paths. Experiments on ImageNet1K and BloodMNIST demonstrate a state-of-the-art faithfulness of GradSkip while requiring over 14 times fewer GFLOPs than the best-performing existing approaches. Additional evaluations using transformerbased segmentation confirm improved localization and alignment with ground-truth regions. Keywords: Vision Transformers; Explainable AI; Gradient based Explanation; Relevance Propagation Method; Computer Vision 1 Introduction Vision Transformers (ViTs) have emerged as a powerful alternative to Convolutional Neural Networks (CNNs) for many computer vision tasks, achieving state-of-the-art results on several benchmarks [41] . Unlike CNNs, which rely on local receptive fields, ViTs process images as sequences of patch tokensand use self-attention to capture long-range dependencies [38, 16] . While this design improves accuracy and scalability, it introduces challenges regarding interpretability. It is therefore crucial to identify which image regions influence a model’s predictions in order to make it possible to trust, debug, and understand learned representations. This need is particularly important in high-stakes fields, such as medical imaging, where understanding the rationale behind a prediction is essential for clinical trust and accountability, and where an opaque decision can have serious consequences. ViT explainability methods can be categorized into three approaches: gradient-based, attentionbased, and perturbation-based. Gradient-based approaches estimate feature importance using backpropagated gradients [32, 34, 4] . Attention-based approaches exploit self-attention weights to trace information flow across layers [19, 17] . Perturbation-based approaches evaluate feature relevance by masking or modifying input regions and measuring the impact on model predictions [14, 43, 29] . While these approaches provide useful insights, they have limitations. These include unstable gradient attributions (for gradient-based approaches), limited causal reliability (for attention-based approaches), and high computational costs (for perturbation-based approaches) . Furthermore, most approaches treat attention heads uniformly and overlook residual connections, which hinders accurate relevance propagation in transformer architectures. Fundamentally, these limitations stem from a shared issue in the propagation of relevance through the two mechanisms that define a transformer block, i.e. , multi-head self-attention and residual connections [21] . Attention heads are known to pl

FAQs

Why are current ViT explainability methods insufficient for relevance propagation?

They often assume uniform importance across attention heads and model skip connections as identity paths, which blurs how much relevance comes from learned transformations versus residual preservation.

What are the two main ideas behind GradSkip?

GradSkip uses adaptive attention-head weighting combining gradient-based flow metrics with Gini-based sparsity, and it performs skip-connection-aware relevance propagation that explicitly models attention and residual paths during backward computation.

How does GradSkip perform in experiments and what efficiency gain does it claim?

On ImageNet1K and BloodMNIST it reports state-of-the-art faithfulness while using over 14× fewer GFLOPs than the best existing methods, and segmentation evaluations show improved localization and alignment with ground-truth regions.

Jul 16, 2026

Uploaded by Theodore