Attention is Not All You Need - Pure Attention Loses Rank Doubly Exponentially with Depth

Aug 2, 2026

Upload by Violet

Attention-based architectures are widely used in machine learning, yet the underlying reasons for their effectiveness remain insufficiently understood. This work analyzes self-attention networks through a decomposition of their outputs into sums of smaller terms, each corresponding to a sequence of attention heads across layers. The study proves a strong inductive bias: without skip connections or MLPs, the network output converges doubly exponentially to a rank-1 matrix with uniform tokens, while skip connections and MLPs prevent degeneration. Experiments confirm the convergence behavior across variants of transformer architectures.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Abstract

Introduction

Attention Doubly Exponentially Loses Rank

Self-attention networks convergence result

View metadata, citation and similar [papers at ](papers at core.ac.uk)[core.ac.uk](papers at core.ac.uk) brought to you by CORE [provided by](provided by arXiv.org)[ arXiv.org](provided by arXiv.org) e-Print Archive Attention is not all you need: pure attention loses rank doubly exponentially with depth arXiv :2103 .03404v 1 [ cs .LG] 5 Mar 2021 Yihe Dong Google [yihed@google.com](yihed@google.com) Jean-Baptiste Cordonnier EPFL jean-baptiste.cordonnier@ep􀀍.ch Andreas Loukas EPFL andreas.loukas@ep􀀍.ch Abstract Attention-based architectures have become ubiquitous in machine learning. Yet our understanding of the reasons for their e􀀋ectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards \token uniformity". Speci􀀌cally, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to arank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identi􀀌ed convergence phenomena on di􀀋erent variants of standard transformer architectures 1 . 1 Introduction The attention mechanism [BCB15] was initially developed to better learn long-range sequential knowledge, and found e􀀋ective use in transformer networks [VSP+ 17] . Since then, attention-based architectures have permeated across data domains machine learning applications, such as in natural language processing [DCLT18], speech recognition [LZL+ 20], and computer vision [RPV+ 19 , BZV+ 19] . As such, it is vital to develop tools to understand the inner workings of transformers and attention in general, both to shed light on existing models, and to design more e􀀋ective future models. This work provides new insights about the operation and inductive bias of networks built by stacking multiple self-attention layers. Surprisingly, we 􀀌nd that pure self-attention networks (SANs), i.e., transformers with skip connections and multi-layer perceptrons (MLPs) disabled, lose expressive power doubly exponentially with respect to network depth. More speci􀀌cally, we prove that the output converges with a cubic rate to a rank one matrix that has identical rows. While we derive the convergence bounds in part by using properties of stochastic matrices, our results go beyond what one would expect based on standard results. In particular, by leveraging the cascading e􀀋ects of speci􀀌cally stacking self-attention modules, we show exponentially faster convergence than what standard theory prescribes. Furthermore, while previous studies have considered the rank of individial self-attention matrices [WLK+ 20 , KVPF20 , CLJ20a], our results are the 􀀌rst to address conditions under which the entire network converges to rank one. This raises the question, why do transformers work? Our analysis indicates that skip connections play a key role in mitigating rank collapse, and MLPs can slow down the convergence by increasing their Lipschitz constant. We characterize these counteracting forces by proving upper and lower bounds of this convergence behavior under SAN architectural variants that resemble transformers. Our results reveal a previously unknown vital utility of skip connections, beyond facilitating optimization and gradient 􀀍ow [HZRS16a, BFL + 18] . 1 Our code is publicly available at [https://github.com/twistedcubic/attention-rank-collapse](https://github.com/twistedcubic/attention-rank-collapse) Layer 1 Layer 2 Figure 1: Two paths in a deep Self-Attention Network (SAN) with H heads and L layers. At each layer, a path can go through one of the heads or bypass the layer. Adding an MLP block after each attention layer forms the transformer architecture. In the process, we develop a ne

FAQs

本文提出了如何理解自注意力网络的分析方法？

通过将自注意力网络的输出分解为多个更小项之和，每一项对应于跨层按顺序运行的注意力头序列，从而得到新的路径分解视角。

纯自注意力网络在深度增加时会发生什么现象？

在禁用跳连（skip connections）和多层感知机（MLPs）时，输出会随网络深度以双重指数速度收敛，并趋向于秩为1且所有token相同的矩阵。

跳连和MLP为何能阻止退化并影响收敛？

跳连在抑制秩塌陷方面起关键作用；MLPs会通过增大其Lipschitz常数来减慢收敛，从而阻止纯注意力导致的退化。

Aug 2, 2026

Upload by Violet