The underlying structures of self-attention - symmetry, directionality, and emergent dynamics in Transformer training

Aug 2, 2026

Upload by Eden

Self-attention is fundamental to Transformer architectures, yet the way information is encoded in self-attention matrices and how training objectives shape this encoding remains insufficiently understood. The work introduces a mathematical framework that derives the structures behind self-attention weight updates. Bidirectional training yields symmetry in weight matrices, while autoregressive training produces directionality and column dominance. Results hold across multiple Transformer models and modalities, and symmetric initialization improves encoder-only performance on language tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

AccessFree to download

Introduction

Autoregressive vs bidirectional training

Related work and analysis gaps

Autoregressive and bidirectional training leads to directional and symmetric weight updates

Interpreting self-attention with bilinear forms

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training Matteo Saponati 1 * Pascal Sager 2 1 * Pau Vilimelis Aceituno 1 Thilo Stadelmann 2 3 Benjamin Grewe 1 4 Abstract Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models —including ModernBERT, GPT, LLaMA3, and Mistral—and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models. 1. Introduction Transformer models now achieve state-of-the-art performance across a wide range of tasks and domains (Radford et al., 2019 ; Dosovitskiy et al., 2021 ; Radford et al., 2023) . Despite their success, the internal mechanisms governing their decision-making processes remain poorly understood, raising concerns regarding model alignment, reliability, and safety (Wang et al., 2023 ; Yao et al., 2024) . A key challenge *Equal contribution 1Institute of Neuroinformatics, ETH Z¨urichand University of Z¨urich, Z¨urich, Switzerland 2 Centre for Artificial Intelligence, Z¨urich University of Applied Sciences, Winterthur, Switzerland 3ECLT European Centre for Living Technology, Venice, Italy 4ETH AI Center, Z¨urich, Switzerland. Correspondence to: Matteo Saponati <[masapo@ini.ethz.ch](masapo@ini.ethz.ch) >, Benjamin Grewe <[bgrewe@ini.ethz.ch](bgrewe@ini.ethz.ch) >. Proceedings of the 42 nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025 . Copyright 2025 by the author(s) . in understanding these models is unraveling the structures of self-attention, which is essential to Transformer architectures. Current literature largely overlooks the nature of the self-attention weight matrices during autoregressive training, where the model predicts the next token in a sequence given previous ones (Radford et al., 2019 ; Black et al., 2021 ; Touvron et al., 2023) and bidirectional training, where the model predicts a missing token given the full sequence (Devlin et al., 2019 ; Bao et al., 2022 ; Warner et al., 2024) . Understanding self-attention requires answering two fundamental questions: How can we interpret the structures learned in the self-attention matrices? What is the impact of different objective functions on these matrices? Previous work used sparse auto-encoders to identify interpretable features (Huben et al., 2024 ; Bricken et al., 2023), circuit analysis to interpret Transformer components (Olah et al., 2020 ; Elhage et al., 2021 ; Olah, 2022), and techniques like the logit lens to analyze self-attention mechanisms (Geva et al., 2021 ; Dar et al., 2023) (for a detailed discussion, see Section 5) . However, these methods do not reveal the structural patterns in self-attention matrices or the transformations they encode. Crucially, how autoregressive and bidirectional training shape specific weight structures remains unclear. To address this gap, we introduce a novel framework for analyzing self-attention matrices and understanding how different objective functions define their weight updates. We then use this framework to derive understandable mathematical structures that should emerge from such updates. Finally, we verify these interpretable structures

FAQs

What problem does the paper address about self-attention matrices?

It studies how information is embedded in self-attention matrices and how different objective functions affect the structures learned during Transformer training.

How do bidirectional and autoregressive training differ in the paper’s framework?

Bidirectional training induces symmetry in the self-attention weight matrices, while autoregressive training produces directionality and column dominance.

How are the theoretical findings validated and applied?

They are tested numerically across many Transformer models and modalities, and symmetric initialization is shown to improve performance of encoder-only models on language tasks.

Aug 2, 2026

Upload by Eden