From Self Attention to Connection Laplacian: A Unified Operator View of Transformers

Jul 16, 2026

Upload by Rowan

Self-attention is treated as an operator-level geometric object for sequence models, where token features form a vector field over a token-position graph and attention operates as a connection walk. Messages are aggregated by a nonnegative walk matrix and transported along edges through learned linear maps. Single-head attention becomes an exact connection propagation step with constant transport, while multi-head attention becomes an edge-dependent walk with attention-gated headwise transports. The generator reduces to a random-walk connection Laplacian under specific stochastic, reversible, and metric-compatible conditions. Empirically, Transformers from 124M to 8B exhibit stable geometric operators in deeper layers and self-organized near scaled isometries, linking attention to classical geometric Laplacians.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages29

Reading Time1 h 13 min

Introduction

Related work

arXiv :2607 . 10677v 1 [ cs .LG] 12 Jul 2026 From Self-Attention to Connection Laplacian From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers Binbin Lin [binbinlin@zju.edu.cn](binbinlin@zju.edu.cn) School of Software Technology, Zhejiang University, China Wei Chen [weichen.cw@zju.edu.cn](weichen.cw@zju.edu.cn) College of Computer Science and Technology, Zhejiang University, China Yalun Li [yalunli@zju.edu.cn](yalunli@zju.edu.cn) College of Computer Science and Technology, Zhejiang University, China Wenxiao Wang [wenxiaowang@zju.edu.cn](wenxiaowang@zju.edu.cn) School of Software Technology, Zhejiang University, China Jieping Ye [yejieping.ye@alibaba-inc.com](yejieping.ye@alibaba-inc.com) Alibaba Cloud, China Xiaofei He [xiaofeihe@cad.zju.edu.cn](xiaofeihe@cad.zju.edu.cn) College of Computer Science and Technology, Zhejiang University, China Abstract Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a vector field over the token-position graph and identify attention as a connection walk: messages are aggregated by a nonnegative walk matrix while being transported along each edge by a learned linear map. Within this framework, we prove that single-head attention (SHA) is exactly a connection propagation step with constant transport, and that multi-head attention (MHA) is exactly a single edge-dependent connection walk whose effective transport is an attentiongated mixture of headwise transports. We further clarify the conditions under which the corresponding generator reduces to a random-walk connection Laplacian, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory: effective attention graphs converge to stable geometric operators in deeper layers, learned transports self-organize into approximate scaled isometries, and both phenomena strengthen consistently with scale. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective. Keywords: connection Laplacian, self-attention, multi-head attention, transformers, geometric deep learning 1 Introduction Transformers and their self-attention mechanism (Vaswani et al., 2017) have reshaped modern deep learning by enabling models to capture long-range dependencies across tokens. In self-attention, each token attends to other tokens through a learned similarity measure, producing a weighted aggregation of value vectors. The resulting attention matrix can be interpreted as a directed weighted graph where edges encode information flow. This paper Lin, Chen, Li, Wang, Ye and He gives an operator-level geometric interpretation of self-attention: token representations forma vector field over the token-position graph (the token graph, for short), and attention acts as a connection walk that mixes tokens while transporting features along edges. Developing this operator view, we clarify the operator structure underlying multi-head attention (MHA), the depthwise behavior, and the geometry of token interactions. 1.1 Related work Existing literature examines attention through various theoretical lenses, which we categorize into graph, tensor, dynamical, energy, kernel, and geometric perspectives. Graph operator view. Self-attention can be viewed as a data-dependent linear operator that mixes token features via the attention matrix. This formulation aligns with graph transformers, where full attention implies all-to-all communication and sparse attention recovers local aggregation (Yun et al., 2019) . Analytic approaches often symmetrize or normalize

FAQs

How does the paper interpret self-attention geometrically at the operator level?

It models tokens as a vector field over a token-position graph and interprets attention as a connection walk that aggregates messages and transports features along edges using learned linear maps.

What is the difference between the unified view of single-head attention and multi-head attention?

Single-head attention is shown to be exactly a connection propagation step with constant transport, while multi-head attention corresponds to an edge-dependent connection walk whose transport is an attention-gated mixture of headwise transports.

When does the generator reduce to a random-walk connection Laplacian?

It reduces under conditions involving stochasticity, reversibility, and metric-compatible transports, which the paper uses to identify the operator relationship to random-walk-style Laplacians.

Jul 16, 2026

Upload by Rowan