How are linear representations learned Exact solutions to the dynamics of abstraction

Jul 16, 2026

Upload by Miles

Linear directions in representation space often encode concepts in artificial and biological neural networks, motivating the linear representation hypothesis used by many interpretability and control methods. This work develops a dynamical framework for how concept directions align during training, termed “abstraction.” In a minimal linear network, exact solutions give the full abstraction trajectory and show how data/target geometry, depth, and initialization scale shape end-of-learning abstraction. For nonlinear networks, erf matches the linear theory, while ReLU depends more on input geometry and exhibits an attenuation law, validated in public models and applied to improve linear probe generalization in LLMs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages81

Reading Time3 h 24 min

Introduction

Linear representation hypothesis and abstraction

Motivation and central questions

Main contributions

arXiv :2607 .08843v 1 [ cs .LG] 9 Jul 2026 How are linear representations learned? Exact solutions to the dynamics of abstraction William W. Yang 1 Andrew M. Saxe∗ , 1 ,2 Peter E. Latham∗ , 1 Abstract In artificial and biological neural networks, concepts are often encoded as consistent linear directions in representation space. In deep learning, this idea is known as the linear representation hypothesis and underpins many interpretability and control methods based on linear probes, from concept detection to activation steering. Yet while prior work has studied whether such directions should exist after training, the dynamics of how they emerge during training remain poorly understood. Here, we develop a framework to study the alignment of concept directions during training – a process we call “abstraction”. In a minimal linear network setting, we obtain exact solutions for the full trajectory of abstraction. These solutions reveal key analytic principles governing abstraction: (i) data and target geometry jointly determine abstraction at the end-of-learning,(ii) abstraction improves with network depth, and (iii) initialization scale controls the maximum abstraction reached during training. Extending our theory to nonlinear networks, we analyze how the choice of nonlinearity affects abstraction dynamics: erf networks approximate the linear theory, while abstraction in ReLU networks depends less on target geometry and more on input geometry. Across both, we prove a striking attenuation law: both nonlinearities weaken abstraction in activations relative to preactivations. We find evidence for this law in open models (DINOv3, Gemma 4) and apply our theory to improve linear probe generalization in LLMs. Together, our results provide a dynamical theory of abstraction with implications for interpretability and control. 1 Introduction A recurring empirical observation in deep neural networks is that high-level concepts often behave like approximately linear directions in representation space [1, 2] . This idea has recently been formalized as the linear representation hypothesis (LRH) [3, 4, 5] . To use a classical example, alinear representation of the “gender” concept would imply that the concept vectors vking − vqueen and vman − vwoman are approximately parallel. A closely related idea exists in neuroscience, where a concept is said to be represented in an “abstract” format when concept vectors are highly aligned across contexts [6, 7, 8, 9, 10, 11, 12, 13] . Borrowing that neuroscience terminology, we use the term “abstraction” to refer to the alignment of concept vectors (e.g. the cosine similarity between vking − vqueen and vman − vwoman ) during training. Understanding how abstract representations arise is important in both deep learning and neuroscience. Many AI interpretability and control methods assume existence of abstract representations by using linear probes to extract concept directions for interpretability and detection [14, 15] or model steering [16, 17] . In neuroscience, abstraction is thought to support the brain’s ability to adapt to changing environments via out-of-distribution and compositional generalization [6, 10, 18] . *Co-senior authors; equal contribution. 1 Gatsby Computational Neuroscience Unit, University College London, London W1T 4JG, United Kingdom. 2 Sainsbury Wellcome Centre, University College London, London W1T 4JG, United Kingdom. Preprint. However, our understanding of how abstraction arises remains limited. Previous theoretical work has argued that abstraction becomes perfect (i.e. cosine similarity = 1) once trained to convergence [4], or at the global minima of the loss landscape [19] . This leaves open a simple question: what are the dynamics of abstraction during training? Empirically, in real-world settings abstraction often exhibits nontrivial dynamics, can be non-monotonic, and seldom reach perfect abstraction as predicted in simple theories; for example, see

FAQs

What does the paper mean by “abstraction” during training?

Abstraction refers to the alignment of concept vectors over training, measured for example by cosine similarity between concept-difference directions across contexts.

What does the paper’s minimal linear network analysis provide?

It derives exact solutions for the full trajectory of abstraction during training, identifying how data/target geometry, network depth, and initialization scale determine abstraction outcomes.

How do erf and ReLU networks differ in abstraction dynamics?

The theory indicates erf networks approximate the linear case, while ReLU abstraction depends less on target geometry and more on input geometry. The paper also proves that nonlinearities attenuate abstraction in activations relative to preactivations.

Jul 16, 2026

Upload by Miles