Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholds

Q: Does the feature-norm threshold act as a passive indicator or an active dynamical target?

It acts as an attractor of the gradient flow: after rescaling the feature norm over a tenfold range, it relaxes back to the same fn* value.

Jul 16, 2026

Upload by Olivia Brown

Neural collapse (NC) describes how penultimate-layer features converge to a simplex equiangular tight frame, with equilibrium behaviour well understood but onset timing largely unexplored. This study identifies a single measurable control: the mean penultimate feature norm fn reaching a pair-specific critical threshold fn*. Across settings, fn* concentrates tightly and is mostly invariant, while training hyperparameters mainly affect the approach rate and activation shifts the threshold itself.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages31

Reading Time1 h 18 min

Introduction

Neural collapse and the onset-time question

Experimental design and hypotheses

Core findings and decomposition

Feature-norm threshold behaviour

Attractor interpretation via rescaling

Architecture–dataset interaction

arXiv :2604 .00230v 3 [ cs .LG] 10 Jul 2026 Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholds Anamika Paul Rupa∗ Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA Abstract Neural collapse (NC), the convergence of penultimate-layer features to a simplex equiangular tight frame, is well characterised at equilibrium, but what governs when and how fast it emerges has received little systematic study. We present a controlled study of NC onset, showing it is marked by a single measurable quantity: the mean penultimate feature norm (fn) reaching a critical threshold fn ∗ specific to each (model, dataset) pair. Three findings support this. First, fn∗ concentrates tightly within each pair (coefficient of variation below 8% at a fixed collapse criterion) and is largely invariant to depth, width, and weight decay, which set only the rate at which fn approaches it, whereas activation shifts the threshold itself. Second, a direct intervention establishes fn ∗ as an attractor of the gradient flow, not a passive correlate: after the feature norm is rescaled over a tenfold range, it relaxes to the same value. Third, a six-cell (architecture) × (dataset) grid reveals a non-additive interaction: the architecture effect on fn ∗ ranges from +22 .7% to +458% depending on the dataset (confirmed by a matched-optimiser control), so fn ∗ must be measured per pair, not extrapolated. A simplified unconstrained-features calculation rationalises the magnitude offn∗ and its insensitivity to weight decay. The temporal ordering is modest: fn crosses fn ∗ just before collapse but co-evolves with it, so fn ∗ is a stable landmark of the collapsed state, not an early-warning signal. Together, these results establish the feature-norm threshold asa reproducible, largely training-invariant characterisation of neural collapse onset. Keywords: Neural collapse, Training dynamics, Feature norm, Deep learning, Weight decay, Implicit bias ∗ Corresponding author. Email address: [anamikal.rupa@howard.edu](anamikal.rupa@howard.edu) (Anamika Paul Rupa) Graphical abstract Neural collapse onset coincides with a feature-norm threshold f feature norm fn ( log ) 101 100 (A) Feature norm falls to a pair-specific f P1 P2 training conditions Neural set the timing collapse threshold f⋆ ≈ 1.06 n 0 50 100 150 200 250 300 training epoch feature norm fn ( log ) 101 100 (B) Rescaled norms self-correct to f | | | | --- | --- | | | | 200 300 400 500 training epoch ✔ Training factors (depth, width, weight decay) set when collapse occurs, not f⋆n ✔ Each (model, dataset) pair has a stable f⋆n (within-pair CV < 8%) ✔ Rescaled feature norms return to the same f⋆n: a gradient-flow attractor Scope: MLPs & ResNets ⋅ MNIST / Fashion-MNIST / CIFAR-10 ⋅ robust across seeds and optimizers (SGD, Adam) 1. Introduction Neural collapse (NC), first documented by Papyan et al. [1], is a striking geometric regularity that emerges at the end of deep network training: penultimate-layer features converge to a simplex equiangular tight frame (ETF), within-class variability vanishes (NC1), and classifier weights align with class means (NC3) . The endpoint of NC is now well characterised theoretically [2, 3, 4, 5], yet a fundamental question has received almost no systematic attention: what controls when and how fast NC emerges as a function of architecture and training hyperparameters? We answer this empirically: across architectures, datasets, and hyperparameters, NC onset is consistently marked by the mean feature norm reaching a model-dataset-specific threshold fn∗ that is largely invariant to training dynamics, is reached just before collapse, and behaves as an attractor of the gradient flow, reducing a qualitative dynamics question to a single trackable quantity. Understanding collapse timing matters scientifically, by clarifying how represen

FAQs

Does the feature-norm threshold act as a passive indicator or an active dynamical target?

It acts as an attractor of the gradient flow: after rescaling the feature norm over a tenfold range, it relaxes back to the same fn* value.

Jul 16, 2026

Upload by Olivia Brown