Cross-seed explainability using Procrustes conditioned Joint End-to-end Top-K Sparse Autoencoders

Jul 16, 2026

A Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) extracts cross-seed universal features from independently trained BERT models, tackling mechanistic interpretability issues caused by non-convex dictionary learning and geometric misalignment. The method computes an orthogonal Procrustes rotation between seeds’ activation spaces prior to joint SAE training, combining Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss. Across five seed pairs on SST-2, Stanford Politeness, and TweetEval Emotion, the full pipeline yields more universal features (Pearson r ≥ 0.70) than post-hoc alignment baselines. Qualitative analysis links high-universality features to interpretable sociolinguistic patterns.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Introduction

Contributions

Methodology

arXiv :2607 .08499v 1 [ cs .CL] 9 Jul 2026 Cross-seed explainability using Procrustes conditioned Joint End-to-end Top-K Sparse Autoencoders Bendeg´uz V´aradi 1,2 and Zolt´an Kmetty 1,2 1 Centre for Social Sciences, CSS-RECENS Research Group, Budapest 1097, Hungary. 2 Department of Sociology, Faculty of Social Sciences, E¨otv¨os Lor´and University, Budapest, Hungary. Abstract We present a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) for extracting cross-seed universal features from independently trained BERT models. Cross-seed feature universality is a fundamental challenge in mechanistic interpretability: because dictionary learning is non-convex, independently trained networks learn misaligned feature spaces, so apparently identical features may differ by random initialization. We address this by computing an orthogonal Procrustes rotation between seeds’ activation spaces before joint SAE training, combining Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss based on previous SAE literature. Evaluating on five independent seed pairs (ten BERT models) across three benchmark datasets (SST-2, Stanford Politeness, TweetEval Emotion), our full pipeline produces more universal features (Pearson r ≥ 0.70 across seeds) than post-hoc alignment baselines on all three datasets. A minimal qualitative analysis confirms that high-universality features encode interpretable sociolinguistic patterns. Keywords: Sparse Autoencoders, XAI, BERT, mechanistic interpretability 1 Introduction In recent years, a large body of research has been published on the interpretability of language models [1] . Much of the recent literature aims to solve two fundamental issues: addressing polysemanticity [2–6] and improving the structural fidelity and functional 1 robustness of the extracted features [4, 7 , 8] . In this paper, we propose an End-toEnd pipeline that combines recent novel methodologies to enhance the reliability of extracting interpretable concepts from BERT models. To demonstrate the feature universalities, we use three benchmark corpora to measure the feature correlations. Sparse Autoencoders (SAEs) have emerged at the forefront of mechanistic interpretability for disentangling polysemantic representations in large language models [1] . Recent advancements have rapidly improved the structural fidelity of these extracted features. TopK SAEs [4] eliminate the shrinkage bias inherent in L1 regularization, End-to-End SAEs [7] prevent feature splitting by optimizing for downstream consistency, and Orthogonal SAEs [9](under review at the time of citation) mitigate feature absorption by applying competition-aware orthogonality constraints to disentangle coactivating concepts. Despite these single-model improvements, achieving cross-seed”universal interpretability” [10] remains a fundamental challenge. Because dictionary learning is non-convex, independently trained networks, even those sharing the same architecture and data, suffer from feature splitting, where identical semantic concepts map to entirely different latent dimensions. Recent solutions, such as the Feature Aligned SAE [8], attempt to mitigate this by training multiple SAEs in parallel with a Mutual Feature Regularization (MFR) penalty to encourage decoder similarity. However, relying solely on training penalties may fail to address the underlying geometric misalignment of the models’ native activation spaces. In this paper, we present a novel architecture that directly solves this spatial misalignment through a Procrustesconditioned Joint End-to-End Top-K Sparse Autoencoder. Rather than penalizing separate dictionaries to force alignment, we calculate an Orthogonal Procrustes rotation matrix to superimpose the activation space of different model seeds before extracting concepts with a single, joint SAE. The Top-K activation enforces sparsity structurally, avoiding the L1 shrinkage bias, the P

FAQs

What problem does cross-seed universal interpretability address in BERT models?

Independently trained networks learn misaligned feature spaces due to non-convex dictionary learning, so the same semantic concepts can map to different latent dimensions. This makes apparently identical features non-universal across random initializations.

How does the proposed method use Procrustes conditioning during training?

An orthogonal Procrustes rotation matrix is computed to align the activation spaces of different seeds before training a single joint SAE. This avoids forcing alignment only through training penalties and targets geometric misalignment directly.

What empirical results demonstrate the effectiveness of the full pipeline?

Evaluations on five independent seed pairs (ten BERT models) across SST-2, Stanford Politeness, and TweetEval Emotion show more universal features than post-hoc alignment baselines, with Pearson correlation r ≥ 0.70 across seeds. High-universality features also correspond to interpretable sociolinguistic patterns in qualitative analysis.

Jul 16, 2026

Upload by Valentina