Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures

Sep 4, 2026

We investigate whether different large language model (LLM) architectures encode high-level concepts in structurally compatible ways. Across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer, revealing a geometric-functional universality dissociation. Using contrastive-difference CKA (CKA∆), a training-free diagnostic on per-sample contrastive differences, the method isolates concept-specific convergence from generic similarity, enabling significant discrimination where standard CKA fails and supporting practical cross-architecture concept monitoring.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Motivation for zero-shot concept monitoring across architectures

Limits of standard activation similarity and CKA

Contributions and proposed diagnostic

Geometric-functional universality dissociation

Contrastive-difference CKA (CKA∆) as a training-free diagnostic

arXiv :2606 . 16897v1 [ cs .CL] 15 Jun 2026 Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures Xueping Gao [hellogxp@gmail.com](hellogxp@gmail.com) Alibaba Cloud Abstract Do different LLM architectures encode high-level concepts in structurally compatible ways? We systematically characterize a geometric-functional universality dissociation: across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Using contrastive-difference CKA (CKA∆ ), a trainingfree diagnostic that computes kernel alignment on per-sample contrastive differences, we isolate concept-specific convergence from generic similarity—achieving significant discrimination where standard CKA cannot. The dissociation replicates across all six concept domains we test (five with p ≤ 0.017 geometric discrimination and safety as a converging-functional trend, p = 0 .08), including two non-instruction concepts (code-vs-NL, reasoning-vs-recall) validated without system prompts; a single 70B–70B pair provides an observational note that universality may strengthen with scale, requiring replication with additional ≥70B models. We position CKA∆ as a practical regime classifier and architectural outlier detector (Gemma: d = 1 .08, AUC = 0 . 79) rather than an absolute transfer-accuracy predictor, providing a training-free diagnostic for cross-architecture concept monitoring. Keywords: Cross-architecture representation similarity; contrastive-difference CKA; persona vectors; concept transfer; mechanistic interpretability 1. Introduction Consider a safety filter trained on Llama-3.1 to detect harmful outputs. Can it transfer to Qwen-2.5 without retraining? If different architectures encode safety concepts in structurally compatible ways, such zero-shot transfer becomes possible—enabling scalable alignment monitoring across heterogeneous model portfolios. As organizations deploy multiple LLMfamilies—Llama, Qwen, Gemma, Mistral, and others—understanding whether concept monitoring tools generalize across architectures is critical for scalable alignment. A persona detector trained on one model should ideally transfer to others without retraining; alignment interventions should be portable across model families. But do different architectures actually encode high-level concepts in structurally compatible ways? Recent work in mechanistic interpretability has established that LLMs develop linear representations for high-level concepts including truthfulness (Marks and Tegmark, 2024), sentiment (Tigges et al. , 2023), and factual knowledge (Nanda et al. , 2023) . The linear representation hypothesis (Park et al. , 2023) suggests that semantically meaningful directions exist in transformer residual streams. If persona dimensions are linearly encoded, the critical question becomes: do different architectures converge on similar representational strategies? This question connects directly to language modeling: concept representations modulate the next-token prediction distribution—an extraverted persona shifts probability mass toward enthusiastic completions, a safety-aligned model suppresses harmful continuations, © 2026 X. Gao. Gao and a formal register selects academic vocabulary. If different architectures converge on structurally compatible concept representations, this reveals a fundamental property of how language models learn to condition generation on high-level concepts, and enables portable alignment tools—a persona monitor trained on Llama could detect sycophancy in Qwen without retraining. Understanding whether this convergence is concept-specific or merely reflects generic representational similarity thus contributes to language-modeling theory and has direct practical implications for scalable deployment. A key methodological insight is that raw activation similarity is insufficient for measuring concept-specific co

FAQs

What question does the paper address about LLM architectures?

The paper asks whether different LLM architectures encode high-level concepts in structurally compatible ways, enabling zero-shot concept transfer across models.

Why is standard CKA insufficient for this task?

Standard CKA on positive-pole activations captures generic representational similarity and cannot reliably distinguish same-trait from cross-trait comparisons, masking concept-specific convergence.

How does contrastive-difference CKA (CKA∆) work conceptually?

CKA∆ computes kernel alignment on per-sample contrastive differences, isolating concept-specific convergence while filtering out generic similarity, and it is training-free.

Sep 4, 2026

Upload by Sophia Brooks