Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Jul 16, 2026

Upload by Aurelia

Fine-tuning large language models to inject new knowledge often triggers a critical mismatch: models memorize fresh facts quickly, yet fail to use them in downstream reasoning. The work formalizes this behavior as the Knowing–Using Gap, capturing both an accuracy gap and a temporal lag between memorization and generalization. Using fine-tuned LLMs with unseen knowledge, it introduces self-patching to map internal knowledge permeation dynamics. Results support a knowledge-circuit misalignment hypothesis and a heuristic recovery strategy recovering 58–75% of oracle headroom, with cross-domain robustness checks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

Abstract

Introduction

Knowing–Using Gap

Self-patching and spatial permeation mapping

Knowledge–circuit misalignment hypothesis

arXiv :2607 .08393v 1 [ cs .AI] 9 Jul 2026 Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning Lu Dai2,1 Ziyang Rao1 Yili Wang1 Hanqing Wang1 Hao Liu1,2 Hui Xiong1,2 1HKUST(GZ) 2HKUST [ldaiae@connect.ust.hk {liuh](ldaiae@connect.ust.hk {liuh) ,[xionghui}@ust.hk](xionghui}@ust.hk) Abstract Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the Knowing–Using Gap, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58–75% of the oracle headroom in generalization failure. Experiments are done cross-domain for the robustness of this finding. Code and data are open at [https://anonymous.4open.science/r/Mem2Gen-71FF](https://anonymous.4open.science/r/Mem2Gen-71FF). 1 Introduction Large language models (LLMs) excel at varieties of tasks but face significant challenges in adapting to unseen information, necessitating effective methods for post-training knowledge updates. While there are approaches like retrieval-augmented generation (RAG) and knowledge editing Meng et al., Gupta et al. [2024], fine-tuning remains a fundamental paradigm for knowledge updating, as it not only operates on parametric memory end-to-end but also injects knowledge in a way that can be reused by the model’s existing reasoning capabilities. Despite sufficient capacity to fit new data Morris et al. [2025], Allen-Zhu and Li, LLMs exhibit a“remembering but not using” failure Ovadia et al. [2024], Soudani et al. [2024], Zhong et al. [2023], Cohen et al. [2024], Berglund et [al. as](al. as) shown in Figure 1: models can memorize new facts (e.g.,“Sydney is located in [Entity]”) but fail to reliably use them in downstream reasoning (e.g.,“The capital of the country Sydney located is . . .”) Zhong et al. [2023], Cohen et al. [2024], Yao et al.[2025], creating a gap between simple memorization and flexible generalization. We term this phenomenon the Knowing–Using Gap, characterized by two distinct disparities: 1) an accuracy gap, where generalization accuracy remains significantly lower than memorization; and 2) a temporal lag, where generalization emerges significantly later after remembering. This observation raises fundamental research questions on the mechanics of fine-tuning: Once a fact is memorized, when and why does it become accessible to the model’s existing reasoning circuits? Preprint. (b) Conceptual illustration: LLM soon learn to memorize two new atomic facts but fail to generalize to reasoning questions (a) Knowing–Using Gap: In a real fine-tuning dynamics above, a significant accuracy and temporal gap is shown between memorization and chaining. (c) Saturation Epochs: Finetuning 1000 cases, the mean and std of epochs where memorization and generalization happens Figure 1: Illustration of the Knowing–Using Gap. To address these questions, we conduct a fine-grained analysis of the training dynamics during knowledge injection. We construct datasets from two real-world domain knowledge bases and eliminate overlap with pretraining. We define two types of reasoning QA tasks to evaluate how LLMs generalize the learned knowledge: chaining task requires resolving a bridge ent

FAQs

What problem does the paper focus on in large language model fine-tuning?

It focuses on why fine-tuned models can memorize new facts yet fail to generalize those facts to downstream reasoning tasks.

How does the paper define the Knowing–Using Gap?

It defines the Knowing–Using Gap as an accuracy gap and a temporal lag between when facts are memorized and when they become usable for generalization.

What is self-patching, and what does it help reveal?

Self-patching is an intervention technique that relocates internal hidden representations from a source run into a target run to measure changes in correct-answer probability. It reveals which layers and positions contain representations that can enable generalization after memorization.

Jul 16, 2026

Upload by Aurelia