Efficient Knowledge Injection in LLMs via Self-Distillation - Self-distillation for internalizing new factual knowledge

Aug 2, 2026

Large language models often need new factual knowledge that is missing from pre-training data, and current knowledge-injection approaches rely mainly on supervised fine-tuning or retrieval-augmented generation (RAG). This work introduces prompt distillation, a self-distillation method that internalizes knowledge from free-form documents without requiring larger teacher models or structured knowledge formats. Experiments across multiple LLM sizes and families show prompt distillation outperforms standard supervised fine-tuning and can surpass RAG. The study further analyzes why it works and how performance scales with training fact coverage.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages43

Reading Time1 h 48 min

AccessFree to download

Abstract

Introduction

Knowledge injection strategies: fine-tuning vs RAG

Why supervised fine-tuning underperforms

Paper goal

This is an electronic reprint of the original article. This reprint may differ from the original in pagination and typographic detail. Kujanpää, Kalle; Marttinen, Pekka; Valpola, Harri; Ilin, Alexander Efficient Knowledge Injection in LLMs via Self-Distillation Published in: Transactions on Machine Learning Research Published: 01/01/2025 Document Version Peer-reviewed accepted author manuscript, also known as Final accepted manuscript or Post-print Published under the following license: CC BY Please cite the original version: Kujanpää, K. , Marttinen, P. , Valpola, H. , & Ilin, A. (2025) . Efficient Knowledge Injection in LLMs via SelfDistillation. Transactions on Machine Learning Research, 2025(August), 1-42. [https://openreview.net/forum?id=drYpdSnRJk](https://openreview.net/forum?id=drYpdSnRJk) This material is protected by copyright and other intellectual property rights, and duplication or sale of all or part of any of the repository collections is not permitted, except that material may be duplicated by you foryour research use or educational purposes in electronic or print form. You must obtain permission for anyother use. Electronic or print copies may not be offered, whether for sale or otherwise to anyone who is not an authorised user. Efficient Knowledge Injection in LLMs via Self-Distillation Kalle Kujanpää Pekka Marttinen Department of Computer Science, Aalto University Finnish Center for Artificial Intelligence (FCAI) [kalle.kujanpaa@aalto.fi](kalle.kujanpaa@aalto.fi)[ ](kalle.kujanpaa@aalto.fi)[pekka.marttinen@aalto.fi](pekka.marttinen@aalto.fi) Harri Valpola Alexander Ilin System 2 AI [harri@system2ai. com](harri@system2ai. com)[ ](harri@system2ai. com)[alexilin@system2ai. com](alexilin@system2ai. com) Reviewed on OpenReview: [https: // openreview. net/ forum? id= drYpdSnRJk](https: // openreview. net/ forum? id= drYpdSnRJk) Abstract In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG) . Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillationbased method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation’s effectiveness and examine how it scales. 1 Correctness (%) 92 90 88 86 84 82 80 78 | | SFT
PD
RAG
PD + RAG
PD (XL) | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Figure 1: Average accuracy of responses across evaluation datasets. The proposed prompt distillation (PD) method performs competitively with RAG and surpasses supervised fine-tuning (SFT) . Combining prompt distillation with RAG (PD+RAG) improves performance over standard RAG. Scaling up the fact coverage in PD training (PD XL) further enhances performance, outperforming all other methods. 1 Code available at [https://github.com/kallekku/prompt-distillation](https://github.com/kallekku/prompt-distillation) 1 Introduction In many practical scenarios, large language models must integrate new, domain-specific factual knowledge absent from their ori

FAQs

What problem does the paper address in large language models?

It addresses how LLMs acquire new domain-specific factual knowledge when that knowledge is not present in their pre-training corpus.

How does prompt distillation differ from standard supervised fine-tuning and RAG?

Prompt distillation internalizes knowledge via a self-distillation approach and does not require larger teacher models or structured knowledge formats, while standard SFT depends on supervised question-answer targets and RAG relies on an external knowledge base.

What are the main empirical results of the proposed method?

Across multiple model sizes and families, prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. Combining prompt distillation with RAG improves performance further, and scaling fact coverage in PD training increases gains.

Aug 2, 2026

Upload by Seraphina