[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117627-en":3,"doc-seo-117627-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117627,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","Efficient Knowledge Injection in LLMs via Self-Distillation - Self-distillation for internalizing new factual knowledge","Large language models often need new factual knowledge that is missing from pre-training data, and current knowledge-injection approaches rely mainly on supervised fine-tuning or retrieval-augmented generation (RAG). This work introduces prompt distillation, a self-distillation method that internalizes knowledge from free-form documents without requiring larger teacher models or structured knowledge formats. Experiments across multiple LLM sizes and families show prompt distillation outperforms standard supervised fine-tuning and can surpass RAG. The study further analyzes why it works and how performance scales with training fact coverage.","This is an electronic reprint of the original article.  \nThis reprint may differ from the original in pagination and typographic detail.  \nKujanpää, Kalle; Marttinen, Pekka; Valpola, Harri; Ilin, Alexander  \nEfficient Knowledge Injection in LLMs via Self-Distillation  \nPublished in:  \nTransactions on Machine Learning Research  \nPublished: 01/01/2025  \nDocument Version  \nPeer-reviewed accepted author manuscript, also known as Final accepted manuscript or Post-print  \nPublished under the following license:  \nCC BY  \nPlease cite the original version:  \nKujanpää, K. , Marttinen, P. , Valpola, H. , & Ilin, A. (2025) . Efficient Knowledge Injection in LLMs via SelfDistillation. Transactions on Machine Learning Research, 2025(August), 1-42.  \n[https://openreview.net/forum?id=drYpdSnRJk](https://openreview.net/forum?id=drYpdSnRJk)  \nThis material is protected by copyright and other intellectual property rights, and duplication or sale of all or part of any of the repository collections is not permitted, except that material may be duplicated by you foryour research use or educational purposes in electronic or print form. You must obtain permission for anyother use. Electronic or print copies may not be offered, whether for sale or otherwise to anyone who is not an authorised user.  \nEfficient Knowledge Injection in LLMs via Self-Distillation  \nKalle Kujanpää Pekka Marttinen  \nDepartment of Computer Science, Aalto University Finnish Center for Artificial Intelligence (FCAI)  \n[kalle.kujanpaa@aalto.fi](kalle.kujanpaa@aalto.fi)[ ](kalle.kujanpaa@aalto.fi)[pekka.marttinen@aalto.fi](pekka.marttinen@aalto.fi)  \nHarri Valpola Alexander Ilin  \nSystem 2 AI  \n[harri@system2ai. com](harri@system2ai. com)[ ](harri@system2ai. com)[alexilin@system2ai. com](alexilin@system2ai. com)  \nReviewed on OpenReview: [https: // openreview. net/ forum? id= drYpdSnRJk](https: // openreview. net/ forum? id= drYpdSnRJk)  \nAbstract  \nIn many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG) . Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillationbased method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation’s effectiveness and examine how it scales. 1  \nCorrectness (%)  \n92  \n90  \n88  \n86  \n84  \n82  \n80  \n78  \n\n|  |  SFT\u003Cbr> PD\u003Cbr> RAG\u003Cbr> PD + RAG\u003Cbr> PD (XL) |  |  |  |  |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  |  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |  |\n\nFigure 1: Average accuracy of responses across evaluation datasets. The proposed prompt distillation (PD) method performs competitively with RAG and surpasses supervised fine-tuning (SFT) . Combining prompt distillation with RAG (PD+RAG) improves performance over standard RAG. Scaling up the fact coverage in PD training (PD XL) further enhances performance, outperforming all other methods.  \n1 Code available at [https://github.com/kallekku/prompt-distillation](https://github.com/kallekku/prompt-distillation)  \n1 Introduction  \nIn many practical scenarios, large language models must integrate new, domain-specific factual knowledge absent from their ori","cbCairFp8ACc5Arw","https://ap.wps.com/l/cbCairFp8ACc5Arw","pdf",954586,1,43,"English","en",105,"# Abstract\n# Introduction\n## Knowledge injection strategies: fine-tuning vs RAG\n## Why supervised fine-tuning underperforms\n## Paper goal","[{\"question\":\"What problem does the paper address in large language models?\",\"answer\":\"It addresses how LLMs acquire new domain-specific factual knowledge when that knowledge is not present in their pre-training corpus.\"},{\"question\":\"How does prompt distillation differ from standard supervised fine-tuning and RAG?\",\"answer\":\"Prompt distillation internalizes knowledge via a self-distillation approach and does not require larger teacher models or structured knowledge formats, while standard SFT depends on supervised question-answer targets and RAG relies on an external knowledge base.\"},{\"question\":\"What are the main empirical results of the proposed method?\",\"answer\":\"Across multiple model sizes and families, prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. Combining prompt distillation with RAG improves performance further, and scaling fact coverage in PD training increases gains.\"}]","Efficient Knowledge Injection in LLMs via Self-Distillation - Self-distillation for internalizing new factual knowledge | PDF",1785677396,108,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"efficient-knowledge-injection-in-llms-via-self-distillation-self-distillation-for-internalizing-new-factual-knowledge","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/efficient-knowledge-injection-in-llms-via-self-distillation-self-distillation-for-internalizing-new-factual-knowledge/117627/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address in large language models?","Question",{"text":75,"@type":76},"It addresses how LLMs acquire new domain-specific factual knowledge when that knowledge is not present in their pre-training corpus.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does prompt distillation differ from standard supervised fine-tuning and RAG?",{"text":80,"@type":76},"Prompt distillation internalizes knowledge via a self-distillation approach and does not require larger teacher models or structured knowledge formats, while standard SFT depends on supervised question-answer targets and RAG relies on an external knowledge base.",{"name":82,"@type":73,"acceptedAnswer":83},"What are the main empirical results of the proposed method?",{"text":84,"@type":76},"Across multiple model sizes and families, prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. Combining prompt distillation with RAG improves performance further, and scaling fact coverage in PD training increases gains.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]