[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-279638-105":53,"doc-detail-279638-en":126},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":119,"head_meta":121,"extra_data":123,"updated_unix":125},105,"en","learning-by-distilling-context-paper-1","LEARNING BY DISTILLING CONTEXT - Paper - 1","","Language models benefit from context tokens such as prompts or scratchpads, gaining better performance from informative instructions and step-by-step reasoning. Those gains vanish when the context is removed, limiting efficiency at inference time. This work introduces context distillation to internalize these improvements: a teacher conditions on instructions plus task input to produce a scratch-pad and final answer, then the student is fine-tuned to predict only the final answer from task input. The approach learns abstract instructions, reasoning traces, and concrete examples, outperforming gradient descent by 9% on SPIDER Text-to-SQL.",{"@graph":63,"@context":118},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/learning-by-distilling-context-paper-1/279638/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/learning-by-distilling-context-paper-1/279638.png","ImageObject",442,249,{"name":88,"@type":89},"Ivy","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-09-22","2026-09-16",true,{"@type":98,"interactionType":99,"userInteractionCount":73},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104,110,114],{"name":105,"@type":106,"acceptedAnswer":107},"What problem does context distillation address for language models?","Question",{"text":108,"@type":109},"Performance improvements from prompts and scratchpads do not persist when those context tokens are removed. The method targets internalizing these gains so the model can rely only on task input at inference time.","Answer",{"name":111,"@type":106,"acceptedAnswer":112},"How does the teacher-student training procedure work?",{"text":113,"@type":109},"The teacher is prompted with instructions plus the task input to generate a scratch-pad and final answer. The student is fine-tuned to predict only the final answer conditioned on the task input, without instructions or the scratch-pad.",{"name":115,"@type":106,"acceptedAnswer":116},"Which types of training signals can context distillation internalize?",{"text":117,"@type":109},"It can internalize abstract task instructions and explanations, step-by-step reasoning for complex tasks, and concrete training examples, with a reported 9% improvement over direct gradient descent on SPIDER Text-to-SQL.","https://schema.org",{"og:url":78,"og:type":120,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":122,"canonical":78},"index,follow",{"doc_id":124,"site_id":56},279638,1790037661,{"code":4,"msg":5,"data":127},{"doc_id":124,"user_id":128,"nickname":88,"user_avatar":129,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":76,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":35,"language":135,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":136,"faqs":137,"seo_title":138,"seo_description":61,"update_tm":139,"read_time":140},549758252649,"https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819","arXiv :2209 . 15189v1 [ cs .CL] 30 Sep 2022  \nLEARNING BY DISTILLING CONTEXT  \nCharlie Snell, Dan Klein, Ruiqi Zhong  \nUniversity of California, Berkeley, EECS Department  \nfcsnell22, klein, [ruiqi-zhong](ruiqi-zhongg@berkeley.edu)[g](ruiqi-zhongg@berkeley.edu)[@berkeley.edu](ruiqi-zhongg@berkeley.edu)  \nABSTRACT  \nLanguage models significantly benefit from context tokens, such as prompts or scratchpads. They perform better when prompted with informative instructions, and they acquire new reasoning capabilities by generating a scratch-pad before predicting the final answers. However, they do not internalize these performance gains, which disappear when the context tokens are gone. Our work proposes to apply context distillation so that a language model can improve itself by internalizing these gains. Concretely, given a synthetic unlabeled input for the target task, we condition the model on “[instructions] + [task-input]” to predict “[scratch-pad]+ [final answer]”; then we fine-tune the same model to predict its own “[final answer]” conditioned on the “[task-input]”, without seeing the “[instructions]” or using the “[scratch-pad]”.  \nWe show that context distillation is a general method to train language models, and it can effectively internalize 3 types of training signals. First, it can internalize abstract task instructions and explanations, so we can iteratively update the model parameters with new instructions and overwrite old ones. Second, it can internalize step-by-step reasoning for complex tasks (e.g., 8-digit addition), and such a newly acquired capability proves to be useful for other downstream tasks.  \nFinally, it can internalize concrete training examples, and it outperforms directly learning with gradient descent by 9% on the SPIDER Text-to-SQL dataset; furthermore, combining multiple context distillation operations can internalize more training examples than what the context window size allows.  \n1 INTRODUCTION  \nRecent work has shown that language models significantly benefit from context tokens. When prompted with task definitions, language models can perform zero-shot learning (Wei et al., 2022a; Sanh et al., 2022), and the performance further improves with additional in-context examples and explanations (Chen et al., 2022; Scheurer et al., 2022) . They also acquire the capability to perform more complex tasks by generating step-by-step reasoning in the context window before predicting the final answer (Nye et al., 2021b; Wei et al., 2022b; Zhou et al., 2022) .  \nHowever, language models cannot internalize these performance gains, which disappear when the context tokens are gone. Consequently, we always need to pay extra computation for running inference on context tokens; this is undesirable, as sometimes the task instructions and the scratch-pad can be more than 10x longer than the actual task inputs. Furthermore, it is unclear how to leverage the context tokens when their total length exceeds the context window size. These shortcomings are analogous to how humans are slow at performing complex cognitive tasks (Wason & Evans, 1974) and can hold only a limited amount of information in the working memory (Baddeley, 1992) .  \nHumans get around this by practicing. Consider, for example, learning to type your friends' phone numbers. The first few times you type it, you need to consciously recall the number using working memory and slowly decide which button to press. After repeatedly typing the same number, it becomes a habit and you can type the number quickly without conscious reasoning. Through repeated practice, the knowledge of your friend's phone number is “distilled” into your muscle memories.1 This mechanism for distilling knowledge is critical for learning complex tasks because it allows us  \n1See declarative learning vs. procedural learning for a friendly but more in-depth discussion. [https:](https:)//[en.wikipedia.org/wiki/Declarative_learning](en.wikipedia.org/wiki/Declarative_learning)  \nModel Input Model ","cbCaigT3W3sgXT67","https://ap.wps.com/l/cbCaigT3W3sgXT67","pdf",1313199,"English","# Abstract\n# 1 Introduction\n## Context tokens and limitations\n## Human practice analogy\n## Overview of the framework\n# Method Overview\n## Teacher and student prompting\n## Fine-tuning objective\n# Results and Claims\n## Internalizing task instructions\n## Internalizing step-by-step reasoning\n## Internalizing concrete examples","[{\"question\":\"What problem does context distillation address for language models?\",\"answer\":\"Performance improvements from prompts and scratchpads do not persist when those context tokens are removed. The method targets internalizing these gains so the model can rely only on task input at inference time.\"},{\"question\":\"How does the teacher-student training procedure work?\",\"answer\":\"The teacher is prompted with instructions plus the task input to generate a scratch-pad and final answer. The student is fine-tuned to predict only the final answer conditioned on the task input, without instructions or the scratch-pad.\"},{\"question\":\"Which types of training signals can context distillation internalize?\",\"answer\":\"It can internalize abstract task instructions and explanations, step-by-step reasoning for complex tasks, and concrete training examples, with a reported 9% improvement over direct gradient descent on SPIDER Text-to-SQL.\"}]","LEARNING BY DISTILLING CONTEXT - Paper - 1 | PDF",1789524737,6]