[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83915-en":3,"doc-seo-83915-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83915,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Rethinking On-Policy Self-Distillation for Thinking Models","Self-distillation uses a language model as both student and teacher by providing privileged information such as solutions, final answers, or feedback. The method is attractive for thinking models that deliberate through test-time reasoning and long rollout traces. However, privileged-context self-distillation degrades thinking models: across five Qwen3 and OLMo thinking models on AIME24, AIME25, and HMMT25, accuracy drops up to 17% at avg@16. The harm grows with withheld privileged context and is strongest at long budgets, indicating token-level signal reversal at high-entropy forking positions.","arXiv :2607 .05 184v 1 [ cs .AI] 6 Jul 2026  \nRethinking On-Policy Self-Distillation for Thinking  \nModels  \nSimran Kaur Narutatsu Ri Yinghui He Liam Fowl Sanjeev Arora  \nPrinceton Language and Intelligence, Princeton University {skaur,nr3764,yh0068,lf2728,[arora}@cs.princeton.edu](arora}@cs.princeton.edu)  \nAbstract  \nSelf-distillation has emerged as a promising recipe for self-improvement in language models [Zhao et al., 2026, Shenfeld et al., 2026, Hübotter et al., 2026] .  \nIn this setting, a model can be used as its own teacher when augmented with privileged information (e.g. a solution to a math problem) . The approach seems especially appealing for thinking models, which can leverage test-time reasoning to fully absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models with respect to long reasoning traces:  \nacross five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged on-policy distillation reverses these gains. Our diagnostics suggest that this failure mode is linked to how privileged teacher context reshapes learning at high-entropy forking positions [Bigelow et al., 2024, Zhang et al., 2026], i.e., rollout positions where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction model rollouts. This leads to an interesting dichotomy wherein privileged context can help instruction-tuned models but hurts more performant thinking models that depend heavily on exploration and rollout quality. This effect is especially visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that applying self-distillation methods to strong thinking models requires further consideration of token-level signal—especially around tokens related to correction and crucial reasoning steps.  \n1 Introduction  \nOn-policy self-distillation (OPSD) has emerged as an exciting approach for self-improvement in language models [Zhao et al., 2026, Shenfeld et al., 2026, Hübotter et al., 2026] . In this setting, a single model plays the role of both a student and teacher. The teacher is provided additional privileged information, such as a gold solution, a final answer, or environmental feedback.  \nThinking models are natural candidates for self-improvement. Post-trained to deliberate at test time [OpenAI, 2024, DeepSeek-AI et al., 2025, Yang et al., 2025], they can branch into cases, verify intermediate steps, hedge, backtrack, and recover from errors before committing to an answer [Arora and Zanette, 2025, Gandhi et al., 2025, Venhoff et al., 2025] . Yet existing methods have mostly been studied outside this regime, using instruction-tuned models, short generation budgets, or  \nPreprint.  \ntrained on rollouts with thinking disabled [Zhao et al., 2026, Shenfeld et al., 2026, Hübotter et al., 2026] . This leaves open whether privileged-context self-distillation remains beneficial when the supervised trajectory is itself the long deliberation trace that thinking models rely on at test time. In this paper, we report a negative result: existing privileged-context on-policy self-distillation methods can degrade thinking models, particularly at long rollout budgets. This degradation is not explained by the short tra","cbCaiaJpqH1hGgwD","https://ap.wps.com/l/cbCaiaJpqH1hGgwD","pdf",998744,1,31,"English","en",105,"# Abstract\n# Introduction\n## On-policy self-distillation setup\n## Thinking models and long test-time deliberation\n## Negative findings and failure mode\n## Mechanistic token-level explanation","[{\"question\":\"What is privileged-context self-distillation in on-policy self-distillation?\",\"answer\":\"A single model acts as both student and teacher, while the teacher receives additional privileged information such as a gold solution, final answer, or feedback.\"},{\"question\":\"What effect does privileged-context distillation have on thinking models?\",\"answer\":\"It degrades thinking models on long reasoning traces, with a reported relative drop of up to 17% in avg@16 accuracy across multiple thinking models and benchmark sets.\"},{\"question\":\"Why does privileged context cause the degradation?\",\"answer\":\"Diagnostics suggest privileged teacher context changes learning at high-entropy forking positions, suppressing exploration behaviors; this coincides with reduced fork rates, weaker self-correction signals, and fewer deliberation markers.\"}]",1784191423,78,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"rethinking-on-policy-self-distillation-for-thinking-models","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/rethinking-on-policy-self-distillation-for-thinking-models/83915/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is privileged-context self-distillation in on-policy self-distillation?","Question",{"text":75,"@type":76},"A single model acts as both student and teacher, while the teacher receives additional privileged information such as a gold solution, final answer, or feedback.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What effect does privileged-context distillation have on thinking models?",{"text":80,"@type":76},"It degrades thinking models on long reasoning traces, with a reported relative drop of up to 17% in avg@16 accuracy across multiple thinking models and benchmark sets.",{"name":82,"@type":73,"acceptedAnswer":83},"Why does privileged context cause the degradation?",{"text":84,"@type":76},"Diagnostics suggest privileged teacher context changes learning at high-entropy forking positions, suppressing exploration behaviors; this coincides with reduced fork rates, weaker self-correction signals, and fewer deliberation markers.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]