[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85653-en":3,"doc-seo-85653-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85653,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","DemoPSD Disagreement-Modulated Policy Self-Distillation","On-policy self-distillation (OPSD) trains large language models by using a single model as both teacher and student with different information access. Recent work shows privileged token-level supervision can overfit in-domain patterns, suppress exploration, and cause privileged information leakage that creates answer-dependent shortcuts at test time. DemoPSD introduces selective adoption of teacher guidance via distribution discrepancy to modulate blending per token, targeting a reverse-KL barycenter. Results on SciKnowEval across four scientific fields outperform GRPO and SDPO, maintaining higher entropy and strong out-of-distribution generalization.","arXiv :2607 .02502v 3 [ cs .LG] 12 Jul 2026  \nDemoPSD: Disagreement-Modulated Policy  \nSelf-Distillation  \nYunhe Li *,1, Hao Shi *,2, Wenhao Liu2 , Mengzhe Ruan 1 , Hanxu Hou3  \nZhongxiang Dai4 , Shuang Qiu†,1, Linqi Song†,1  \n1 City University of Hong Kong 2Tsinghua University  \n3 Shenzhen University of Advanced Technology 4 Chinese University of Hong Kong, Shenzhen  \n[uuen. li@my. cityu. edu. hk](uuen. li@my. cityu. edu. hk) [shih22@mails. tsinghua. edu. cn](shih22@mails. tsinghua. edu. cn)[ ](shih22@mails. tsinghua. edu. cn){shuanqiu, [linqi. song}@cityu. edu. hk](linqi. song}@cityu. edu. hk)  \nAbstract  \nOn-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher’s dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance: the student adopts the teacher’s guidance when their distributions remain reasonably consistent, and relies more on its own reasoning when their distributions substantially diverge, indicating that the teacher’s output is overly influenced by privileged information. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student’s own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.  \n1. Introduction  \nReinforcement learning with verifiable rewards (RLVR) has become a central paradigm for post-training large language models on reasoning tasks (Shao et al., 2024, DeepSeek-AI, 2025, Yu et al., 2026a) . Methods such as Group Relative Policy Optimization (GRPO) train models by sampling multiple rollouts per question and using  \n*  \nEqual contribution.†Corresponding author.  \n(a) Policy entropy over training steps. DemoPSD main- (b) Best@16 for each SciKnowEval domain over training tains 33-98% higher entropy than SDPO across all do- steps.  \nmains, avoiding policy entropy collapse.  \nFigure 1: DemoPSD preserves higher entropy (left), which translates into better best@16 performance (right) .  \noutcome correctness as a reward signal. While effective, RLVR suffers from a fundamental credit assignment bottleneck: standard RLVR methods distribute a rollout-level reward uniformly among all tokens in a rollout, offering coarse token-level credit signals that fail to distinguish individual token contributions (Hübotter et al., 2026) .  \nOn-policy distillation (OPD) addresses this bottleneck by introducing dense, token-level supervision from a teacher model on the student’s self-generated trajectories (Agarwal et al., 2024, Gu et al., 2024, Lu and Thinking Machines Lab, 2025) . Unlike off-policy distillation, which trains on teacher-generated texts but suffers from exposure bias (Ross et al., 2011, Song and Zheng, 2026b), OPD ","cbCaijW9b7UIna1s","https://ap.wps.com/l/cbCaijW9b7UIna1s","pdf",3066409,2,1,21,"English","en",105,"# Introduction\n## Reinforcement learning with verifiable rewards (RLVR)\n## On-policy distillation (OPD)\n## On-policy self-distillation (OPSD) and privileged information leakage\n## DemoPSD selective teacher adoption","[{\"question\":\"What problem does privileged information leakage create in OPSD?\",\"answer\":\"Privileged information available only to the teacher provides extra signal about next-token prediction after conditioning on the question and generated prefix. This drives the student to encode answer-dependent shortcuts unavailable at test time, causing performance to degrade over time.\"},{\"question\":\"How does DemoPSD decide when to follow the teacher vs rely on the student?\",\"answer\":\"DemoPSD measures distribution discrepancy between teacher and student outputs. When the distributions remain reasonably consistent, the student adopts teacher guidance; when they substantially diverge, blending shifts toward the student’s own reasoning to reduce reliance on privileged cues.\"},{\"question\":\"What training target does DemoPSD use to balance teacher learning and student reasoning?\",\"answer\":\"Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter, a weighted geometric combination of teacher and student distributions. This construction aims to balance learning from the teacher while preserving the student’s reasoning capacity.\"}]",1784205378,53,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"demopsd-disagreement-modulated-policy-self-distillation","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/demopsd-disagreement-modulated-policy-self-distillation/85653/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-19","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does privileged information leakage create in OPSD?","Question",{"text":75,"@type":76},"Privileged information available only to the teacher provides extra signal about next-token prediction after conditioning on the question and generated prefix. This drives the student to encode answer-dependent shortcuts unavailable at test time, causing performance to degrade over time.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does DemoPSD decide when to follow the teacher vs rely on the student?",{"text":80,"@type":76},"DemoPSD measures distribution discrepancy between teacher and student outputs. When the distributions remain reasonably consistent, the student adopts teacher guidance; when they substantially diverge, blending shifts toward the student’s own reasoning to reduce reliance on privileged cues.",{"name":82,"@type":73,"acceptedAnswer":83},"What training target does DemoPSD use to balance teacher learning and student reasoning?",{"text":84,"@type":76},"Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter, a weighted geometric combination of teacher and student distributions. This construction aims to balance learning from the teacher while preserving the student’s reasoning capacity.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]