[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-279639-105":53,"doc-detail-279639-en":126},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":119,"head_meta":121,"extra_data":123,"updated_unix":125},105,"en","reinforcement-learning-via-self-distillation-self-distillation-policy-optimization-sdpo","Reinforcement Learning via Self-Distillation - Self-Distillation Policy Optimization (SDPO)","","Large language models increasingly rely on reinforcement learning in verifiable domains like code and math, but RLVR methods typically learn only from a single scalar outcome reward, creating a strong credit-assignment bottleneck. Many environments instead provide rich tokenized feedback, such as runtime errors or judge evaluations. SDPO formalizes reinforcement learning with rich feedback and converts tokenized feedback into a dense learning signal without an external teacher or explicit reward model, improving sample efficiency and final accuracy. SDPO also speeds discovery on binary-reward tasks at test time.",{"@graph":63,"@context":118},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/reinforcement-learning-via-self-distillation-self-distillation-policy-optimization-sdpo/279639/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/reinforcement-learning-via-self-distillation-self-distillation-policy-optimization-sdpo/279639.png","ImageObject",442,249,{"name":88,"@type":89},"Maeve","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-09-24","2026-09-16",true,{"@type":98,"interactionType":99,"userInteractionCount":9},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104,110,114],{"name":105,"@type":106,"acceptedAnswer":107},"What problem does RLVR face in verifiable domains?","Question",{"text":108,"@type":109},"RLVR learns from a scalar outcome reward per attempt, which creates a credit-assignment bottleneck and masks the underlying reasons for failure.","Answer",{"name":111,"@type":106,"acceptedAnswer":112},"How does SDPO use rich feedback instead of scalar rewards?",{"text":113,"@type":109},"SDPO treats the current feedback-conditioned model as a self-teacher, distilling its feedback-informed next-token predictions into the policy to produce a dense learning signal.",{"name":115,"@type":106,"acceptedAnswer":116},"What benefits does SDPO show on LiveCodeBench v6?",{"text":117,"@type":109},"SDPO improves sample efficiency and final accuracy over strong RLVR baselines, and it can match GRPO’s final accuracy with about 4× fewer generations.","https://schema.org",{"og:url":78,"og:type":120,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":122,"canonical":78},"index,follow",{"doc_id":124,"site_id":56},279639,1790219054,{"code":4,"msg":5,"data":127},{"doc_id":124,"user_id":128,"nickname":88,"user_avatar":129,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":9,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":32,"language":135,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":136,"faqs":137,"seo_title":138,"seo_description":61,"update_tm":139,"read_time":40},5909877438554,"https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272","arXiv :2601 .20802v2 [ cs .LG] 16 Feb 2026  \nReinforcement Learning via Self-Distillation  \nJonas Hübotter1 Frederike Lübeck∗, 1, 2 Lejs Behric∗, 1 Anton Baumann∗, 1 Marco Bagatella1, 2 Daniel Marta1 Ido Hakimi1 Idan Shenfeld3 Thomas Kleine Buening1 Carlos Guestrin4 Andreas Krause1  \n1ETH Zurich 2Max Planck Institute for Intelligent Systems 3MIT 4Stanford  \n[https://github.com/lasgroup/SDPO](https://github.com/lasgroup/SDPO)  \nAbstract  \nLarge language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model’s ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselinesin standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3× fewer attempts.  \n1 Introduction  \nProgress in deep reinforcement learning has shown that iterating on experience—acting, receiving feedback, and updating a policy—can unlock capabilities that are difficult to obtain from static supervision alone (Mnihet al., 2015; Silver et al., 2016; 2017; Berner et al., 2019) . The same theme now appears in large language models (LLMs): large-scale post-training with reinforcement learning (RL) has substantially improved performance on reasoning-heavy tasks, especially in settings with programmatic or otherwise verifiable evaluation (Jaech et al., 2024; Guo et al., 2025; Kimi et al., 2025; Olmo et al., 2025) .  \nNevertheless, the dominant RL recipe for LLM post-training remains bottlenecked by  \nAccuracy  \n0.50 0.45 0.40 0.35  \n0.30  \nLiveCodeBench v6  \n\n| |  |\n| --- | --- |\n| | |\n\n0 5000 10000 15000 20000  \nNumber of Generations  \nFigure 1: SDPO substantially outperformsan improved version of Group Relative Policy Optimization (GRPO) on LCB v6 with Qwen3-8B. Further, SDPO achieves GRPO’s final accuracy in 4× fewer generations. Claude Sonnet 4 is the strongest instruct model on the public LCBv6 leaderboard. Shaded regions show the standard deviation across 3 seeds.  \n∗ Equal [second authorship. Correspondence to](second authorship. Correspondence to jonas.huebotter@inf.ethz.ch)[ jonas.huebotter@inf.ethz.ch](second authorship. Correspondence to jonas.huebotter@inf.ethz.ch).  \nQuestion:  \nHow much impulse did the thrusters generate for the  \nMars Climate Answer:  \n100 lbf-s  \nOrbiter?  \nincorrect!  \nReinforcement Learning with Verifiable Rewards (RLVR)  \n-1.0  \n Agent   \nreward action  \n Environment   \nReinforcement Learning with Rich Feedback (RLRF)  \nricher signal  \nFigure 2: Comparison of RLVR and RLRF settings. In Reinforcement Learning with Verifiable Rewards (RLVR), the agent learns from a scalar reward r, which often acts asan information bottleneck by masking the underlying environment state. In contrast, Reinforcement Learning with Rich Feedback (RLRF) utilizes t","cbCaihRZqJnUujCk","https://ap.wps.com/l/cbCaihRZqJnUujCk","pdf",1304168,"English","# Abstract\n# Introduction\n## Reinforcement Learning with Verifiable Rewards (RLVR)\n## Reinforcement Learning with Rich Feedback (RLRF)\n## Self-Distillation Policy Optimization (SDPO)","[{\"question\":\"What problem does RLVR face in verifiable domains?\",\"answer\":\"RLVR learns from a scalar outcome reward per attempt, which creates a credit-assignment bottleneck and masks the underlying reasons for failure.\"},{\"question\":\"How does SDPO use rich feedback instead of scalar rewards?\",\"answer\":\"SDPO treats the current feedback-conditioned model as a self-teacher, distilling its feedback-informed next-token predictions into the policy to produce a dense learning signal.\"},{\"question\":\"What benefits does SDPO show on LiveCodeBench v6?\",\"answer\":\"SDPO improves sample efficiency and final accuracy over strong RLVR baselines, and it can match GRPO’s final accuracy with about 4× fewer generations.\"}]","Reinforcement Learning via Self-Distillation - Self-Distillation Policy Optimization (SDPO) | PDF",1789524739]