[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83945-en":3,"doc-seo-83945-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83945,687197207639,"Asher","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Weak-to-Strong Generalization via Direct On-Policy Distillation","Reinforcement learning with verifiable rewards (RLVR) can improve language-model reasoning, but repeating it for every newly scaled strong model is costly because training requires many rollouts from the target policy. This work studies a weak-to-strong alternative: perform RL on a cheaper smaller model, then transfer what the RL run learned to train a stronger target. Direct On-Policy Distillation (Direct-OPD) transfers the teacher’s RL-induced policy shift as an implicit dense reward on student on-policy states, avoiding sparse-reward RL on the target. Experiments show consistent gains, including Qwen3-1.7B improving from 48.3% to 58.3% on AIME 2024 in 4 hours on 8 A100 GPUs.","arXiv :2607 .05394v2 [ cs .LG] 8 Jul 2026  \nWeak-to-Strong Generalization via Direct On-Policy Distillation  \nShiyuan Feng* 1 ,2 ,4 , Huan-ang Gao*,‡1 ,2 ,3 , Haohan Chi* 1 ,2 ,3 , Hanlin Wu1 ,2 , Zhilong Zhang1 ,2 ,  \nZheng Jiang3 , Bingxiang He3 , Wei-Ying Ma1 ,2 , Ya-Qin Zhang1 ,2 , Hao Zhou† 1 ,2  \n1 SIA-Lab of Tsinghua AIR and ByteDance Seed  \n2 Institute for AI Industry Research (AIR), Tsinghua University  \n3 Department of Computer Science and Technology, Tsinghua University  \n4 Peking University  \nAbstract  \nReinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving languagemodel reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher’s final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher’s RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student’sown on-policy states. This directly reuses the weak model’s RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.  \nDate: July 9, 2026  \nCorrespondence: [zhouhao@air.tsinghua.edu.cn](zhouhao@air.tsinghua.edu.cn)  \nProject Page: [https://bytedtsinghua-sia.github.io/Direct-OPD/](https://bytedtsinghua-sia.github.io/Direct-OPD/)  \n1 Introduction  \nReinforcement learning with verifiable rewards (RLVR) has become a dominant post-training recipe for eliciting strong reasoning in large language models [1, 2] . Its cost, however, is tied to the model being trained: every update requires the current policy to generate rollouts, receive verifiable outcome scores, and update on those trajectories. Larger target models make each rollout slower and each RL iteration more expensive. Thus, as reasoning models continue to scale, re-running RLVR from scratch for every new strong model risks making post-training itself a bottleneck.  \nIn this paper, we propose Direct On-Policy Distillation (Direct-OPD), a weak-to-strong post-training paradigm.  \n* Equal contribution. ‡Project Lead. †Corresponding author.  \nAIME 24 ave@32  \n62  \n58  \n54  \n50  \n(a) Direct-OPD vs. OPD  \n0 100 200 300 Training step  \nAIME 24 ave@32  \n80  \n70  \n60  \n50  \n40  \nFigure 1 Direct-OPD transfers the effect of small-model RL rather than imitating the small model. (a) Starting from R1-Distill-7B, vanilla OPD toward the post-RL JustRL-1.5B teacher degrades performance, whereas Direct-OPD transfers the JustRL-1.5B − R1-Distill-1.5B policy shift and improves the student. (b) The same policy shift improves Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B on AIME 2024, including students whose initial accuracy already exceeds the post-RL teacher.  \nThe setting is simple: run RL where it is cheap, on a weak model, and use what that RL run learned to train a stronger target model. Crucially, the weak model is not assumed to be a better reasoner than the student; it is merely an inexpensive vehicle whi","cbCaie4KUDFH4diJ","https://ap.wps.com/l/cbCaie4KUDFH4diJ","pdf",1318820,6,1,24,"English","en",105,"# Introduction\n## Weak-to-strong post-training motivation\n## Direct-OPD concept and policy shift\n## Weak model checkpoints as supervision signal","[{\"question\":\"Why is RLVR expensive when scaling to new strong language models?\",\"answer\":\"RLVR cost depends on the model being trained because each RL update requires generating rollouts with the current policy, scoring them with verifiable outcomes, and updating on those trajectories. Larger target models make each rollout slower and each RL iteration more expensive, so repeating RLVR from scratch becomes a bottleneck.\"},{\"question\":\"What does Direct-OPD transfer from the weak model to the strong model?\",\"answer\":\"Direct-OPD transfers the weak model’s RL-induced policy shift rather than distilling the weak model’s final post-RL policy. It contrasts the weak model before and after RL and uses the log-ratio shift as an implicit dense reward signal applied to the student’s own on-policy states.\"},{\"question\":\"How does Direct-OPD avoid sparse-reward RL on the target model?\",\"answer\":\"Direct-OPD applies the transferred RL signal directly during on-policy training of the stronger student states, anchored by a KL term to the student initialization. It does not train an explicit reward model or run sparse-reward RL on the target.\"}]",1784191610,60,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"weak-to-strong-generalization-via-direct-on-policy-distillation","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/weak-to-strong-generalization-via-direct-on-policy-distillation/83945/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"Why is RLVR expensive when scaling to new strong language models?","Question",{"text":76,"@type":77},"RLVR cost depends on the model being trained because each RL update requires generating rollouts with the current policy, scoring them with verifiable outcomes, and updating on those trajectories. Larger target models make each rollout slower and each RL iteration more expensive, so repeating RLVR from scratch becomes a bottleneck.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What does Direct-OPD transfer from the weak model to the strong model?",{"text":81,"@type":77},"Direct-OPD transfers the weak model’s RL-induced policy shift rather than distilling the weak model’s final post-RL policy. It contrasts the weak model before and after RL and uses the log-ratio shift as an implicit dense reward signal applied to the student’s own on-policy states.",{"name":83,"@type":74,"acceptedAnswer":84},"How does Direct-OPD avoid sparse-reward RL on the target model?",{"text":85,"@type":77},"Direct-OPD applies the transferred RL signal directly during on-policy training of the stronger student states, anchored by a KL term to the student initialization. It does not train an explicit reward model or run sparse-reward RL on the target.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,110,114,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":29,"slug":109},5,"Comic","comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":107,"slug":137},19,"General","general"]