[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85276-en":3,"doc-seo-85276-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85276,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",6,"Technology","EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models","Conventional language-model distillation often depends on fixed teacher-generated data, which may miss states that an evolving student policy will encounter during deployment. On-policy distillation (OPD) addresses the mismatch by collecting teacher or evaluator supervision on student-generated rollouts. Existing OPD approaches remain fragmented across supervision interfaces, tokenizer compatibility, teacher access, and supervision granularity. EASYOPD is a unified on-policy distillation framework built on verl, enabling method-local modules with shared distributed execution to reproduce and extend OPD methods.","EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models  \nJie Sun 1 ,3 ,∗ ,§ , Mao Zheng2 ,∗ , Mingyang Song2 ,∗ , Qiyong Zhong 1 ,∗ ,§ Gengsheng Li 1 ,∗ ,§ , Zhepei Hong 1 , Chang Wu 1 , Pengfei Liu3  \nJunfeng Fang4 ,†, Xiang Wang 1 ,†  \n1 University of Science and Technology of China 2 LLM Department, Tencent  \n3 Shanghai Innovation Institute 4 National University of Singapore  \n§ [https://github.com/lds-ustc/EasyOPD](https://github.com/lds-ustc/EasyOPD)  \narXiv :2607 . 1 10 12v 1 [ cs .CL] 13 Jul 2026  \nAbstract  \nConventional language-model distillation often relies on fixed teacher-generated data, which may not cover the states encountered by an evolving student policy. On-policy distillation (OPD) instead collects teacher or evaluator supervision on student-generated rollouts. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend. We present EASYOPD, an on-policy distillation framework built on verl, a distributed reinforcement-learning framework for large language models. EASYOPD separates user-side configuration, method-specific supervision logic, and verl-based execution. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation. We instantiate representative methods for three OPD settings—cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD. Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectivesand task-dependent performance profiles. We release EASYOPD with runnable YAML configurations, documentation, and an installable demonstration package and video.  \n1 Introduction  \nLarge language model distillation transfers capabilities from a stronger teacher to a smaller, more efficient student, aiming to reduce deployment costs while retaining reasoning, coding, and instructionfollowing abilities (Alkhulaifi et al., 2021) . Conventional distillation uses teacher responses or to-  \n∗ Equal Contribution.  \n§ Work done during internship  \n† Corresponding Authors. at Tencent.  \nken distributions on fixed contexts (Yang et al., 2024b) . At inference, however, the student conditions on its own prefixes and may encounter contexts poorly covered by offline supervision (Guet al., 2024 ; Ko et al., 2024) . On-policy distillation (OPD) mitigates this mismatch through a common loop: the current student generates rollouts, a teacher or evaluator supervises them, and the resulting signals guide student updates (Song and Zheng, 2026 ; Lu and Thinking Machines Lab, 2025) . OPD methods vary by supervision interface, tokenizer compatibility, and granularity, with overlapping settings including logit-based distillation, cross-tokenizer alignment (Sun et al., 2026), onpolicy self-distillation, black-box or rubric-based feedback, and step-wise or multi-turn agentic supervision (Zhong et al., 2026 ; Li et al., 2026a) .  \nA shared algorithmic paradigm does not yield a common system interface. Depending on the supervision, OPD may span rollout generation, teacher inference, cross-tokenizer alignment, reward construction, distributed data flow, and optimization. For example, cross-tokenizer OPD requires alignment across teacher and student tokenizations before constructing compatible supervision, whereas black-box or rubric-based OPD may convert textual judgments or scores into rewards; these patterns require different extension points. Existing frameworks address complementary parts of the OPD workflow: TRL (von Werra et al., 2020) provides trainer-centric distillation APIs; verl (Sheng et al., 2024) and slime (Zhu et al., 2025) support scalable on","cbCaivwspEBG9lyu","https://ap.wps.com/l/cbCaivwspEBG9lyu","pdf",478400,1,10,"English","en",105,"# Introduction\n## Motivation and OPD mismatch\n## Fragmentation in existing OPD systems\n## EASYOPD unified framework and architecture","[{\"question\":\"What problem does EASYOPD address in conventional language-model distillation?\",\"answer\":\"Conventional distillation relies on fixed teacher data that may not cover contexts the student encounters during inference. EASYOPD targets this mismatch by enabling on-policy distillation using supervision on student-generated rollouts.\"},{\"question\":\"How does on-policy distillation (OPD) differ from fixed-context distillation?\",\"answer\":\"OPD uses a loop where the current student generates rollouts, a teacher or evaluator supervises them, and the resulting signals update the student. This aligns training supervision with student-discovered states.\"},{\"question\":\"What is EASYOPD’s core design for unifying different OPD methods?\",\"answer\":\"EASYOPD adds a lightweight abstraction layer on top of verl so method modules plug into explicit extension points for loss, rollouts, rewards, tokenizer alignment, and teacher-side computation. Multiple OPD settings can then run on the same distributed backend selected via YAML configurations.\"}]",1784202210,25,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"easyopd-an-easy-to-use-on-policy-distillation-framework-for-large-language-models","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/easyopd-an-easy-to-use-on-policy-distillation-framework-for-large-language-models/85276/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does EASYOPD address in conventional language-model distillation?","Question",{"text":74,"@type":75},"Conventional distillation relies on fixed teacher data that may not cover contexts the student encounters during inference. EASYOPD targets this mismatch by enabling on-policy distillation using supervision on student-generated rollouts.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does on-policy distillation (OPD) differ from fixed-context distillation?",{"text":79,"@type":75},"OPD uses a loop where the current student generates rollouts, a teacher or evaluator supervises them, and the resulting signals update the student. This aligns training supervision with student-discovered states.",{"name":81,"@type":72,"acceptedAnswer":82},"What is EASYOPD’s core design for unifying different OPD methods?",{"text":83,"@type":75},"EASYOPD adds a lightweight abstraction layer on top of verl so method modules plug into explicit extension points for loss, rollouts, rewards, tokenizer alignment, and teacher-side computation. Multiple OPD settings can then run on the same distributed backend selected via YAML configurations.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,112,117,122,127,130,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":110,"slug":111},50,"technology",{"id":113,"doc_module":4,"doc_module_name":45,"category_name":114,"show_sort_weight":115,"slug":116},7,"Healthcare",40,"healthcare",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":119,"show_sort_weight":120,"slug":121},8,"Research & Report",30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":21,"slug":132},"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]