[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273464-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273464-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","step-level-on-policy-distillation-interpolating-between-on-policy-distillation-and-supervised-fine-tuning-abstract","Step-Level On-Policy Distillation - Interpolating Between On-Policy Distillation and Supervised Fine-Tuning - Abstract","","On-policy distillation (OPD) trains a student to match a teacher’s logits over student-generated trajectories, often outperforming off-policy distillation with less data. Yet token-level OPD yields only fragmented corrections and cannot provide a full, correct repair path. Step-Level On-Policy Distillation (SOPD) merges supervised fine-tuning’s long-horizon supervision with OPD’s on-policy benefits by offering step-level supervision over complete student trajectories. Under step-length limits, SOPD reduces to SFT or approximates OPD. Experiments on reasoning and agent tasks show substantial gains, including a 13.4-point success-rate improvement on ALFWorld versus Vanilla OPD.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/step-level-on-policy-distillation-interpolating-between-on-policy-distillation-and-supervised-fine-tuning-abstract/273464/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/step-level-on-policy-distillation-interpolating-between-on-policy-distillation-and-supervised-fine-tuning-abstract/273464.png","ImageObject",300,407,{"name":42,"@type":43},"Ava Thompson","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-23","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",11,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does SOPD address in standard token-level on-policy distillation?","Question",{"text":63,"@type":64},"Standard token-level OPD gives only fragmented one-token corrections and cannot unfold a complete and correct repair path along an erroneous trajectory.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does SOPD combine SFT and OPD?",{"text":68,"@type":64},"SOPD provides step-level supervision over a complete student-generated trajectory: after the student finishes, the teacher generates one step conditioned on each student prefix.",{"name":70,"@type":61,"acceptedAnswer":71},"What do the limiting cases of SOPD correspond to?",{"text":72,"@type":64},"When the number of steps approaches one, SOPD reduces to SFT; when the step length approaches one token, it approximates forward-KL OPD.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273464,1789471549,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":113,"preview_page_count":4},1649267921044,"https://us-avatar.wpscdn.com/avatar/1800007509477c92dfb?_k=1786009248482753345","arXiv :2608 . 16333v1 [ cs .CL] 17 Aug 2026  \nSTEP-LEVEL ON-POLICY DISTILLATION: INTERPOLATING BETWEEN ON-POLICY DISTILLATION AND SUPERVISED FINE-TUNING  \nChanghui Sun 1 ,2 , Lanbo Liu2 , Hang Lei2 , Tong Ling2 ,3 Jiahang Xie2 ,4 , Zhiyong Zheng2 , Yujia Wang2 , Hao Liu2 Feng Xiao2 , Lu Liu2 , Yanlong Du2 , Zifeng Cheng 1† Ziwei Jiang 1†, Qing Gu 1  \n1 State Key Laboratory for Novel Software Technology, Nanjing University, China  \n2XingYun Lab, HUJING Digital Media & Entertainment Group  \n3University of Chinese Academy of Sciences, Beijing, China  \n4 School of Data Science, Fudan University, Shanghai, China  \nABSTRACT  \nOn-policy distillation (OPD) aligns a student model with a teacher’s logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose Step-Level On-Policy Distillation (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers anew perspective for future research on distillation methods.  \n1 INTRODUCTION  \nRecently, on-policy distillation (OPD) (Agarwal et al., 2024; Yang et al., 2025; Lu & Lab, 2025) has emerged as an effective post-training paradigm for improving the capabilities of large language models (LLMs) . Unlike earlier off-policy distillation methods (Taori et al., 2023; Guha et al., 2025), which train the student on teacher-generated trajectories, OPD allows the student to learn from teacher supervision—namely, predicted logits—on student-generated tokens.  \nStandard OPD queries a teacher distribution at every token generated by the student. Although this provides relatively dense supervision, it also fragments the teacher’s correction signal: in a single OPD rollout, the student receives only one-token guidance at each position rather than a continuous and correct segment-level correction (Jiang et al., 2026) . Recent work has revisited the potential of DAgger-style methods for training LLM agents, seeking to preserve SFT’s long-horizon supervision while generating more training trajectories from student states (Lauffer et al., 2025) . DAgger, however, inherently inserts teacher responses into the student-generated trajectory, substantially changing the trajectory the student would otherwise have produced.  \nWe combine the on-policy property of OPD with the long-horizon supervision of SFT and propose  \nStep-Level On-Policy Distillation (SOPD) . SOPD provides step-level supervision over a complete †Corresponding authors.  \nstudent-generated trajectory. After the student completes the full trajectory, SOPD asks the teacher to generate one step from the student prefix at every student step. Our method reduces to SFT when the number of steps approaches one and approximates forward-KL OPD when the step length approaches one token.  \nWe show that SOPD outperforms both SFT and OPD across different tasks. For agent tasks, we simply define a step as one actual environment-interaction turn. SOPD substantially improves over both SFT and OPD while reducing the number of interaction rounds.","cbCailECtfWYFHLd","https://ap.wps.com/l/cbCailECtfWYFHLd","pdf",1427170,12,"English","# Abstract\n# Introduction\n## Background: on-policy vs off-policy distillation\n## Proposed method: Step-Level On-Policy Distillation (SOPD)\n## Key results and practical advantages\n# Related Work\n## Knowledge distillation\n## On-policy language-model distillation","[{\"question\":\"What problem does SOPD address in standard token-level on-policy distillation?\",\"answer\":\"Standard token-level OPD gives only fragmented one-token corrections and cannot unfold a complete and correct repair path along an erroneous trajectory.\"},{\"question\":\"How does SOPD combine SFT and OPD?\",\"answer\":\"SOPD provides step-level supervision over a complete student-generated trajectory: after the student finishes, the teacher generates one step conditioned on each student prefix.\"},{\"question\":\"What do the limiting cases of SOPD correspond to?\",\"answer\":\"When the number of steps approaches one, SOPD reduces to SFT; when the step length approaches one token, it approximates forward-KL OPD.\"}]","Step-Level On-Policy Distillation - Interpolating Between On-Policy Distillation and Supervised Fine-Tuning - Abstract | PDF"]