[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84858-en":3,"doc-seo-84858-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84858,8796095461564,"Liam","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training","On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student’s trajectories, but long-horizon agentic tasks are not sufficiently addressed. TurnOPD targets two inefficiencies in vanilla agent OPD: wasted compute on tail turns with weak/noisy KL supervision, and trajectory-level KL objectives that overfit shallow tokens while under-training deeper decision turns. TurnOPD introduces adaptive rollout-depth budgeting and progressive turn-normalized loss budgeting. Experiments on ALFWorld, WebShop, and Multi-Hop Search show improved accuracy under equal wall-clock budgets.","arXiv :2607 .05804v 1 [ cs .AI ] 7 Jul 2026  \nTurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training  \nYuhang Zhou1,2, Kai Zheng2,*,†, Haoling Li2, Dengyun Peng1, Can Xu2,†, Jingjing Chen1,†  \n1Fudan University, 2Tencent Hunyuan  \nAbstract  \nOn-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student’sown trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the losson shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy–time frontier beyond vanilla OPD.  \nCorrespondence: [ralph.yh.zhou@gmail.com](ralph.yh.zhou@gmail.com), {kevinezheng,[leocaxu}@tencent.com](leocaxu}@tencent.com), [chenjingjing@fudan.edu.cn](chenjingjing@fudan.edu.cn)  \n1 Introduction  \nLanguage models are increasingly deployed as agents for planning, tool use, and environment interaction [4, 7, 16, 17, 38] . On-policy distillation (OPD) offers a promising training framework for such agentic models [14, 15], where a student samples rollouts and a stronger teacher supervises via a reverse-KL objective at visited states. This keeps supervision on-policy and provides dense feedback while avoiding sparse reward signals.  \nYet, applying OPD to long-horizon agent tasks is challenging because these are more than simple sequencegeneration problems [24, 36] . Agent rollouts involve multiple turns, tool calls, environmental shifts, and state changes. Early decisions impact all subsequent states, and later turns often contain key but infrequent decisions, so token-level feedback alone may not provide proper supervision for all crucial points.  \nWe systematically analyze OPD in long-horizon agent tasks, asking: How are supervision signals and optimization budgets allocated along the interaction trajectory, and with what consequences? Our diagnosis identifies two main problems: (1) Per-turn KL distribution: The reverse-KL signal is heavily skewed toward early turns, changes over training, and becomes less informative for deeper steps. Student and teacher outputs often converge as  \n∗ Project lead.  \n†Corresponding authors.  \nVanilla OPD  \n\n|  A  External mismatch\u003Cbr>Long-horizon agent trajectory over rounds\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>r1 r2 r3 r4 ... rT\u003Cbr>\u003Cbr>Shallow rounds:\u003Cbr>frequent & shallow\u003Cbr>\u003Cbr>Deep rounds:\u003Cbr>low-signal tail &\u003Cbr>critical decisions | Reverse KL signal vs. round depth |  |  |\n| --- | --- | --- | --- |\n|  | KL Strength |  |  |\n|  |  | 1 | Rollout Depth rT |\n| \u003Cbr>Fixed full-length rollout\u003Cbr>\u003Cbr>\u003Cbr>Informative front segment Low-signal tail (wasted compute)\u003Cbr>Fixed fulllength rollout wastes compute |  |  |  |\n\nTurnOPD  \n C  Adaptive rollout depth controller  \n| D Linear round-normalization mixing Gradually shift |  |  |  |\n| --- | --- | --- | --- |\n| \u003Cbr>Lmix = (1-a)Ltoken + a Lround\u003Cbr>Increase linearly\u003Cbr>\u003Cbr>0 0% 50% 100% Training progress | Share of\u003Cbr>KL loss High\u003Cbr>Early training\u003Cbr>...\u003Cbr>Low ~~ ~~~~ ~~~~ ~~~~ ~~\u003Cbr>\u003Cbr>1\u003Cbr>Round depth\u003Cbr>rT | High Late training\u003Cbr> |  |","cbCaifosx02XQMNt","https://ap.wps.com/l/cbCaifosx02XQMNt","pdf",2835655,1,26,"English","en",105,"# Introduction\n## Key inefficiencies in vanilla agent OPD\n## Proposed TurnOPD approach\n## Turn-level budgeting controllers\n# Experiments and results","[{\"question\":\"What problem does TurnOPD address in long-horizon agent training with on-policy distillation?\",\"answer\":\"TurnOPD addresses two inefficiencies: wasted wall-clock compute on low-signal tail turns and trajectory-level KL objectives that concentrate loss on shallow tokens, leaving deeper decisions under-trained.\"},{\"question\":\"How does TurnOPD improve rollout efficiency?\",\"answer\":\"It uses an adaptive rollout-depth controller that selects rollout length dynamically via probe-based turn statistics, guided by survivor-weighted KL and coverage thresholds.\"},{\"question\":\"How does TurnOPD balance the KL loss across turns?\",\"answer\":\"It applies progressive turn-normalized loss budgeting, gradually shifting KL weighting from token-level aggregation toward turn-balanced supervision across training.\"}]",1784198856,66,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon-agent-training","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon-agent-training/84858/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does TurnOPD address in long-horizon agent training with on-policy distillation?","Question",{"text":75,"@type":76},"TurnOPD addresses two inefficiencies: wasted wall-clock compute on low-signal tail turns and trajectory-level KL objectives that concentrate loss on shallow tokens, leaving deeper decisions under-trained.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does TurnOPD improve rollout efficiency?",{"text":80,"@type":76},"It uses an adaptive rollout-depth controller that selects rollout length dynamically via probe-based turn statistics, guided by survivor-weighted KL and coverage thresholds.",{"name":82,"@type":73,"acceptedAnswer":83},"How does TurnOPD balance the KL loss across turns?",{"text":84,"@type":76},"It applies progressive turn-normalized loss budgeting, gradually shifting KL weighting from token-level aggregation toward turn-balanced supervision across training.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]