TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

Jul 16, 2026

Upload by Liam

On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student’s trajectories, but long-horizon agentic tasks are not sufficiently addressed. TurnOPD targets two inefficiencies in vanilla agent OPD: wasted compute on tail turns with weak/noisy KL supervision, and trajectory-level KL objectives that overfit shallow tokens while under-training deeper decision turns. TurnOPD introduces adaptive rollout-depth budgeting and progressive turn-normalized loss budgeting. Experiments on ALFWorld, WebShop, and Multi-Hop Search show improved accuracy under equal wall-clock budgets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

Introduction

Key inefficiencies in vanilla agent OPD

Proposed TurnOPD approach

Turn-level budgeting controllers

Experiments and results

arXiv :2607 .05804v 1 [ cs .AI ] 7 Jul 2026 TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training Yuhang Zhou1,2, Kai Zheng2,*,†, Haoling Li2, Dengyun Peng1, Can Xu2,†, Jingjing Chen1,† 1Fudan University, 2Tencent Hunyuan Abstract On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student’sown trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the losson shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy–time frontier beyond vanilla OPD. Correspondence: [ralph.yh.zhou@gmail.com](ralph.yh.zhou@gmail.com), {kevinezheng,[leocaxu}@tencent.com](leocaxu}@tencent.com), [chenjingjing@fudan.edu.cn](chenjingjing@fudan.edu.cn) 1 Introduction Language models are increasingly deployed as agents for planning, tool use, and environment interaction [4, 7, 16, 17, 38] . On-policy distillation (OPD) offers a promising training framework for such agentic models [14, 15], where a student samples rollouts and a stronger teacher supervises via a reverse-KL objective at visited states. This keeps supervision on-policy and provides dense feedback while avoiding sparse reward signals. Yet, applying OPD to long-horizon agent tasks is challenging because these are more than simple sequencegeneration problems [24, 36] . Agent rollouts involve multiple turns, tool calls, environmental shifts, and state changes. Early decisions impact all subsequent states, and later turns often contain key but infrequent decisions, so token-level feedback alone may not provide proper supervision for all crucial points. We systematically analyze OPD in long-horizon agent tasks, asking: How are supervision signals and optimization budgets allocated along the interaction trajectory, and with what consequences? Our diagnosis identifies two main problems: (1) Per-turn KL distribution: The reverse-KL signal is heavily skewed toward early turns, changes over training, and becomes less informative for deeper steps. Student and teacher outputs often converge as ∗ Project lead. †Corresponding authors. Vanilla OPD | A External mismatch Long-horizon agent trajectory over rounds r1 r2 r3 r4 ... rT Shallow rounds: frequent & shallow Deep rounds: low-signal tail & critical decisions | Reverse KL signal vs. round depth | | | | --- | --- | --- | --- | | | KL Strength | | | | | | 1 | Rollout Depth rT | | Fixed full-length rollout Informative front segment Low-signal tail (wasted compute) Fixed fulllength rollout wastes compute | | | | TurnOPD C Adaptive rollout depth controller | D Linear round-normalization mixing Gradually shift | | | | | --- | --- | --- | --- | | Lmix = (1-a)Ltoken + a Lround Increase linearly 0 0% 50% 100% Training progress | Share of KL loss High Early training ... Low ~~ ~~~~ ~~~~ ~~~~ ~~ 1 Round depth rT | High Late training | |

FAQs

What problem does TurnOPD address in long-horizon agent training with on-policy distillation?

TurnOPD addresses two inefficiencies: wasted wall-clock compute on low-signal tail turns and trajectory-level KL objectives that concentrate loss on shallow tokens, leaving deeper decisions under-trained.

How does TurnOPD improve rollout efficiency?

It uses an adaptive rollout-depth controller that selects rollout length dynamically via probe-based turn statistics, guided by survivor-weighted KL and coverage thresholds.

How does TurnOPD balance the KL loss across turns?

It applies progressive turn-normalized loss budgeting, gradually shifting KL weighting from token-level aggregation toward turn-balanced supervision across training.

Jul 16, 2026

Upload by Liam