Every Coin Has Two Sides - On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models

Sep 15, 2026

Uploaded by Aurora

On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student’s own policy, but its generalization behavior is poorly understood. A controlled study varies one generalization factor at a time, covering in-domain distribution shifts, cross-domain transfer, and a multi-teacher setting. Results show OPD transfers reasoning behavior rather than fixed answers, is insensitive to training difficulty, and even leverages problems the teacher never solves.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages30

Reading Time1 h 16 min

AccessFree to download

Introduction

Research questions and controlled study design

Findings on in-domain shifts (RQ1)

Findings on cross-domain transfer (RQ2)

Implications for multi-teacher OPD (RQ3)

arXiv : 2608 . 16647v2 [cs .CL] 23 Aug 2026 Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models Zhaoyi Li1,3,∗ , Deyang Kong2,3,∗ , Yuan Wei3,∗ , Evan Yang3 , Ranran Shen1 , Mahardika Krisna Ihsani4 , Ming Yang3 , Wei Zhang3 , Chuan Hao3 , Jian Yang3 , Ran Tao3 , Bryan Dai3 , Shikun Zhang2 , Wei Ye2 , Ying Wei5 , Defu Lian1 1University of Science and Technology of China, 2 Peking University, 3 IQuest Research, 4 MBZUAI, 5 Zhejiang University ∗ Equal Contribution [lizhaoyi777@mail.ustc.edu.cn](lizhaoyi777@mail.ustc.edu.cn) , [kong.deyang@foxmail.com](kong.deyang@foxmail.com) Abstract On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student’s own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher’s reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher’s influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizesand offer a useful perspective for diagnosing multi-teacher OPD. 1. Introduction On-policy distillation (OPD) [1] has become increasingly important for transferring capabilities from strong teacher models to smaller or less capable students during large language model post-training [14, 50, 52] . Unlike conventional offline distillation, OPD samples trajectories from the student’s current policy and queries the teacher on states that the student actually visits. The resulting supervision therefore reduces the exposure bias [35] between the training trajectories and the student’s own generation behavior. Most existing studies evaluate OPD in a single training domain and on benchmarks closely related to the training data. Such evaluations establish that OPD can improve target-task performance, but they do not distinguish local fitting from broader policy transfer [8], a distinction critical for both understanding the mechanisms of OPD [28] and designing effective multi-teacher integration strategies [39] . In particular, it remains unclear how OPD behaves when training and evaluation differ in language, reasoning horizon or task domain. To fill this gap, we conduct a controlled study that varies one generalization factor at a time while holding the remaining conditions fixed, and organize it around three questions of increasing scope. RQ1: How robust is OPD to in-domain distribution shifts? Within a fixed domain (math), we study two aspects. First, we examine how the relative difficulty [6, 59, 62] of the training problems, measured by the teacher’s and the student’s pass rates, affects the transfer of the teacher’s in-domain performance [20, 62] . Second, keeping the domain fixed but shifting the evaluation distribution, in language [32, 48](English→Chinese) and reasoning horizon [36, 47](short→long-horizon composed problems), we examine how well the teacher’s performance transfers under such shifts. For RQ1, OPD is largely insensitive to training-problem difficulty: problems the teacher never solves are as useful as those it always solves, suggesting OPD conveys the teacher’s reasoning patterns rather than answers to particular prob

FAQs

What generalization behavior does on-policy distillation primarily transfer?

OPD transfers the teacher’s reasoning behavior rather than its answers to specific problems, including cases where the teacher never solves the problem during training.

How does training-problem difficulty affect OPD transfer in-domain?

Transfer is largely insensitive to training-problem difficulty: problems the teacher never solves can be as useful as those it always solves.

Why is multi-teacher OPD harder to control than single-teacher OPD?

Each teacher’s influence extends beyond its assigned domain, so combining domain experts produces a mixture-dependent “seesaw” among capabilities rather than independent domain skill composition.

Sep 15, 2026

Uploaded by Aurora