Weak-to-Strong Generalization via Direct On-Policy Distillation

Jul 16, 2026

Uploaded by Asher

Reinforcement learning with verifiable rewards (RLVR) can improve language-model reasoning, but repeating it for every newly scaled strong model is costly because training requires many rollouts from the target policy. This work studies a weak-to-strong alternative: perform RL on a cheaper smaller model, then transfer what the RL run learned to train a stronger target. Direct On-Policy Distillation (Direct-OPD) transfers the teacher’s RL-induced policy shift as an implicit dense reward on student on-policy states, avoiding sparse-reward RL on the target. Experiments show consistent gains, including Qwen3-1.7B improving from 48.3% to 58.3% on AIME 2024 in 4 hours on 8 A100 GPUs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

AccessFree to download

Introduction

Weak-to-strong post-training motivation

Direct-OPD concept and policy shift

Weak model checkpoints as supervision signal

arXiv :2607 .05394v2 [ cs .LG] 8 Jul 2026 Weak-to-Strong Generalization via Direct On-Policy Distillation Shiyuan Feng* 1 ,2 ,4 , Huan-ang Gao*,‡1 ,2 ,3 , Haohan Chi* 1 ,2 ,3 , Hanlin Wu1 ,2 , Zhilong Zhang1 ,2 , Zheng Jiang3 , Bingxiang He3 , Wei-Ying Ma1 ,2 , Ya-Qin Zhang1 ,2 , Hao Zhou† 1 ,2 1 SIA-Lab of Tsinghua AIR and ByteDance Seed 2 Institute for AI Industry Research (AIR), Tsinghua University 3 Department of Computer Science and Technology, Tsinghua University 4 Peking University Abstract Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving languagemodel reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher’s final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher’s RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student’sown on-policy states. This directly reuses the weak model’s RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate. Date: July 9, 2026 Correspondence: [zhouhao@air.tsinghua.edu.cn](zhouhao@air.tsinghua.edu.cn) Project Page: [https://bytedtsinghua-sia.github.io/Direct-OPD/](https://bytedtsinghua-sia.github.io/Direct-OPD/) 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become a dominant post-training recipe for eliciting strong reasoning in large language models [1, 2] . Its cost, however, is tied to the model being trained: every update requires the current policy to generate rollouts, receive verifiable outcome scores, and update on those trajectories. Larger target models make each rollout slower and each RL iteration more expensive. Thus, as reasoning models continue to scale, re-running RLVR from scratch for every new strong model risks making post-training itself a bottleneck. In this paper, we propose Direct On-Policy Distillation (Direct-OPD), a weak-to-strong post-training paradigm. * Equal contribution. ‡Project Lead. †Corresponding author. AIME 24 ave@32 62 58 54 50 (a) Direct-OPD vs. OPD 0 100 200 300 Training step AIME 24 ave@32 80 70 60 50 40 Figure 1 Direct-OPD transfers the effect of small-model RL rather than imitating the small model. (a) Starting from R1-Distill-7B, vanilla OPD toward the post-RL JustRL-1.5B teacher degrades performance, whereas Direct-OPD transfers the JustRL-1.5B − R1-Distill-1.5B policy shift and improves the student. (b) The same policy shift improves Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B on AIME 2024, including students whose initial accuracy already exceeds the post-RL teacher. The setting is simple: run RL where it is cheap, on a weak model, and use what that RL run learned to train a stronger target model. Crucially, the weak model is not assumed to be a better reasoner than the student; it is merely an inexpensive vehicle whi

FAQs

Why is RLVR expensive when scaling to new strong language models?

RLVR cost depends on the model being trained because each RL update requires generating rollouts with the current policy, scoring them with verifiable outcomes, and updating on those trajectories. Larger target models make each rollout slower and each RL iteration more expensive, so repeating RLVR from scratch becomes a bottleneck.

What does Direct-OPD transfer from the weak model to the strong model?

Direct-OPD transfers the weak model’s RL-induced policy shift rather than distilling the weak model’s final post-RL policy. It contrasts the weak model before and after RL and uses the log-ratio shift as an implicit dense reward signal applied to the student’s own on-policy states.

How does Direct-OPD avoid sparse-reward RL on the target model?

Direct-OPD applies the transferred RL signal directly during on-policy training of the stronger student states, anchored by a KL term to the student initialization. It does not train an explicit reward model or run sparse-reward RL on the target.

Jul 16, 2026

Uploaded by Asher