Nested ReFT Efficient Reinforcement Learning for Large Language Model Fine Tuning via Off Policy Rollouts

Jul 16, 2026

Upload by Logic

Standard Reinforced Fine-Tuning (ReFT) views off-policyness—the mismatch between behavior and target policies—as a harmful quantity to minimize. Nested-ReFT reverses this assumption by deliberately inducing controlled off-policyness during training. Nested behavior policies are built as nested instances of the target model to synchronize parameters automatically. The method supports unbiased gradient estimates under ensemble behavior policies and is validated across scales (1.5B, 7B) and domains (math, code).

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Research Gap and Questions

Contributions

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts Maxime Heuillet 1 2 Yufei Cui 3 Boxing Chen 3 Audrey Durand 1 2 4 Prasanna Parthasarathi 3 arXiv :2508 . 10 123v 3 [ cs .LG] 12 Jul 2026 Abstract Standard Reinforced Fine-Tuning (ReFT) treats off-policyness—the discrepancy between behavior and target policies—as a detrimental artifact to be minimized. We challenge this dogma with Nested-ReFT, a framework that deliberately induces controlled off-policyness during training. By constructing behavior policies as nested instances of the target model, we achieve automatic parameter synchronization. Theoretically, we prove that this formulation yields unbiased gradient estimates under ensemble behavior policies. Empirically, we conduct a systematic study across model scales (1.5B, 7B) and domains (math, code), establishing three key findings: (1) LLMs exhibit ”emergent robustness” to structural off-policyness, particularly at larger scales; (2) Mixup Nesting combined with Retrace variance reduction effectively stabilizes the off-policy gap; and (3) Nested-ReFT reduces rollout time by with minimal performance impact. Our work establishes off-policy intensity not as an error to be avoided, but as a tunable design dimension for efficient ReFT. 1. Introduction Large language models (LLMs) are increasingly capable at solving complex reasoning problems (Cobbe et al., 2021) . This progress is partly driven by LLMs’ ability to generate chain-of-thought (CoT) completions, which include the final response to a problem as well as the intermediate reasoning steps helpful to reach that response (Wei et al., 2022) . To improve the generalization performance of LLMs, an increasingly popular post-training technique consists of using an LLM to generate CoT completions and use them for 1Universit Laval (IID), Canada 2Mila-Qubec AI Institute, Canada 3Huawei Noah’s Ark Lab (Montreal Research Center), Canada 4 Canada CIFAR AI Chair. Correspondence to: Maxime Heuillet <[maxime.heuillet.1@ulaval.ca](maxime.heuillet.1@ulaval.ca) > . Preliminary work. Accepted at Neurips 2025 Workshop on Efficient Learning. fine-tuning (Kumar et al., 2025 ; Shao et al., 2024 ; Xie et al., 2024 ; Silver et al., 2016) . Such post-training techniques are dubbed reinforced finetuning (ReFT) and are based on reinforcement learning (Sutton & Barto, 1998) . The CoT completions are generated by a behavior LLM policy through a process referred to as rollout. Then, the completions are scored using a reward function. In domains with verifiable rewards (e.g., math, programming), the reward function is a simple heuristic (Trung et al., 2024 ; Shao et al., 2024 ; Liu et al., 2025) . The scored completions are then used to propagate gradients back to fine-tune the target LLM policy. The rollout step in ReFT generates a substantial amount of new training data to fine-tune the target LLM through inference. Appealing generalization gains are achievable using ReFT (Trung et al., 2024), yet the rollout step comes with a high computational cost. While completions are easily verifiable and thus rewards are cheap to acquire, the computational cost of generating completions from a behavior LLM policy can be substantial (Kazemnejad et al., 2025 ; Shao et al., 2024) . This completion generation cost can add up significantly to the compute cost of updating the target LLM’s parameters. Practitioners have explored several avenues to decrease this cost. Open-source frameworks like TRL (von Werra et al., 2020a) allow doing rollouts with the behavior LLM loaded with VLLM backend (Kwon et al., 2023), which specializes in inference acceleration. Recently, Zhang et al. (2026) propose using speculative decoding to accelerate rollout inference. However, both strategies introduce discrepancies between the behavior and target policy, which can alter training stability. Indeed, having a distributional gap between the behavior and

FAQs

What problem does Nested-ReFT address in standard ReFT?

Standard ReFT treats the off-policy discrepancy between behavior and target policies as something to minimize for stability. Nested-ReFT challenges this by treating off-policyness as a controllable design factor during training.

How does Nested-ReFT construct behavior policies during training?

It dynamically builds behavior LLM policies as nested instances of the target model using on-the-fly instantiation from the target model’s current parameters, enabling automatic parameter tracking.

What empirical results are reported for Nested-ReFT?

Across model scales (1.5B, 7B) and domains (math, code), the work reports emergent robustness to structural off-policyness, stabilization of the off-policy gap via Mixup Nesting and Retrace variance reduction, and reduced rollout time with minimal performance impact.

Jul 16, 2026

Upload by Logic