[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84437-en":3,"doc-seo-84437-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84437,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","Nested ReFT Efficient Reinforcement Learning for Large Language Model Fine Tuning via Off Policy Rollouts","Standard Reinforced Fine-Tuning (ReFT) views off-policyness—the mismatch between behavior and target policies—as a harmful quantity to minimize. Nested-ReFT reverses this assumption by deliberately inducing controlled off-policyness during training. Nested behavior policies are built as nested instances of the target model to synchronize parameters automatically. The method supports unbiased gradient estimates under ensemble behavior policies and is validated across scales (1.5B, 7B) and domains (math, code).","Nested-ReFT: Efficient Reinforcement Learning for Large Language Model  \nFine-Tuning via Off-Policy Rollouts  \nMaxime Heuillet 1 2 Yufei Cui 3 Boxing Chen 3 Audrey Durand 1 2 4 Prasanna Parthasarathi 3  \narXiv :2508 . 10 123v 3 [ cs .LG] 12 Jul 2026  \nAbstract  \nStandard Reinforced Fine-Tuning (ReFT) treats off-policyness—the discrepancy between behavior and target policies—as a detrimental artifact to be minimized. We challenge this dogma with Nested-ReFT, a framework that deliberately induces controlled off-policyness during training.  \nBy constructing behavior policies as nested instances of the target model, we achieve automatic parameter synchronization. Theoretically, we prove that this formulation yields unbiased gradient estimates under ensemble behavior policies. Empirically, we conduct a systematic study across model scales (1.5B, 7B) and domains (math, code), establishing three key findings: (1) LLMs exhibit ”emergent robustness” to structural off-policyness, particularly at larger scales; (2) Mixup Nesting combined with Retrace variance reduction effectively stabilizes the off-policy gap;  \nand (3) Nested-ReFT reduces rollout time by with minimal performance impact. Our work establishes off-policy intensity not as an error to be avoided, but as a tunable design dimension for efficient ReFT.  \n1. Introduction  \nLarge language models (LLMs) are increasingly capable at solving complex reasoning problems (Cobbe et al., 2021) . This progress is partly driven by LLMs’ ability to generate chain-of-thought (CoT) completions, which include the final response to a problem as well as the intermediate reasoning steps helpful to reach that response (Wei et al., 2022) . To improve the generalization performance of LLMs, an increasingly popular post-training technique consists of using an LLM to generate CoT completions and use them for  \n1Universit Laval (IID), Canada 2Mila-Qubec AI Institute, Canada 3Huawei Noah’s Ark Lab (Montreal Research Center), Canada 4 Canada CIFAR AI Chair. Correspondence to: Maxime Heuillet \u003C[maxime.heuillet.1@ulaval.ca](maxime.heuillet.1@ulaval.ca) > .  \nPreliminary work. Accepted at Neurips 2025 Workshop on Efficient Learning.  \nfine-tuning (Kumar et al., 2025 ; Shao et al., 2024 ; Xie et al., 2024 ; Silver et al., 2016) .  \nSuch post-training techniques are dubbed reinforced finetuning (ReFT) and are based on reinforcement learning (Sutton & Barto, 1998) . The CoT completions are generated by a behavior LLM policy through a process referred to as rollout. Then, the completions are scored using a reward function. In domains with verifiable rewards (e.g., math, programming), the reward function is a simple heuristic (Trung et al., 2024 ; Shao et al., 2024 ; Liu et al., 2025) . The scored completions are then used to propagate gradients back to fine-tune the target LLM policy.  \nThe rollout step in ReFT generates a substantial amount of new training data to fine-tune the target LLM through inference. Appealing generalization gains are achievable using ReFT (Trung et al., 2024), yet the rollout step comes with a high computational cost. While completions are easily verifiable and thus rewards are cheap to acquire, the computational cost of generating completions from a behavior LLM policy can be substantial (Kazemnejad et al., 2025 ; Shao et al., 2024) . This completion generation cost can add up significantly to the compute cost of updating the target LLM’s parameters.  \nPractitioners have explored several avenues to decrease this cost. Open-source frameworks like TRL (von Werra et al., 2020a) allow doing rollouts with the behavior LLM loaded with VLLM backend (Kwon et al., 2023), which specializes in inference acceleration. Recently, Zhang et al. (2026) propose using speculative decoding to accelerate rollout inference. However, both strategies introduce discrepancies between the behavior and target policy, which can alter training stability. Indeed, having a distributional gap between the behavior and","cbCaiq7Y1tFn28Pi","https://ap.wps.com/l/cbCaiq7Y1tFn28Pi","pdf",1556004,1,12,"English","en",105,"# Abstract\n# Introduction\n## Research Gap and Questions\n## Contributions","[{\"question\":\"What problem does Nested-ReFT address in standard ReFT?\",\"answer\":\"Standard ReFT treats the off-policy discrepancy between behavior and target policies as something to minimize for stability. Nested-ReFT challenges this by treating off-policyness as a controllable design factor during training.\"},{\"question\":\"How does Nested-ReFT construct behavior policies during training?\",\"answer\":\"It dynamically builds behavior LLM policies as nested instances of the target model using on-the-fly instantiation from the target model’s current parameters, enabling automatic parameter tracking.\"},{\"question\":\"What empirical results are reported for Nested-ReFT?\",\"answer\":\"Across model scales (1.5B, 7B) and domains (math, code), the work reports emergent robustness to structural off-policyness, stabilization of the off-policy gap via Mixup Nesting and Retrace variance reduction, and reduced rollout time with minimal performance impact.\"}]",1784195625,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"nested-reft-efficient-reinforcement-learning-for-large-language-model-fine-tuning-via-off-policy-rollouts","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/nested-reft-efficient-reinforcement-learning-for-large-language-model-fine-tuning-via-off-policy-rollouts/84437/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does Nested-ReFT address in standard ReFT?","Question",{"text":74,"@type":75},"Standard ReFT treats the off-policy discrepancy between behavior and target policies as something to minimize for stability. Nested-ReFT challenges this by treating off-policyness as a controllable design factor during training.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does Nested-ReFT construct behavior policies during training?",{"text":79,"@type":75},"It dynamically builds behavior LLM policies as nested instances of the target model using on-the-fly instantiation from the target model’s current parameters, enabling automatic parameter tracking.",{"name":81,"@type":72,"acceptedAnswer":82},"What empirical results are reported for Nested-ReFT?",{"text":83,"@type":75},"Across model scales (1.5B, 7B) and domains (math, code), the work reports emergent robustness to structural off-policyness, stabilization of the off-policy gap via Mixup Nesting and Retrace variance reduction, and reduced rollout time with minimal performance impact.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":120},"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]