[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85931-en":3,"doc-seo-85931-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85931,7971461740909,"Levi","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples","Reinforcement learning improves the reasoning abilities of large language models, yet training often becomes fragile due to over-optimization, where models exploit heuristics and lose generalizable reasoning. Reverse KL regularization is analyzed as insufficient because it cannot guarantee coverage of the reference distribution. ARMOR (Anchor Rollout and Mixed Optimization for RL) stabilizes samples using off-policy anchor rollout and mixed optimization, reducing validation collapse and sustaining gains over extended training.","ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples  \nKexin Huang1 , Junkang Wu1 , Jinda Lu1 , Yang Shuo2 , Chiyu Ma3 , Jiancan Wu1 , Xiang Wang1 , Xiangnan He1 , Guoyin Wang, Jingren Zhou  \n1University of Science and Technology of China 2Peking University 3Dartmouth College  \n[huangkx@mail.ustc.edu.cn](huangkx@mail.ustc.edu.cn), {xiangwang1223, [xiangnanhe}@gmail.com](xiangnanhe}@gmail.com)  \narXiv :2607 . 1048 1v 1 [ cs .LG] 11 Jul 2026  \nAbstract  \nReinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile. In this work, we investigate a critical source of this instability:  \nover-optimization, where models exploit training heuristics at the expense of generalizable reasoning. While reverse KL regularization is the standard defense against such degradation, our analysis reveals that it is often insufficient in this regime, as it fails to ensure comprehensive coverage of the reference distribution. To address this, we propose ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization. ARMOR comprises two key components: (1) Anchor Rollout, which leverages off-policy data from the reference policy to preserve established solution patterns; and (2) Mixed Optimization, which reformulates the policy objective to enable controlled exploration without relying on auxiliary losses. Extensive experiments on reasoning benchmarks validate that ARMOR effectively mitigates validation collapse, enabling sustained performance improvements over extended training horizons.  \n1 Introduction  \nReinforcement learning has become a central algorithmic driver of recent advances in large language models, substantially enhancing their ability to solve complex tasks and enabling a new class of reasoning-focused models such as OpenAI o1 (Jaech et al., 2024), DeepSeek R1 (Guo et al., 2025), and Qwen3 (Yang et al., 2025) . Yet, scaling RL for reasoning over long horizons remains fragile: stable optimization can still yield unstable generalization.  \nWhile the community has extensively addressed system-level instabilities, such as the traininginference mismatch (Zheng et al., 2025b ; Zhao  \nTraining Reward & Length of DAPO  \nLength  \nReward  \n0.4  \n0.2  \n0.0100 200 300  \nTraining Steps  \n(a) Training behaviors  \nARMOR   \nBaseline  \n(b) Validation score (AIME24)  \nFigure 1: Illustration of the over-optimization issue and the efficacy of ARMOR (on Qwen2.5-Math-7B) . (a) Training reward and response length steadily rise, indicating stable optimization. (b) However, validation performance reveals a critical generalization failure: the baseline performance degrades after an initial ascent. By integrating anchor rollout to stabilize the sample distribution and mixed optimization to enhance exploration, ARMOR prevents degradation and sustains continuous performance gains.  \net al., 2025a ; Ma et al., 2025 ; Zheng et al., 2025a)—an architectural inconsistency that disrupts optimization (Yao et al., 2025 ; Liu et al., 2025b)—we focus on a more fundamental algorithmic failure: over-optimization (Gao et al., 2023) . As illustrated in Fig. 1, we observe a distinctive reward– validation gap: training reward (answer correctness) improves steadily, yet validation performance decouples and degrades. In this regime, stable reward curves can even be a warning sign rather thana reassurance. This phenomenon, echoed in recent works (Mao et al., 2025 ; Zhang et al., 2025a), indicates a breakdown in generalization rather than an optimization collapse.  \nCrucially, this failure differs from the classical form of reward hacking that arises from a discrepancy between a proxy reward and the true evaluation metric (Weng, 2024) . In reasoning tasks with verifiable rewards (Lambert et al., 2024), the reward signal remains consistent across training and validation. Nevertheless, t","cbCaisjz9flrUruU","https://ap.wps.com/l/cbCaisjz9flrUruU","pdf",738540,1,15,"English","en",105,"# Introduction\n## Over-optimization and reward–validation gap\n## Limits of reverse KL regularization\n## ARMOR framework: Anchor Rollout and Mixed Optimization","[{\"question\":\"What problem does ARMOR address in on-policy LLM reinforcement learning?\",\"answer\":\"ARMOR targets over-optimization, where models increase training rewards by exploiting heuristics while validation performance degrades due to poor generalization.\"},{\"question\":\"Why is reverse KL regularization often insufficient in the over-optimization regime?\",\"answer\":\"Reverse KL suffers from two limitations: it can be mode-seeking, collapsing onto shortcuts without covering the reference distribution, and it applies a uniform penalty that dampens exploration needed to find better reasoning paths.\"},{\"question\":\"How do Anchor Rollout and Mixed Optimization work in ARMOR?\",\"answer\":\"Anchor Rollout injects off-policy samples from the reference policy during rollout to preserve established generalizable solution patterns. Mixed Optimization removes the auxiliary penalty and optimizes a mixture objective α·πθ+(1−α)·πref to enable controlled exploration via an adaptive trust region.\"}]",1784207232,38,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"armor-stabilizing-on-policy-llm-rl-with-off-policy-anchor-samples","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/armor-stabilizing-on-policy-llm-rl-with-off-policy-anchor-samples/85931/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does ARMOR address in on-policy LLM reinforcement learning?","Question",{"text":74,"@type":75},"ARMOR targets over-optimization, where models increase training rewards by exploiting heuristics while validation performance degrades due to poor generalization.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Why is reverse KL regularization often insufficient in the over-optimization regime?",{"text":79,"@type":75},"Reverse KL suffers from two limitations: it can be mode-seeking, collapsing onto shortcuts without covering the reference distribution, and it applies a uniform penalty that dampens exploration needed to find better reasoning paths.",{"name":81,"@type":72,"acceptedAnswer":82},"How do Anchor Rollout and Mixed Optimization work in ARMOR?",{"text":83,"@type":75},"Anchor Rollout injects off-policy samples from the reference policy during rollout to preserve established generalizable solution patterns. Mixed Optimization removes the auxiliary penalty and optimizes a mixture objective α·πθ+(1−α)·πref to enable controlled exploration via an adaptive trust region.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]