[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86054-en":3,"doc-seo-86054-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86054,687197207057,"Sage","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Predictive Divergence Masks for LLM RL","Reinforcement learning for large language models (LLMs) stabilizes off-policy updates using trust-region masks. A PPO-style rule masks tokens using the sampled-token importance ratio with proximity and direction criteria, while recent DPPO improves proximity by switching to a behavior–training probability divergence. This leaves the direction criterion as a single-sample proxy that can disagree in sign with the divergence change. The proposed predictive divergence mask predicts whether the next gradient step will increase or decrease the same divergence, using closed-form analysis for softmax policies and top-K estimators from rollout engines, improving RL training across scales and precision settings.","arXiv :2607 . 10848v 1 [ cs .LG] 12 Jul 2026  \nPredictive Divergence Masks for LLM RL  \nXiangxin Zhou 1 ,∗ Jiarui Yao 1 ,2 ,∗ Penghui Qi3 ,∗ Bowen Ping 1 Jiaqi Tang 1 Haonan Wang 1 Tianyu Pang 1 ,‡  \n1 Tencent Hunyuan 2 UIUC 3 NUS  \n∗ Equal contribution ‡Corresponding author  \nAbstract. Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away. Recent work DPPO improves the proximity criterion by replacing PPO’s ratio-based test with a probability divergence between the behavior and training policies. However, its direction criterion is still inherited from PPO. A token can be masked only when the sampled-token importance ratio moves away from one. We observe that this ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence that defines the proximity criterion. We therefore propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For the discrete softmax policies used in LLM RL, we derive this prediction in closed form. Because production rollout engines expose only a truncated (top-K ) view of the vocabulary, we develop two lightweight top-K estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.  \nDate: July 7, 2026  \n1 Introduction  \nReinforcement learning (RL) has become a central post-training tool for improving the reasoning and alignment behavior of large language models (LLMs) (Ouyang et al. , 2022 ; Guo et al. , 2025 ; Shao et al. , 2024) . In this setting, an LLM is optimized as an autoregressive token-level policy against scalar sequence-level rewards. In practice, however, the data used for optimization is not generated by exactly the same policy being updated. Rollouts are often produced by inference engines whose numerical behavior differs from the training stack, creating a training-inference mismatch (Qi et al. , 2025 ; Yao et al. , 2025) . Policy staleness adds another mismatch, since the policy is typically updated multiple times on minibatches drawn from a fixed batch of rollout data (Zheng et al. , 2025) . Together, these effects make practical LLM RL inherently off-policy.  \nWidely used methods (Schulman et al. , 2017 ; Shao et al. , 2024 ; Yu et al. , 2025 ; Chen et al. , 2025 ; Liu et al. , 2025b) therefore follow a common trust-region recipe of optimizing a clipped or masked surrogate objective on rollouts collected by the behavior policy. Such masks are asymmetric. Being outside the trust region is not sufficient for masking. A token is masked only when its advantageweighted update would further increase the relevant deviation from the behavior policy; updates that reduce this deviation remain active. Thus a mask has two components: the proximity criterion asks whether the trust region is exceeded, and the direction criterion asks whether the update points further outward. In Proximal Policy Optimization (PPO) (Schulman et al. , 2017), both decisions are made from the sampled token’s importance ratio. Once the ratio leaves a fixed window around one and the advantage pushes it further away, the token is masked and its gradient vanishes.  \nThe sampled importance ratio, however, is a poor proxy for distributional shift in LLMs (Qi et al. , 2026) . It observes only the sampled token and ignores how probability mass moves over the rest of the vocabulary. Recent divergence-mask methods such as DPPO therefore replace","cbCaif4kT3vVdGee","https://ap.wps.com/l/cbCaif4kT3vVdGee","pdf",1162980,1,22,"English","en",105,"# Introduction\n## Trust-region masking and PPO-style criteria\n## Divergence-mask methods and the direction inconsistency\n## Predictive divergence mask and closed-form prediction\n## Top-K estimators for rollout engines","[{\"question\":\"Why does PPO-style token masking rely on a sampled-token importance ratio?\",\"answer\":\"PPO decides masking using whether the importance ratio leaves a window around one and whether the advantage pushes it farther away from the behavior policy.\"},{\"question\":\"What inconsistency does the document highlight in divergence-mask methods like DPPO?\",\"answer\":\"DPPO changes the proximity criterion to use behavior–training divergence, but it still inherits PPO’s direction criterion based on the sampled-token ratio. This ratio can disagree in sign with the actual divergence change.\"},{\"question\":\"How does the predictive divergence mask improve token masking for LLM RL?\",\"answer\":\"It predicts whether the next policy-gradient step will increase or decrease the same divergence used by the trust region, deriving a closed-form first-order change for softmax policies and using lightweight top-K estimators when only truncated token views are available.\"}]",1784208114,55,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"predictive-divergence-masks-for-llm-rl","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/predictive-divergence-masks-for-llm-rl/86054/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why does PPO-style token masking rely on a sampled-token importance ratio?","Question",{"text":74,"@type":75},"PPO decides masking using whether the importance ratio leaves a window around one and whether the advantage pushes it farther away from the behavior policy.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What inconsistency does the document highlight in divergence-mask methods like DPPO?",{"text":79,"@type":75},"DPPO changes the proximity criterion to use behavior–training divergence, but it still inherits PPO’s direction criterion based on the sampled-token ratio. This ratio can disagree in sign with the actual divergence change.",{"name":81,"@type":72,"acceptedAnswer":82},"How does the predictive divergence mask improve token masking for LLM RL?",{"text":83,"@type":75},"It predicts whether the next policy-gradient step will increase or decrease the same divergence used by the trust region, deriving a closed-form first-order change for softmax policies and using lightweight top-K estimators when only truncated token views are available.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]