[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327151-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327151-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","dont-let-gains-fade-breaking-down-policy-gradient-weights-in-rl","Don’t Let Gains FADE - Breaking Down Policy Gradient Weights in RL","","Reinforcement learning post-training improves LLM reasoning but often suffers from training instability and diversity collapse. Advantage functions can reshape the objective by reweighting which rollouts drive learning, yet the proliferation of variants makes it unclear which one to choose. A unifying framework decomposes any advantage into positive and negative gradient mass along orthogonal axes. This reveals sign-imbalance and difficulty-related trade-offs. Based on training dynamics, FADE self-adapts its focal advantage schedule, reaching faster peak pass@1 and delivering strong accuracy-diversity trade-offs on LiveCodeBench and AIME.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/dont-let-gains-fade-breaking-down-policy-gradient-weights-in-rl/327151/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/dont-let-gains-fade-breaking-down-policy-gradient-weights-in-rl/327151.png","ImageObject",300,407,{"name":42,"@type":43},"Eliana","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-24","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",9,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper address in RL post-training for LLMs?","Question",{"text":63,"@type":64},"It addresses training instability and diversity collapse, which can limit improvements in language-model reasoning.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the paper unify different advantage functions?",{"text":68,"@type":64},"It decomposes any policy weight into positive and negative gradient mass along two orthogonal axes, separating effects related to sign imbalance and difficulty focus.",{"name":70,"@type":61,"acceptedAnswer":71},"What is FADE and how does it adapt during training?",{"text":72,"@type":64},"FADE (Focal Advantage with Dynamic Entropy) reads past training dynamics—such as entropies and solve rates—to schedule gradient weights, balancing early learning speed with sustained diversity and accuracy.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327151,1790293866,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145,"preview_page_count":4},4398048949847,"https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267","arXiv :2607 .0 1490v 1 [ cs .LG] 1 Jul 2026  \nDon’t Let Gains FADE: Breaking Down Policy Gradient Weights in RL  \nJuliette Decugis1 ,2 , Sean O’Brien∗ , Francis Bach2 , Gabriel Synnaeve1 , Taco Cohen 1  \n1 FAIR at Meta, 2 Inria, Ecole Normale Supérieure  \n∗Work done at Meta, now working at the University of California, San Diego  \nReinforcement learning post-training dramatically improves LLM reasoning, but suffers from training instability and diversity collapse. Advantage functions offer an appealing fix: they reshape the training objective, reweight which rollouts drive learning, and are trivial to implement. Yet a proliferation of methods makes it unclear which advantage to use and when. We cut through the confusion with a unifying framework that decomposes any advantage into its positive and negative gradient mass (mS , mF ) along two orthogonal axes. On the sign axis, imbalanced updates collapse either entropy or weight geometry. On the difficulty axis, hard-problem focus sharpens signal but costs sample size. Both trade-offs shift during training: exploration favors balance and hard focus; exploitation favors suppression and medium focus. This motivates FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage that reads training dynamics to schedule the gradient weight automatically. FADE reaches peak pass@1 20k steps earlier than the best static baseline at the 7B scale and 2k steps earlier at the 32B , while achieving the best accuracy-diversity trade-off across all pass@k on LiveCodeBench and AIME.  \nDate: July 3, 2026  \nCorrespondence: Juliette Decugis at [jdecugis@meta.com](jdecugis@meta.com)  \n1 Introduction  \nRecent developments in reinforcement learning with verifiable rewards (RLVR) have unlocked rapid gains in language models’ capabilities, especially in easily-verifiable domains like code generation and mathematics (OpenAI, 2024 ; Guo et al. , 2025 ; Shao et al. , 2024 ; Liu et al. , 2025a) . Although sparse rewards over long sequences make credit assignment challenging (Minsky, 1961 ; Sutton, 1988 ; Zhang, 2026), pretrained LLMs provide strong behavioral priors (Gan and Isola, 2026 ; Yan et al. , 2025) and fully resettable environments enable parallel rollout collection. These methods therefore follow a common recipe: sample multiple rollouts per problem, score them with a binary verifier, and update the policy via a weighted policy gradient (Williams, 1992 ; Schulman et al. , 2015) . The weights, commonly called “advantage functions,” rarely correspond to the classical advantage (value of an action minus the value of the average action in a state); they simply determine how much each rollout contributes to the gradient. To avoid this overloading, we use the term policy weights throughout this paper.  \nSince GRPO (Shao et al. , 2024), which uses the mean reward as a baseline for updates, a host of alternative policy weights have appeared: DAPO (Yu et al. , 2025), DR-GRPO (Liu et al. , 2025a), pass@k-based objectives (Tang et al. , 2025 ; Chen et al. , 2025), log-mean-exp weighting (Jiang et al. , 2025), and more. Each claims improvements, yet comparing them is difficult because they differ along multiple axes simultaneously. Consider for example the pass@8 normalization (Tang et al. , 2025), which only upweights a correct rollout when it is the sole success in a batch of eight. This simultaneously shifts gradient mass toward hard problems, drops all negative gradient signal since incorrect rollouts receive zero weight, and reduces the overall gradient magnitude because most batches contain either zero or more than one success. Other methods such as Skew-R (Thrampoulidis et al. , 2025) keep the sign balance of GRPO E [A] = 0 but emphasize high variance samples. When these methods under or outperform GRPO, it is unclear which change is responsible.  \nWe argue that the confusion stems from conflating three orthogonal design axes. Similar to Thrampoulidiset al. (2025), we decompose policy weights int","cbCainB1wJbdQS6z","https://ap.wps.com/l/cbCainB1wJbdQS6z","pdf",2013390,33,"English","# Introduction\n## Framework for Policy Weight Analysis","[{\"question\":\"What problem does the paper address in RL post-training for LLMs?\",\"answer\":\"It addresses training instability and diversity collapse, which can limit improvements in language-model reasoning.\"},{\"question\":\"How does the paper unify different advantage functions?\",\"answer\":\"It decomposes any policy weight into positive and negative gradient mass along two orthogonal axes, separating effects related to sign imbalance and difficulty focus.\"},{\"question\":\"What is FADE and how does it adapt during training?\",\"answer\":\"FADE (Focal Advantage with Dynamic Entropy) reads past training dynamics—such as entropies and solve rates—to schedule gradient weights, balancing early learning speed with sustained diversity and accuracy.\"}]","Don’t Let Gains FADE - Breaking Down Policy Gradient Weights in RL | PDF",1789976555,83]