[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-503760-105":59,"doc-detail-503760-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","rollmux-phase-level-multiplexing-for-disaggregated-rl-post-training","ROLLMUX: Phase-Level Multiplexing for Disaggregated RL Post-Training","","Rollout–training disaggregation is emerging as the standard architecture for reinforcement learning (RL) post-training, separating memory-bound rollout and compute-bound training across purpose-built clusters. On-policy synchronization, however, creates severe dependency bubbles, forcing idle time in one phase while the other runs. ROLLMUX introduces cross-cluster orchestration using co-execution groups with two-tier scheduling, plus a residency constraint for warm-start switching. Evaluations on 328 H20 and 328 H800 GPUs show 1.84× better cost efficiency than standard disaggregation and 1.38× over colocated baselines, while achieving 100% SLO attainment.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/rollmux-phase-level-multiplexing-for-disaggregated-rl-post-training/503760/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/rollmux-phase-level-multiplexing-for-disaggregated-rl-post-training/503760.png","ImageObject",300,407,{"name":92,"@type":93},"Valentina","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-12","2026-10-03",true,{"@type":102,"interactionType":103,"userInteractionCount":29},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What problem does ROLLMUX address in disaggregated RL post-training?","Question",{"text":112,"@type":113},"ROLLMUX targets dependency bubbles caused by strict synchronization in on-policy RL, where one cluster idles while the other executes a dependent phase.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does ROLLMUX reduce dependency bubbles across clusters?",{"text":117,"@type":113},"It orchestrates multiple RL jobs into a co-execution group, weaving their rollout and training phases across rollout and training pools to enable efficient time-multiplexing.",{"name":119,"@type":110,"acceptedAnswer":120},"What scheduling mechanisms and constraints does ROLLMUX use?",{"text":121,"@type":113},"It uses a two-tier scheduling architecture: an inter-group scheduler based on conservative stochastic planning and an intra-group scheduler using a provably optimal round-robin schedule, along with a residency constraint to keep model states cached for warm-start switching.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},503760,1791306253,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":29,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},13056703020460,"https://ap-avatar.wpscdn.com/avatar/be000253dac470eee5d?_k=1778207105932848923","arXiv :2512 . 11306v2 [ cs .DC] 15 Dec 2025  \nROLL  \nROLLMUX: Phase-Level Multiplexing for Disaggregated RL Post-Training  \nTianyuan Wu†, Lunxi Cao†, Yining Wei‡, Wei Gao†, Yuheng Zhao†, Dakai An†, Shaopan Xiong§, Zhiqiang Lv§, Ju Huang§, Siran Yang§, Yinghao Yu§, Jiamang Wang§, Lin Qu§, Wei Wang†† Hong Kong University of Science and Technology, ‡ UIUC,§Alibaba Group  \nAbstract  \nRollout–training disaggregation is emerging as the standard architecture for Reinforcement Learning (RL) post-training, where memory-bound rollout and compute-bound training are physically disaggregated onto purpose-built clusters to maximize hardware efficiency. However, the strict synchronization required by on-policy algorithms introduces severe dependency bubbles, forcing one cluster to idle while the dependent phase is running on the other. We present ROLLMUX, a cluster scheduling framework that reclaims these bubbles through cross-cluster orchestration. ROLLMUX is built on the insight that the structural idleness of one job can be effectively utilized by the active phase of another. To realize this, we introduce the co-execution group abstraction, which partitions the cluster into isolated locality domains. This abstraction enables a two-tier scheduling architecture: an inter-group scheduler that optimizes job placement using conservative stochastic planning, and an intra-group scheduler that orchestrates a provably optimal round-robin schedule. The group abstraction also imposes a residency constraint, ensuring that massive model states remain cached in host memory to enable “warm-start” context switching. We evaluate ROLLMUX on a production-scale testbed with 328 H20 and 328 H800 GPUs. ROLLMUX improves cost efficiency by 1.84× over standard disaggregation and 1.38 × over state-of-the-art colocated baselines, all while achieving 100% SLO attainment.  \n1 Introduction  \nThe focus of Large Language Model (LLM) development has shifted from pre-training to Reinforcement Learning (RL) post-training [1, 3, 6], a critical technique for unlocking reasoning capabilities in complex domains such as mathematics [2], coding [31], and tool use [8, 51] . To achieve optimal performance and model stability, production practices have converged on synchronous, on-policy algorithms [6, 35, 39] . This paradigm mandates a strict, iterative learning process comprising three phases with distinct resource bottlenecks: (1)  \nNaïve Disaggregation  \n Dependency Bubbles  \nRollout Pool Training Pool  \n|  | Train-A ① |  |\n| --- | --- | --- |\n|  |  |  |\n\nSolo iteration time (􀀡!\"\\#\" )  \n\n| | \u003Cbr>Rollout-A ② |\n| --- | --- |\n|  |  |\n\nSync-A  \nDisaggregation & Co-Scheduling  \nA meta-iteration of Co-execution Group {A, B}  \n\n|  |\n| --- |\n| Train-A ② |\n\nRollout Pool  \nTraining Pool  \nFigure 1: Comparison between existing disaggregated RL architecture and ROLLMUX’s co-scheduling paradigm.  \nrollout, a memory-bandwidth-bound inference stage where the model generates tokens as experience trajectories; (2) training, a compute-intensive stage where model parameters are optimized based on the rewards; and (3) synchronization, a network-bound stage where updated model parameters are propagated back to inference workers.  \nTo accommodate the divergent resource demands of these phases, production deployment increasingly employs a disaggregated architecture [10, 44, 49, 58] . Unlike monolithic provisioning, this architecture separates the rollout and training phases onto purpose-built clusters (see Figure 1): a rollout pool consisting of cost-effective, inference-optimized GPUs (e.g., NVIDIA H20) and a training pool of high-performance, compute-optimized GPUs (e.g., NVIDIA H800) . By aligning hardware capabilities with specific phase characteristics, disaggregation resolves the resource mismatches inherent ina monolithic setup. Consequently, it achieves superior cost efficiency and comparable throughput despite the introduction of cross-cluster synchronization overheads [10, 44, 49, 58] .  \nH","cbCaih3G0w2FBkTG","https://ap.wps.com/l/cbCaih3G0w2FBkTG","pdf",1534785,17,"English","# Abstract\n# Introduction\n## Disaggregated RL Post-Training\n## Dependency Bubbles and Limitations of Existing Approaches\n## ROLLMUX Overview and Key Insight\n## Challenges in Production Co-Scheduling","[{\"question\":\"What problem does ROLLMUX address in disaggregated RL post-training?\",\"answer\":\"ROLLMUX targets dependency bubbles caused by strict synchronization in on-policy RL, where one cluster idles while the other executes a dependent phase.\"},{\"question\":\"How does ROLLMUX reduce dependency bubbles across clusters?\",\"answer\":\"It orchestrates multiple RL jobs into a co-execution group, weaving their rollout and training phases across rollout and training pools to enable efficient time-multiplexing.\"},{\"question\":\"What scheduling mechanisms and constraints does ROLLMUX use?\",\"answer\":\"It uses a two-tier scheduling architecture: an inter-group scheduler based on conservative stochastic planning and an intra-group scheduler using a provably optimal round-robin schedule, along with a residency constraint to keep model states cached for warm-start switching.\"}]","ROLLMUX: Phase-Level Multiplexing for Disaggregated RL Post-Training | PDF",1791060243,43]