[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82853-en":3,"doc-seo-82853-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82853,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","Direct Model State Migration for Elastic Training of Large Language Models","Large language model training in shared clusters must handle elastic resource changes such as preemption and scaling, which require migrating model state when hybrid-parallel configurations change. Existing checkpoint-based approaches serialize complete states to persistent storage and then reshard, forcing GPU-wide stalls and incurring tens to hundreds of seconds of migration latency due to cross-hierarchy tensor transfers and ignoring GPU-resident locality. ETC is a checkpoint-free state migration framework that exploits state locality and uses a cost-matrix abstraction plus communication coalescing to reduce inter-GPU movement. Integrated with Megatron-LM, ETC cuts migration overhead by 2.33×–6.37× and enables practical elastic training in production.","Direct Model State Migration for Elastic Training  \nof Large Language Models  \nWeijian Liu 1 ,2 , Mingzhen Li 1 ,2∗, Rui Kang3 , Chen Sun3 , Guangming Tan 1 ,2 , Weile Jia 1 ,2∗  \nSKLP, Institute of Computing Technology, CAS 1  \nUniversity of Chinese Academy of Sciences2  \nHuawei Technologies Co., Ltd.3  \narXiv :2607 .04749v 1 [ cs .DC] 6 Jul 2026  \nAbstract—Large language model (LLM) training shall adapt to dynamic resources in shared clusters to tackle the elasticity, including passive preemption and optimistic scaling. State migration across device sets is required when altering the hybrid-parallel configuration due to dynamic resources. Existing solutions rely on checkpoint-based mechanisms, which persist complete states to storage for resuming with re-assigned resources, forcing all GPUs to stall when transferring model states. Despite optimization efforts, checkpoint-based solutions incur prohibitive latency (tens to hundreds of seconds) due to data movement across memory hierarchies and disregard for GPU-resident state locality. We propose ETC, a checkpointfree state migration framework for elastic hybrid-parallel LLM training. We exploit the state locality to minimize inter-GPU data movement, replacing persistence with direct device-to-device communication. We leverage the cost matrix abstraction to quantify data movement between parallel configurations and find the optimal migration sketch through a cost-matrix-driven approach. Besides, we eliminate node fragmentation through communication coalescing. Integrated with Megatron-LM, ETC reduces migration overhead by 2.33× to 6.37× compared to checkpoint-based solutions across diverse parallel configurations. By enabling efficient migration, ETC unlocks practical elastic training in production environments.  \nI. INTRODUCTION  \nLarge language models (LLM), based on the transformer architecture, have emerged as foundational models driving diverse applications. However, training these models incurs substantial computational and memory demands. To achieve stateof-the-art performance in quality and sample efficiency [1], contemporary LLMs are progressively scaled up in model size. The immense computational demands and prolonged execution times of LLM training necessitate deployment on shared large-scale clusters, where hybrid-parallel training jobs are submitted to the cluster job scheduler.  \nIn shared clusters, the GPU resources visible to a longrunning training job do not remain static. Job arrivals, job completions, quota reclaiming, and scheduler-driven reallocation can continuously reshape the device set assigned to a running job. Therefore, elastic hybrid-parallel training should be prepared to migrate model state whenever the available device set changes, rather than assuming a fixed cluster partition throughout the entire training process. To achieve better resource utilization, a training job should scale in/out when fluctuating GPU resources become available due to cluster load change, and the scheduler should frequently reconcile the GPU allocation among jobs in a proactive manner (i.e.,  \npreemption and scaling), especially on shared clusters [2]–[4] that co-locates inference jobs and training jobs. Therefore, state migration—the process of transferring model states when moving a training job from device set A to device set B—is essential for hybrid-parallel training jobs, especially those deployed in shared clusters.  \nStandard solutions of state migration for a hybrid-parallel training job predominantly rely on the checkpointing mechanism, which checkpoints complete states (parameters, optimizer states, etc.) to persistent storage before migration and then resumes from the states after the restart. If the parallel configuration (i.e., pipeline/tensor-parallel dimension) should be altered according to dynamic resources, the state should be resharded to match the new parallel configuration before resuming. Unfortunately, checkpointing and resharding reside on the ","cbCaimuMhEg1neIe","https://ap.wps.com/l/cbCaimuMhEg1neIe","pdf",1569752,1,12,"English","en",105,"# Introduction\n## Problem: Elastic training and state migration\n## Limitations of checkpoint-based migration\n## Proposed solution: ETC (checkpoint-free migration)","[{\"question\":\"Why is state migration necessary in elastic hybrid-parallel LLM training?\",\"answer\":\"In shared clusters, the GPU device set assigned to a running job can change due to arrivals, completions, quota reclaiming, or scheduler reallocation. When the device set changes, the model state must be transferred from device set A to device set B to continue training under the updated hybrid-parallel configuration.\"},{\"question\":\"What makes checkpoint-based state migration costly?\",\"answer\":\"Checkpointing persists full model states (parameters and optimizer states) to storage and then resharding runs on the critical training path. Frequent migrations therefore stall all GPUs, and latency becomes tens to hundreds of seconds because massive state tensors must move across memory hierarchy (GPU→host DRAM→disk) while disregarding GPU-resident state locality.\"},{\"question\":\"How does ETC reduce migration overhead compared with checkpointing?\",\"answer\":\"ETC migrates states without checkpoints by exploiting state locality to minimize inter-GPU data movement and replacing persistence with direct device-to-device communication. It also uses a cost-matrix abstraction to choose an optimal migration sketch and applies communication coalescing to avoid node fragmentation, reducing overhead by 2.33×–6.37× with Megatron-LM.\"}]",1784183456,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"direct-model-state-migration-for-elastic-training-of-large-language-models","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/direct-model-state-migration-for-elastic-training-of-large-language-models/82853/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why is state migration necessary in elastic hybrid-parallel LLM training?","Question",{"text":74,"@type":75},"In shared clusters, the GPU device set assigned to a running job can change due to arrivals, completions, quota reclaiming, or scheduler reallocation. When the device set changes, the model state must be transferred from device set A to device set B to continue training under the updated hybrid-parallel configuration.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What makes checkpoint-based state migration costly?",{"text":79,"@type":75},"Checkpointing persists full model states (parameters and optimizer states) to storage and then resharding runs on the critical training path. Frequent migrations therefore stall all GPUs, and latency becomes tens to hundreds of seconds because massive state tensors must move across memory hierarchy (GPU→host DRAM→disk) while disregarding GPU-resident state locality.",{"name":81,"@type":72,"acceptedAnswer":82},"How does ETC reduce migration overhead compared with checkpointing?",{"text":83,"@type":75},"ETC migrates states without checkpoints by exploiting state locality to minimize inter-GPU data movement and replacing persistence with direct device-to-device communication. It also uses a cost-matrix abstraction to choose an optimal migration sketch and applies communication coalescing to avoid node fragmentation, reducing overhead by 2.33×–6.37× with Megatron-LM.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":120},"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]