[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-327638-105":59,"doc-detail-327638-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","can-model-merging-improve-aggregation-in-diloco-research-findings-and-method-comparison","Can Model Merging Improve Aggregation in DiLoCo? - Research findings and method comparison","","Model merging aggregates independently fine-tuned models into one to combine capabilities, while distributed learning methods such as local SGD and DiLoCo reduce communication through periodic aggregation of local models. However, FLOP-matched performance can degrade as the number of local workers and inner training steps grows. The study draws an analogy between pseudo-gradient aggregation and task arithmetic merging, evaluates state-of-the-art merging methods, and identifies Iso-C as promising. It introduces IsoLoCo using Nesterov momentum, showing IsoLoCo outperforms DiLoCo and gap widens with more workers.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/can-model-merging-improve-aggregation-in-diloco-research-findings-and-method-comparison/327638/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/can-model-merging-improve-aggregation-in-diloco-research-findings-and-method-comparison/327638.png","ImageObject",300,407,{"name":92,"@type":93},"Noah","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-05","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":29},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What problem does the work address in DiLoCo training?","Question",{"text":112,"@type":113},"It addresses performance degradation in FLOP-matched settings as the number of workers or local training steps increases, which makes DiLoCo harder to scale without losing efficiency.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does the paper connect local SGD/DiLoCo aggregation with model merging?",{"text":117,"@type":113},"It draws an explicit analogy between the pseudo-gradient aggregation step used in local SGD/DiLoCo and task arithmetic-based model merging, enabling merging methods to be used for distributed optimization.",{"name":119,"@type":110,"acceptedAnswer":120},"What are Iso-C and IsoLoCo, and how do they differ?",{"text":121,"@type":113},"Iso-C is identified as a promising aggregation method for improving DiLoCo, and IsoLoCo adapts Iso-C for distributed training by adding Nesterov momentum, which improves performance over DiLoCo.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},327638,1790418711,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":29,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":36,"preview_page_count":4},8796095462418,"https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780","arXiv :2607 .030 1 1v 1 [ cs .LG] 3 Jul 2026  \nCan Model Merging Improve Aggregation in DiLoCo?  \nStefan Horoi 1 ,2 ∗ Benjamin Thérien 1 ,2 Guy Wolf1 ,2 Eugene Belilovsky2 ,3  \n1Université de Montréal 2Mila – Quebec AI Institute 3 Concordia University  \nAbstract  \nModel merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a broad array of methods having been proposed to tackle this problem. Simultaneously, an emerging trend in distributed learning has been the use of methods such as local SGD and DiLoCo, which greatly reduce communication costs by periodically aggregating the independently trained local models. However, these communication-efficient methods have been shown to degrade in performance relative to the FLOP-matched data-parallel gold standard as the number of independent local models grows and as the number of local training steps before global communication is increased. In this work, we draw an explicit analogy between the pseudo-gradient aggregation step in local SGD/DiLoCo and task arithmetic-based model merging, establishing a straightforward way to utilize merging methods in the context of distributed optimization. We then evaluate multiple state-of-the-art model merging methods in this setting and identify one method in particular, Iso-C, as a promising approach for improving DiLoCo. We find that DiLoCo SGD with Iso-C aggregation outperforms not only simple pseudo-gradient averaging but even the momentum-based DiLoCo, despite lacking a momentum mechanism itself.  \nBuilding on this finding, we propose IsoLoCo, which adapts Iso-C for distributed training by equipping it with Nesterov momentum. Our empirical evaluations on language model pre-training across varying numbers of local workers show that IsoLoCo significantly outperforms DiLoCo, with the gap between them widening as the number of workers increases. This advantage remains present across model sizes and inner step counts, confirming that merging-inspired aggregation is an effective strategy for low-communication distributed training.  \n1 Introduction  \nThe recent success of large language models (LLMs) has relied in large part on the scaling of model parameters, dataset size and amount of compute used for training. However, training models with parameter counts in the billions or trillions on internet-scale datasets comes with significant engineering challenges. The standard practice is to use data-parallel training, where multiple replicas of the model process mini-batches of data in parallel and their gradients are aggregated to update model parameters via gradient-based optimization. This requires large clusters of tightly co-located accelerators connected with high-bandwidth links, along with careful engineering to orchestrate communication and ensure efficient resource utilization. Furthermore, such synchronized training is vulnerable to device failures and makes poor use of heterogeneous hardware with varying processing speeds or network topologies.  \nRecently, [6] introduced Distributed Low-Communication (DiLoCo) training, a variant of FedAvg [23] and FedOpt [28] with AdamW [21] as the inner optimizer and stochastic gradient  \n∗ Correspondence to: [stefan.horoi@umontreal.ca](stefan.horoi@umontreal.ca), [eugene.belilovsky@concordia.ca](eugene.belilovsky@concordia.ca)[ ](eugene.belilovsky@concordia.ca)Code available at: [https://github.com/shoroi/isoloco](https://github.com/shoroi/isoloco)  \nPreprint.  \ndescent (SGD) with Nesterov momentum [24, 30] as the outer optimizer. DiLoCo enables the training of deep learning models on multiple “islands” of poorly connected computational clusters.  \nEach cluster, or worker, trains its own model replica through multiple local optimization steps before sharing the accumulated parameter changes, i.e. pseudogradients, with the remaining clusters for a global optimization step. Performing many","cbCaisK5wstWIgfN","https://ap.wps.com/l/cbCaisK5wstWIgfN","pdf",504971,16,"English","# Abstract\n# 1 Introduction","[{\"question\":\"What problem does the work address in DiLoCo training?\",\"answer\":\"It addresses performance degradation in FLOP-matched settings as the number of workers or local training steps increases, which makes DiLoCo harder to scale without losing efficiency.\"},{\"question\":\"How does the paper connect local SGD/DiLoCo aggregation with model merging?\",\"answer\":\"It draws an explicit analogy between the pseudo-gradient aggregation step used in local SGD/DiLoCo and task arithmetic-based model merging, enabling merging methods to be used for distributed optimization.\"},{\"question\":\"What are Iso-C and IsoLoCo, and how do they differ?\",\"answer\":\"Iso-C is identified as a promising aggregation method for improving DiLoCo, and IsoLoCo adapts Iso-C for distributed training by adding Nesterov momentum, which improves performance over DiLoCo.\"}]","Can Model Merging Improve Aggregation in DiLoCo? - Research findings and method comparison | PDF",1789977696]