[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-125628-en":3,"doc-seo-125628-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},125628,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",6,"Technology","Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol - 7 May 2023","Distributed Machine Learning (DML) systems accelerate model training across data centers and edge nodes, commonly using the Parameter Server (PS) communication architecture. However, many-to-one incast traffic creates severe long-tail latency that throttles training throughput. The Loss-tolerant Transmission Protocol (LTP) enables partial gradient loss during synchronization to avoid unnecessary retransmissions. LTP realizes loss tolerance via out-of-order transmission and ACKs, guided by Early Close and bubble-filling for accuracy, and is integrated into PyTorch. Experiments with 8 workers and 1 PS show up to 30x throughput improvement versus traditional TCP congestion control without final accuracy loss.","Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol  \nZixuan Cheny , Lei Shiy , Xuandong Liuy , Xin Aiy , Sen Liuyz , and Yang Xuy}􀀃  \ny School of Computer Science, Fudan University, Shanghai, China zInstitute of Fintech, Fudan University, Shanghai, China  \n}Peng Cheng Laboratory, Shenzhen, China  \narXiv :2305 .04279v1 [ cs .DC] 7 May 2023  \nAbstract—Distributed Machine Learning (DML) systems are utilized to enhance the speed of model training in data centers (DCs) and edge nodes. The Parameter Server (PS) communication architecture is commonly employed, but it faces severe long-tail latency caused by many-to-one ”incast” traffic patterns, negatively impacting training throughput. To address this challenge, we design the Loss-tolerant Transmission Protocol (LTP), which permits partial loss of gradients during synchronization to avoid unneeded retransmission and contributes to faster synchronization per iteration. LTP implements loss-tolerant transmission through out-of-order transmission and out-of-order Acknowledges (ACKs). LTP employs Early Close to adjust the loss-tolerant threshold based on network conditions and bubble-filling for data correction to maintain training accuracy. LTP is implemented by C++ and integrated into PyTorch. Evaluations on a testbed of 8 worker nodes and one PS node demonstrate that LTP can significantly improve DML training task throughput by up to 30x compared to traditional TCP congestion controls, with no sacrifice to final accuracy.  \nIndex Terms—Distributed Machine Learning, Transmission Protocol, Parameter Server.  \nI. INTRODUCTION  \nWith the explosion of dataset and model size in Machine Learning (ML) applications, Distributed Machine Learning (DML) has been widely adopted to leverage the power of multiple worker nodes during large-scale training. To achieve synchronization among the distributed worker nodes, several DML communication architectures, such as the Parameter Server architecture (PS) [1] and Ring-AllReduce [2], are proposed, improving the efficiency of the distributed training systems.  \nThe PS architecture has become a prevalent communication architecture in DML due to its simplicity and efficiency. In the PS architecture, all computing nodes, also known as worker nodes, are managed by one or multiple PS(es) . The worker  \nnodes learn a portion of the training dataset and communicate * Corresponding author: Yang Xu ([xuy@fudan.edu.cn](xuy@fudan.edu.cn))  \nFirst author: Zixuan Chen ([zxchen20@fudan.edu.cn](zxchen20@fudan.edu.cn))  \nThis work is sponsored by the Key-Area Research and Development Program of Guangdong Province (2021B0101400001), National Natural Science Foundation of China (62150610497, 62172108, 62002066), Natural Science Foundation of Shanghai (23ZR1404900), Shanghai Pujiang Program (2020PJD005), the Major Key Project of PCL (PCL2021A15), and Open Research Projects of Zhejiang Lab (2022QA0AB07) .  \nThis paper will be published on IWQoS 2023 . Preview version only. 979-8-3503-9973-8/23/$31.00 ©2023 IEEE  \nFig. 1. PS architectures in DML training.  \nwith the associated PS to synchronize training results with other worker nodes (as shown in Figure 1) . During each iteration, each worker node trains the model using a portion of the dataset and sends the computed gradients to the PS, which aggregates the gradients from all worker nodes and updates the global model. Finally, the PS sends the latest model to each worker node, preparing it for the next iteration. The parallel computation of multiple worker nodes greatly improves the efficiency of the model training process.  \nAs the scale of the DML system increases, the network connecting the worker nodes and the PS is becoming a major limitation for the system. This is due to two main reasons. Firstly, most of the existing DML training tasks use the Bulk Synchronous Parallel (BSP) synchronization model [3], in which all nodes must fully synchronize their gradients with the PS at the end of","cbCailC02GUrCi49","https://ap.wps.com/l/cbCailC02GUrCi49","pdf",977000,1,11,"English","en",105,"# Introduction\n## Background: DML and PS architecture\n## Problem: incast long-tail latency\n## New scenarios and non-congestion packet loss\n## Prior solutions and limitations\n## Proposed approach: LTP","[{\"question\":\"What problem does LTP address in distributed machine learning training?\",\"answer\":\"LTP targets long-tail latency caused by many-to-one incast traffic in Parameter Server (PS) systems, which reduces synchronization efficiency and training throughput.\"},{\"question\":\"How does LTP improve synchronization efficiency?\",\"answer\":\"LTP allows partial loss of gradients during synchronization to avoid unneeded retransmissions, implementing loss-tolerant behavior through out-of-order transmission and out-of-order ACKs.\"},{\"question\":\"Does LTP affect final model accuracy?\",\"answer\":\"Experiments integrating LTP into PyTorch report up to 30x throughput improvement compared with traditional TCP congestion control without sacrificing final accuracy.\"}]","Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol - 7 May 2023 | PDF",1785900302,28,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"boosting-distributed-machine-learning-training-through-loss-tolerant-transmission-protocol-7-may-2023","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/technology/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/boosting-distributed-machine-learning-training-through-loss-tolerant-transmission-protocol-7-may-2023/125628/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does LTP address in distributed machine learning training?","Question",{"text":75,"@type":76},"LTP targets long-tail latency caused by many-to-one incast traffic in Parameter Server (PS) systems, which reduces synchronization efficiency and training throughput.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does LTP improve synchronization efficiency?",{"text":80,"@type":76},"LTP allows partial loss of gradients during synchronization to avoid unneeded retransmissions, implementing loss-tolerant behavior through out-of-order transmission and out-of-order ACKs.",{"name":82,"@type":73,"acceptedAnswer":83},"Does LTP affect final model accuracy?",{"text":84,"@type":76},"Experiments integrating LTP into PyTorch report up to 30x throughput improvement compared with traditional TCP congestion control without sacrificing final accuracy.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":46,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":46,"category_name":120,"show_sort_weight":121,"slug":122},8,"Research & Report",30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]