[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-126717-en":3,"doc-seo-126717-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},126717,962084925782,"Ava Thompson","https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc",8,"Research & Report","MAD-Max Beyond Single-Node - Enabling Large Machine Learning Model Acceleration on Distributed Systems","Training and deploying large-scale machine learning models is time-consuming, demands substantial distributed computing infrastructure, and drives high operational costs. Real-world analysis of datacenter-scale large-model training shows that 14–32% of total GPU hours are consumed by communication with no overlapping computation. To reduce this communication latency and other at-scale inefficiencies, the paper presents MAD-Max, an agile performance modeling framework for optimizing parallelization and enabling hardware-software co-design. Evaluations on modern GPU clusters report up to 2.24× throughput gains for pretraining and up to 5.27× for inference.","arXiv :2310 .02784v3 [ cs .DC] 10 Jun 2024  \nMAD-Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on  \nDistributed Systems  \nSamuel Hsia 1 ,2 , Alicia Golden 1 ,2 , Bilge Acun 1 , Newsha Ardalani 1 , Zachary DeVito 1 ,  \nGu-Yeon Wei2 , David Brooks2 , Carole-Jean Wu 1  \n1FAIR at Meta, 2Harvard University  \n[shsia@g.harvard.edu](shsia@g.harvard.edu), [carolejeanwu@meta.com](carolejeanwu@meta.com)  \nAbstract—Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs. Our analysis, grounded in real-world large model training on datacenter-scale infrastructures, reveals that 14∼32% of all GPU hours are spent on communication with no overlapping computation. To minimize this outstanding communication latency and other inherent at-scale inefficiencies, we introduce an agile performance modeling framework, MAD-Max. This framework is designed to optimize parallelization strategies and facilitate hardware-software co-design opportunities. Through the application of MAD-Max to a suite of real-world largescale ML models on state-of-the-art GPU clusters, we showcase potential throughput enhancements of up to 2.24× for pretraining and up to 5.27× for inference scenarios, respectively.  \nI. INTRODUCTION  \nBillion-parameter large language models (LLMs) [9], [49],[61], [62] power applications that have shown far-reaching impact across different domains [15], [16], [38], [48] . Similarly, trillion-parameter recommendation models [40], [72] have demonstrated state-of-the-art user modeling and content understanding across search [6], [11], [31], [76], social media [1], [18], [19], [71], e-commerce [78], [79], and entertainment [20] . As these large-scale ML models increase in size and complexity [18], [19], the corresponding training and inference workloads become ever more resource-intensive. Without efficient mappings between these large-scale ML workloadsand their underlying distributed systems, model training and exploration can easily require millions of GPU hours, levying high operational costs, compute resource requirements, and energy consumption [9], [61], [62] .  \nFigure 1 shows the projected resource-performance pareto frontier of training a state-of-the-art deep learning recommendation model (DLRM) using default workload-system mapping strategy on public cloud instances. In this case, we quantify compute resource requirements with aggregate GPU hours per 1 billion samples, where aggregate GPU hours of different generations of GPUs are normalized based on the A100’s peak FLOPS. Further improving upon this  \nFig. 1. Our performance model – MAD-Max – improves upon the resourceperformance pareto frontier of large-scale ML workloads by identifying new hardware-software mappings and solutions.  \nresource-performance pareto frontier requires researchers to take into account underlying distributed systems [14], [28]–[30], [42], [43], [45], [46] and how we map models and tasks onto underlying distributed systems – parallelization strategy. In this paper, we propose a distributed ML performance model – MAD-Max – for identifying potential avenues for improvement (green, dotted line) . Nonetheless, pinpointing the specific distributed systems and parallelization strategies needed for realizing these improvements in performance and operational compute resource requirements is challenging, as evidenced by the three general approaches for optimizing runtime performance of large ML models.  \nThe first option involves applying industry-standard parallelization strategies (Figure 1: blue, dotted line) that target feasibility without fully optimizing hardware usage (e.g., FSDP [54], [75]) . The second option is to custom-design custom hierarchical parallelization strategies specific to the model, task, and system [59] . This maximizes hardware efficiency but is complex from an engineering standpoint and not easily a","cbCaihKecNA2PN7o","https://ap.wps.com/l/cbCaihKecNA2PN7o","pdf",1740073,1,16,"English","en",105,"# Introduction\n## Problem of communication overhead in large-model training\n## Three approaches for improving runtime performance\n# MAD-Max performance modeling framework\n## Inputs, per-device traces, and system-level estimation\n## Communication/computation overlap and breakdowns\n# Evaluation results","[{\"question\":\"What key bottleneck do the authors identify in large-model training?\",\"answer\":\"They find that 14–32% of all GPU hours are spent on communication without overlapping computation, creating substantial latency and inefficiency at scale.\"},{\"question\":\"What is MAD-Max and what problem does it target?\",\"answer\":\"MAD-Max is an agile performance modeling framework designed to optimize parallelization strategies and support hardware-software co-design to minimize communication and other at-scale inefficiencies.\"},{\"question\":\"What performance improvements does MAD-Max demonstrate on real-world models?\",\"answer\":\"Across experiments on state-of-the-art GPU clusters, MAD-Max shows potential throughput enhancements up to 2.24× for pretraining and up to 5.27× for inference scenarios.\"}]","MAD-Max Beyond Single-Node - Enabling Large Machine Learning Model Acceleration on Distributed Systems | PDF",1785934381,40,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"mad-max-beyond-single-node-enabling-large-machine-learning-model-acceleration-on-distributed-systems","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/mad-max-beyond-single-node-enabling-large-machine-learning-model-acceleration-on-distributed-systems/126717/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What key bottleneck do the authors identify in large-model training?","Question",{"text":75,"@type":76},"They find that 14–32% of all GPU hours are spent on communication without overlapping computation, creating substantial latency and inefficiency at scale.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is MAD-Max and what problem does it target?",{"text":80,"@type":76},"MAD-Max is an agile performance modeling framework designed to optimize parallelization strategies and support hardware-software co-design to minimize communication and other at-scale inefficiencies.",{"name":82,"@type":73,"acceptedAnswer":83},"What performance improvements does MAD-Max demonstrate on real-world models?",{"text":84,"@type":76},"Across experiments on state-of-the-art GPU clusters, MAD-Max shows potential throughput enhancements up to 2.24× for pretraining and up to 5.27× for inference scenarios.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":29,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]