[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82724-en":3,"doc-seo-82724-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82724,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","HyperParallel Mpipe A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters","Modern AI expands from text-only interaction to multimodal scenarios, making multimodal large language models (MLLMs) central to research and industry. Large-scale MLLM training often suffers much lower model FLOPs utilization (MFU) than decoder-only LLM training due to modality encoder variability at the pipeline input. Mpipe introduces a schedule algebra that derives concrete runtime behavior from a compact specification, enabling transpose to relocate encoder computation into pipeline warmup bubbles. On Ascend 910C clusters, it delivers 2.70× and 1.21× speedups without changing training loss.","arXiv :2607 .03229v 1 [ cs .DC] 3 Jul 2026  \nHyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters  \nChong Li 1 , Zhengdao Yu 1 , Nelson Lossing 1 , Thibaut Tachon 1 , Pierre Leca 1 , Etienne Filhol 1 , Yujie Yuan 1 , Chong Bao2 , Teng Su2  \n1Huawei Fourier Research Center, Paris, France  \n2Huawei Technologies Co., Ltd, Hangzhou, China  \nAbstract  \nModern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We analyze the key pain points in MLLM training and introduce Mpipe, which uses a schedule algebra to derive concrete runtime behavior from a compact schedule specification. From this algebra, Mpipe derives transpose, a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into otherwise idle pipeline regions. On Ascend 910C NPU clusters, Mpipe achieves 2.70× speedup in a small-scale setting and 1.21 × speedup in a 512-card large-scale setting.  \n1. Introduction  \nModern AI applications are no longer limited to text-only interaction. In many widely used scenarios, such as visual question answering, document understanding, video analysis, embodied agents, and content generation, models are expected to jointly understand language together with images, videos, and other modal inputs. This trend has made multimodal large language models (MLLMs) an increasingly important direction in both research and industry.  \nAs model parameters rapidly scale, training frontier models increasingly depends on large accelerator clusters. For example, Llama 3 reports production training on clusters with up to 16K GPUs (Grattafiori et al., 2024) . MLLMs further increase this demand beyond decoder-only LLMs. In addition to the language backbone, they must process high-resolution images, videos, and other modality inputs through extra model components, which increases computation, memory footprint, and data movement.  \nHowever, large-scale MLLM training often achieves much lower MFU (model FLOPs utilization) than decoder-only LLM training (Zhang et al., 2025) . Figure 1 compares the MFU of representative LLM and MLLM training workloads.  \nThe dominant source of this gap is the modality encoder, which sits at the input side of the pipeline: its highly variable work depends only on data available before the LLM pipeline begins. Such source-side variance need not be  \nchased at runtime, by per-iteration schedule search (Xue et al., 2026b) or data-dependent parallelism (Xue et al., 2026a) . It can instead be relocated off the critical path into the pipeline’s intrinsic warmup bubbles by a single static placement. We propose Mpipe, which formalizes parallel schedules as a small composable algebra and derives transpose, a schedule that places encoder computation in those bubbles, as one point in that algebra. The resulting schedule stays static and invariant to the modality mix, with no runtime scheduling overhead. Our contributions are:  \n• A schedule algebra for parallel training. We formalize parallel pipeline schedules as a small composable algebra in which a schedule is a list of perregion skeletons, and a single derivation maps a cut, schedule, and model to concrete runtime behavior: device placement, collective communication, and execution order, from which we read a cost model that predicts the step makespan. We prove a backwardfootprint lemma and a schedule-invariance corollary: one schedule is correct whether the modality encoder is frozen or trained.  \n• Transpose. From this algebra Mpipe derives transpose, a static schedule that replicates the encoder across pipeline ranks, runs it inside the warmup bubbles, and gathers its output into the first LLM stage. The placement is decided once, adds no runtime scheduli","cbCaifIQcfHqyY5U","https://ap.wps.com/l/cbCaifIQcfHqyY5U","pdf",329255,2,1,10,"English","en",105,"# Introduction\n# From LLM to MLLM Training\n# HyperParallel-Mpipe Overview\n# Transpose Schedule and Runtime Behavior\n# Experimental Results on Ascend 910C","[{\"question\":\"Why does large-scale MLLM training often achieve lower MFU than decoder-only LLM training?\",\"answer\":\"The modality encoder at the pipeline input causes highly variable work that depends on data available before the LLM pipeline begins, creating source-side variance that reduces MFU compared with decoder-only workloads.\"},{\"question\":\"What is Mpipe and how does it relate a schedule specification to runtime behavior?\",\"answer\":\"Mpipe formalizes parallel pipeline schedules as a small composable algebra and uses a derivation to map a schedule and model to concrete runtime actions, including device placement, collective communication, and execution order, from which a cost model predicts step makespan.\"},{\"question\":\"How does transpose improve performance on NPU clusters?\",\"answer\":\"Transpose replicates the encoder across pipeline ranks and executes it inside the pipeline warmup bubbles, then gathers outputs into the first LLM stage. It is decided once, adds no runtime scheduling overhead, and remains invariant to the modality mix.\"}]",1784182501,25,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"hyperparallel-mpipe-a-composable-algebra-system-for-optimizing-mllm-training-over-supernode-clusters","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/hyperparallel-mpipe-a-composable-algebra-system-for-optimizing-mllm-training-over-supernode-clusters/82724/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does large-scale MLLM training often achieve lower MFU than decoder-only LLM training?","Question",{"text":75,"@type":76},"The modality encoder at the pipeline input causes highly variable work that depends on data available before the LLM pipeline begins, creating source-side variance that reduces MFU compared with decoder-only workloads.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is Mpipe and how does it relate a schedule specification to runtime behavior?",{"text":80,"@type":76},"Mpipe formalizes parallel pipeline schedules as a small composable algebra and uses a derivation to map a schedule and model to concrete runtime actions, including device placement, collective communication, and execution order, from which a cost model predicts step makespan.",{"name":82,"@type":73,"acceptedAnswer":83},"How does transpose improve performance on NPU clusters?",{"text":84,"@type":76},"Transpose replicates the encoder across pipeline ranks and executes it inside the pipeline warmup bubbles, then gathers outputs into the first LLM stage. It is decided once, adds no runtime scheduling overhead, and remains invariant to the modality mix.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":22,"slug":133},"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]