[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-124245-en":3,"doc-seo-124245-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},124245,687197207057,"Sage","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources - Technical Disclosure Commons Defensive Publications Series","As machine learning models scale in size and complexity, conventional serving approaches struggle with memory saturation, limited throughput, and poor accelerator utilization—especially for embedding-heavy and transformer-style architectures. Fixed-resource accelerators and mismatched elasticity between CPU and TPU further hinder efficient vertical or uniform scaling. The disclosure introduces a distributed serving system and method that partitions ML models into smaller shards and intelligently deploys them across diverse hardware resources, leveraging each platform’s strengths while addressing limitations of homogeneous distributed serving, disaggregated serving, and embedding server designs.","Technical Disclosure Commons  \nDefensive Publications Series  \n14 Apr 2025  \nDistributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources  \nVince Gatto Junyan Li Chi Chen Hang Fu Brad Froehle  \nSee next page for additional authors  \nFollow this and additional works at: [https://www.tdcommons.org/dpubs_series](https://www.tdcommons.org/dpubs_series)  \nRecommended Citation  \nGatto, Vince; Li, Junyan; Chen, Chi; Fu, Hang; Froehle, Brad; Hinsu, Smit; Jain, Devanshu; Nath, Aniruddh; Jia, Ruolin; Chang, Shuo; Ma, He; and Jiang, Daokun, \"Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources\", Technical Disclosure Commons,(April 14, 2025)  \n[https://www.tdcommons.org/dpubs_series/8008](https://www.tdcommons.org/dpubs_series/8008)  \nThis work is licensed under a Creative Commons Attribution 4.0 License.  \nThis Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons.  \nInventor(s)  \nVince Gatto, Junyan Li, Chi Chen, Hang Fu, Brad Froehle, Smit Hinsu, Devanshu Jain, Aniruddh Nath, Ruolin Jia, Shuo Chang, He Ma, and Daokun Jiang  \nThis article is available at Technical Disclosure Commons: [https://www.tdcommons.org/dpubs_series/8008](https://www.tdcommons.org/dpubs_series/8008)  \nDistributed Serving of Partitioned Machine Learning Models Across Heterogeneous  \nHardware Resources  \nAs machine learning (ML) models grow in size and complexity, traditional model serving approaches face significant challenges. This is particularly true for ML models with large embedding tables and transformer-style architectures. Single-node deployments or homogeneous hardware configurations (e.g., graphics processing unit (GPU)-only or tensor processing unit  \n(TPU)-only systems) often suffer from memory saturation, suboptimal throughput, and/or  \nunderutilization of computational resources. Embedding-heavy models exacerbate these issues  \ndue to memory and bandwidth demands. For example, massive embedding tables can cause  \nmemory saturation on the host machine, vocabulary lookup or feature pre-processing can create  \npreprocessing computational bottlenecks, and data exchanges between the host memory and TPU can cause memory bandwidth saturation. As a result, powerful accelerators like TPUs tend to be  \nunderutilized, limiting the overall serving throughput and efficiency.  \nEfforts such as model compression and vertical scaling on individual machines fall short  \nin addressing the fundamental problem of mismatched compute and memory needs across  \ndifferent stages of inference. For example, TPU hardware has fixed resources (e.g., fixed Google  \nCompute Unit (GCU), fixed Random Access Memory (RAM), and fixed High Bandwidth  \nMemory (HBM)), and thus a TPU job cannot easily scale vertically. Moreover, hardware  \nresources such as Central Processing Units (CPUs) and TPUs offer different elasticities in cloud  \nenvironments, making it inefficient to scale them uniformly. For example, CPU resources are  \nmore elastic than TPU resources, and thus using TPU-only serving results in the serving system  \nthat is less elastic and less cost-efficient to scale. Thus, there exists a need for a distributed  \nserving system capable of partitioning model components and intelligently deploys them across  \nmultiple and diverse hardware resources, leveraging the respective strengths of each serving  \nhardware.  \n1  \nPublished by Technical Disclosure Commons, 2025 2  \nExisting technologies, such as homogenous distributed serving, disaggregated serving,  \nand embedding server fail to adequately address the above-noted deficiencies. Homogenous  \ndistributed serving is a classical method for serving a ML model to a large number of users, in  \nwhich all machines in the cluster utilize the same type of hardware (e.g., CPU, GPU, or TPU  \nonly). Ev","cbCaijIXD4Zmz0Ku","https://ap.wps.com/l/cbCaijIXD4Zmz0Ku","pdf",269267,1,12,"English","en",105,"# Overview and problem statement\n## Challenges of traditional serving for large embedding and transformer models\n# Limitations of existing approaches\n## Homogeneous distributed serving\n## Disaggregated serving\n## Embedding server\n# Proposed system and method\n## Partitioning models into shards for heterogeneous deployment","[{\"question\":\"Why do traditional ML model serving methods underperform for large embedding and transformer models?\",\"answer\":\"They often trigger memory saturation and bandwidth bottlenecks, and they can leave accelerators like TPUs underutilized, limiting overall throughput and efficiency.\"},{\"question\":\"What key issue does fixed-resource accelerator hardware introduce for scaling?\",\"answer\":\"Accelerators such as TPUs have fixed resources (e.g., fixed compute, RAM, and HBM), so a TPU job cannot easily scale vertically, and differing elasticity across hardware makes uniform scaling inefficient.\"},{\"question\":\"How does the proposed approach improve upon homogeneous, disaggregated, and embedding-server serving?\",\"answer\":\"It partitions the ML model into smaller shards and deploys them across multiple heterogeneous hardware resources, enabling each stage to use the most suitable hardware capabilities rather than relying on single-type clusters or rigid dedicated embedding infrastructure.\"}]","Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources - Technical Disclosure Commons Defensive Publications Series | PDF",1785821206,30,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"distributed-serving-of-partitioned-machine-learning-models-across-heterogeneous-hardware-resources-technical-disclosure-commons-defensive-publications-series","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/distributed-serving-of-partitioned-machine-learning-models-across-heterogeneous-hardware-resources-technical-disclosure-commons-defensive-publications-series/124245/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do traditional ML model serving methods underperform for large embedding and transformer models?","Question",{"text":75,"@type":76},"They often trigger memory saturation and bandwidth bottlenecks, and they can leave accelerators like TPUs underutilized, limiting overall throughput and efficiency.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What key issue does fixed-resource accelerator hardware introduce for scaling?",{"text":80,"@type":76},"Accelerators such as TPUs have fixed resources (e.g., fixed compute, RAM, and HBM), so a TPU job cannot easily scale vertically, and differing elasticity across hardware makes uniform scaling inefficient.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the proposed approach improve upon homogeneous, disaggregated, and embedding-server serving?",{"text":84,"@type":76},"It partitions the ML model into smaller shards and deploys them across multiple heterogeneous hardware resources, enabling each stage to use the most suitable hardware capabilities rather than relying on single-type clusters or rigid dedicated embedding infrastructure.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":29,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]