Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources - Technical Disclosure Commons Defensive Publications Series

Aug 4, 2026

Upload by Sage

As machine learning models scale in size and complexity, conventional serving approaches struggle with memory saturation, limited throughput, and poor accelerator utilization—especially for embedding-heavy and transformer-style architectures. Fixed-resource accelerators and mismatched elasticity between CPU and TPU further hinder efficient vertical or uniform scaling. The disclosure introduces a distributed serving system and method that partitions ML models into smaller shards and intelligently deploys them across diverse hardware resources, leveraging each platform’s strengths while addressing limitations of homogeneous distributed serving, disaggregated serving, and embedding server designs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Overview and problem statement

Challenges of traditional serving for large embedding and transformer models

Limitations of existing approaches

Homogeneous distributed serving

Disaggregated serving

Embedding server

Proposed system and method

Partitioning models into shards for heterogeneous deployment

Technical Disclosure Commons Defensive Publications Series 14 Apr 2025 Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources Vince Gatto Junyan Li Chi Chen Hang Fu Brad Froehle See next page for additional authors Follow this and additional works at: [https://www.tdcommons.org/dpubs_series](https://www.tdcommons.org/dpubs_series) Recommended Citation Gatto, Vince; Li, Junyan; Chen, Chi; Fu, Hang; Froehle, Brad; Hinsu, Smit; Jain, Devanshu; Nath, Aniruddh; Jia, Ruolin; Chang, Shuo; Ma, He; and Jiang, Daokun, "Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources", Technical Disclosure Commons,(April 14, 2025) [https://www.tdcommons.org/dpubs_series/8008](https://www.tdcommons.org/dpubs_series/8008) This work is licensed under a Creative Commons Attribution 4.0 License. This Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons. Inventor(s) Vince Gatto, Junyan Li, Chi Chen, Hang Fu, Brad Froehle, Smit Hinsu, Devanshu Jain, Aniruddh Nath, Ruolin Jia, Shuo Chang, He Ma, and Daokun Jiang This article is available at Technical Disclosure Commons: [https://www.tdcommons.org/dpubs_series/8008](https://www.tdcommons.org/dpubs_series/8008) Distributed Serving of Partitioned Machine Learning Models Across Heterogeneous Hardware Resources As machine learning (ML) models grow in size and complexity, traditional model serving approaches face significant challenges. This is particularly true for ML models with large embedding tables and transformer-style architectures. Single-node deployments or homogeneous hardware configurations (e.g., graphics processing unit (GPU)-only or tensor processing unit (TPU)-only systems) often suffer from memory saturation, suboptimal throughput, and/or underutilization of computational resources. Embedding-heavy models exacerbate these issues due to memory and bandwidth demands. For example, massive embedding tables can cause memory saturation on the host machine, vocabulary lookup or feature pre-processing can create preprocessing computational bottlenecks, and data exchanges between the host memory and TPU can cause memory bandwidth saturation. As a result, powerful accelerators like TPUs tend to be underutilized, limiting the overall serving throughput and efficiency. Efforts such as model compression and vertical scaling on individual machines fall short in addressing the fundamental problem of mismatched compute and memory needs across different stages of inference. For example, TPU hardware has fixed resources (e.g., fixed Google Compute Unit (GCU), fixed Random Access Memory (RAM), and fixed High Bandwidth Memory (HBM)), and thus a TPU job cannot easily scale vertically. Moreover, hardware resources such as Central Processing Units (CPUs) and TPUs offer different elasticities in cloud environments, making it inefficient to scale them uniformly. For example, CPU resources are more elastic than TPU resources, and thus using TPU-only serving results in the serving system that is less elastic and less cost-efficient to scale. Thus, there exists a need for a distributed serving system capable of partitioning model components and intelligently deploys them across multiple and diverse hardware resources, leveraging the respective strengths of each serving hardware. 1 Published by Technical Disclosure Commons, 2025 2 Existing technologies, such as homogenous distributed serving, disaggregated serving, and embedding server fail to adequately address the above-noted deficiencies. Homogenous distributed serving is a classical method for serving a ML model to a large number of users, in which all machines in the cluster utilize the same type of hardware (e.g., CPU, GPU, or TPU only). Ev

FAQs

Why do traditional ML model serving methods underperform for large embedding and transformer models?

They often trigger memory saturation and bandwidth bottlenecks, and they can leave accelerators like TPUs underutilized, limiting overall throughput and efficiency.

What key issue does fixed-resource accelerator hardware introduce for scaling?

Accelerators such as TPUs have fixed resources (e.g., fixed compute, RAM, and HBM), so a TPU job cannot easily scale vertically, and differing elasticity across hardware makes uniform scaling inefficient.

How does the proposed approach improve upon homogeneous, disaggregated, and embedding-server serving?

It partitions the ML model into smaller shards and deploys them across multiple heterogeneous hardware resources, enabling each stage to use the most suitable hardware capabilities rather than relying on single-type clusters or rigid dedicated embedding infrastructure.

Aug 4, 2026

Upload by Sage