CrossPool - Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

Aug 21, 2026

Emerging LLM services increasingly host sparse Mixture-of-Experts (MoE) models, yet most remain cold, creating GPU memory imbalance: model weights are stable while KV-cache is transient and demand-driven. Worst-case per-model KV reservation wastes memory, but naive KV sharing fails when weights and KV-cache share a monolithic pool. CrossPool separates FFN weights and KV-cache into distinct GPU memory pools, planning and virtualizing shared KV-cache while keeping attention local. It reduces CPU-GPU control overhead and improves long-context support, achieving up to 10.4× lower P99 TBT versus state of the art.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Introduction

Cold-model serving and memory imbalance

Opportunity: different lifetimes of weights vs KV-cache

Limits of unified memory sharing in multi-LLM systems

CrossPool framework

Disaggregated FFN weights pool and dynamic KV-cache pool

Pool boundary and hidden-state exchange

Design challenges and contributions

KV-cache planner and virtualizer

Evaluation context (figures and metrics)

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang† School of Software, Beihang University {yezr, woty, dinghaoxue, mingmingzhang,yuchenteng,hucm, [renyuyang}@buaa.edu.cn](renyuyang}@buaa.edu.cn) arXiv :2606 .24506v2 [ cs .DC] 26 Jun 2026 Abstract Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and demand-determined. Because cold models rarely reach peak KV-cache demand at the same time, reserving worst-case KV capacity per model wastes memory; a shared KV-cache pool can instead provision aggregate active demand. However, KV-cache sharing is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. Static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support. We present CrossPool, a serving engine for cold MoE models that separates FFN weights and KV-cache into two GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local to KV-cache. CrossPool combines a KV-cache planner and virtualizer, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU control overhead. With efficient GPU memory pooling, CrossPool underpins bursty long-context requests and outperforms the state-of-the-art kvcached-based multi-LLM serving system, reducing P99 TBT by up to 10.4× . 1 Introduction Emerging large language models (LLMs) now power applications ranging from chatbots [14, 20, 29] to agentic assistants [4, 30, 31]. To serve this demand, model providers [2, 6, 11] deploy increasingly capable, fast-evolving LLMs in their datacenters. Many recent frontier and open-weight models support long context for complex tasks and use Mixture-ofExperts (MoE) architectures [9, 13, 15, 18, 22, 25, 35] to scale total parameters while keeping computation moderate. This trend makes cold-model serving increasingly important. Recent studies [17, 19, 36] show that a few hot models serve most requests, while many cold models are underutilized. A serving cluster that keeps many such models online therefore pays the memory cost of large MoE weights even †Corresponding Author. when most models have few active requests. Long-context serving adds a second pressure point: KV-cache can grow to a large fraction of GPU memory, but unlike weights, it is allocated only while requests are active and is reclaimed after decode finishes. The key opportunity is that weights and KV-cache have different lifetimes. Model weights are model-determined and stable, whereas KV-cache is demand-determined and transient: its footprint varies with request rate, context length, and concurrency. Since cold models rarely peak simultaneously, reserving worst-case KV-cache for each model wastes GPU memory. Hence, a shared KV-cache pool can instead provision for aggregate active demand. Simple memory sharing within a unified GPU memory pool is insufficient. Existing multi-LLM serving systems [17, 38] improve GPU sharing through multiplexing or elastic memory management, but they still colocate static weightsand dynamic KV-cache under the same GPU pool. This couples the KV-cache capacity visible to a request with the amount of memory already occupied by weights. It also exposes an algorithm-system mismatch: KV-head-limited attention algorithms such as MLA [12] and MQA [15, 34] reduce per-token KV-cache, while serving engines often use DP attention for such models to increase aggregate

FAQs

为什么冷 MoE 模型会导致 GPU 显存利用率低？

权重是模型决定且稳定的，而 KV-cache 随请求的上下文长度、并发和速率而变化且在解码后回收。冷模型很少同时达到 KV-cache 峰值，导致按最坏情况为每个模型预留 KV 容量会浪费显存。

CrossPool 如何解决仅做 KV-cache 共享仍不够的问题？

CrossPool 将 FFN 权重和 KV-cache 分到不同的 GPU 内存池：权重池汇聚多个冷模型的 FFN 权重，KV-cache 池动态共享用于活动请求的 KV-cache，并通过池边界交换隐藏状态而非 KV 张量以保持注意力本地访问。

CrossPool 的哪些机制降低开销并提升长上下文能力？

它包含 KV-cache planner 与 virtualizer，并使用层级流水调度隐藏隐藏状态传输，同时采用持久化内核与降低控制开销的策略来减少 CPU-GPU 控制开销，从而在支持突发长上下文请求时保持竞争性能。

CrossPool 的性能提升体现在什么指标上？

文中指出在效率的 GPU 内存池支持下，CrossPool 能在突发长上下文请求场景中表现优于基于 kvcached 的多-LLM 服务系统，并将 P99 TBT 降低最多 10.4 倍。

Aug 21, 2026

Uploaded by นรินทร์