[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-135426-en":3,"doc-seo-135426-105":31,"detail-sidebar-cat-0-en-105":97},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":28,"seo_description":14,"update_tm":29,"read_time":30},135426,2336475104957,"นรินทร์","https://ap-avatar.wpscdn.com/avatar/22000c4c6bd8a5076e1?x-image-process=image/resize,m_fixed,w_180,h_180&k=1787554080175789136",8,"Research & Report","CrossPool - Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation","Emerging LLM services increasingly host sparse Mixture-of-Experts (MoE) models, yet most remain cold, creating GPU memory imbalance: model weights are stable while KV-cache is transient and demand-driven. Worst-case per-model KV reservation wastes memory, but naive KV sharing fails when weights and KV-cache share a monolithic pool. CrossPool separates FFN weights and KV-cache into distinct GPU memory pools, planning and virtualizing shared KV-cache while keeping attention local. It reduces CPU-GPU control overhead and improves long-context support, achieving up to 10.4× lower P99 TBT versus state of the art.","CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation  \nZhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang†  \nSchool of Software, Beihang University  \n{yezr, woty, dinghaoxue, mingmingzhang,yuchenteng,hucm, [renyuyang}@buaa.edu.cn](renyuyang}@buaa.edu.cn)  \narXiv :2606 .24506v2 [ cs .DC] 26 Jun 2026  \nAbstract  \nEmerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and demand-determined. Because cold models rarely reach peak KV-cache demand at the same time, reserving worst-case KV capacity per model wastes memory; a shared KV-cache pool can instead provision aggregate active demand. However, KV-cache sharing is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. Static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support. We present CrossPool, a serving engine for cold MoE models that separates FFN weights and KV-cache into two GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local to KV-cache. CrossPool combines a KV-cache planner and virtualizer, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU control overhead. With efficient GPU memory pooling, CrossPool underpins bursty long-context requests and outperforms the state-of-the-art kvcached-based multi-LLM serving system, reducing P99 TBT by up to 10.4× .  \n1 Introduction  \nEmerging large language models (LLMs) now power applications ranging from chatbots [14, 20, 29] to agentic assistants [4, 30, 31]. To serve this demand, model providers [2, 6, 11] deploy increasingly capable, fast-evolving LLMs in their datacenters. Many recent frontier and open-weight models support long context for complex tasks and use Mixture-ofExperts (MoE) architectures [9, 13, 15, 18, 22, 25, 35] to scale total parameters while keeping computation moderate.  \nThis trend makes cold-model serving increasingly important. Recent studies [17, 19, 36] show that a few hot models serve most requests, while many cold models are underutilized. A serving cluster that keeps many such models online therefore pays the memory cost of large MoE weights even  \n†Corresponding Author.  \nwhen most models have few active requests. Long-context serving adds a second pressure point: KV-cache can grow to a large fraction of GPU memory, but unlike weights, it is allocated only while requests are active and is reclaimed after decode finishes.  \nThe key opportunity is that weights and KV-cache have different lifetimes. Model weights are model-determined and stable, whereas KV-cache is demand-determined and transient: its footprint varies with request rate, context length, and concurrency. Since cold models rarely peak simultaneously, reserving worst-case KV-cache for each model wastes GPU memory. Hence, a shared KV-cache pool can instead provision for aggregate active demand.  \nSimple memory sharing within a unified GPU memory pool is insufficient. Existing multi-LLM serving systems [17, 38] improve GPU sharing through multiplexing or elastic memory management, but they still colocate static weightsand dynamic KV-cache under the same GPU pool. This couples the KV-cache capacity visible to a request with the amount of memory already occupied by weights. It also exposes an algorithm-system mismatch: KV-head-limited attention algorithms such as MLA [12] and MQA [15, 34] reduce per-token KV-cache, while serving engines often use DP attention for such models to increase aggregate ","cbCaihS5RLejjMfX","https://ap.wps.com/l/cbCaihS5RLejjMfX","pdf",1883792,7,1,9,"English","en",105,"# Introduction\n## Cold-model serving and memory imbalance\n## Opportunity: different lifetimes of weights vs KV-cache\n## Limits of unified memory sharing in multi-LLM systems\n# CrossPool framework\n## Disaggregated FFN weights pool and dynamic KV-cache pool\n## Pool boundary and hidden-state exchange\n## Design challenges and contributions\n# KV-cache planner and virtualizer\n# Evaluation context (figures and metrics)","[{\"question\":\"为什么冷 MoE 模型会导致 GPU 显存利用率低？\",\"answer\":\"权重是模型决定且稳定的，而 KV-cache 随请求的上下文长度、并发和速率而变化且在解码后回收。冷模型很少同时达到 KV-cache 峰值，导致按最坏情况为每个模型预留 KV 容量会浪费显存。\"},{\"question\":\"CrossPool 如何解决仅做 KV-cache 共享仍不够的问题？\",\"answer\":\"CrossPool 将 FFN 权重和 KV-cache 分到不同的 GPU 内存池：权重池汇聚多个冷模型的 FFN 权重，KV-cache 池动态共享用于活动请求的 KV-cache，并通过池边界交换隐藏状态而非 KV 张量以保持注意力本地访问。\"},{\"question\":\"CrossPool 的哪些机制降低开销并提升长上下文能力？\",\"answer\":\"它包含 KV-cache planner 与 virtualizer，并使用层级流水调度隐藏隐藏状态传输，同时采用持久化内核与降低控制开销的策略来减少 CPU-GPU 控制开销，从而在支持突发长上下文请求时保持竞争性能。\"},{\"question\":\"CrossPool 的性能提升体现在什么指标上？\",\"answer\":\"文中指出在效率的 GPU 内存池支持下，CrossPool 能在突发长上下文请求场景中表现优于基于 kvcached 的多-LLM 服务系统，并将 P99 TBT 降低最多 10.4 倍。\"}]","CrossPool - Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation | PDF",1787312014,23,{"code":4,"msg":32,"data":33},"ok",{"site_id":25,"language":24,"slug":34,"title":13,"keywords":35,"description":14,"schema_data":36,"social_meta":92,"head_meta":94,"extra_data":96,"updated_unix":29},"crosspool-efficient-multi-llm-serving-for-cold-moe-models-through-kv-cache-and-weight-disaggregation","",{"@graph":37,"@context":91},[38,55,70],{"@type":39,"itemListElement":40},"BreadcrumbList",[41,45,49,52],{"item":42,"name":43,"@type":44,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":46,"name":47,"@type":44,"position":48},"https://docshare.wps.com/document/","Document",2,{"item":50,"name":12,"@type":44,"position":51},"https://docshare.wps.com/document/research-report/",3,{"item":53,"name":13,"@type":44,"position":54},"https://docshare.wps.com/document/crosspool-efficient-multi-llm-serving-for-cold-moe-models-through-kv-cache-and-weight-disaggregation/135426/",4,{"url":53,"name":13,"@type":56,"author":57,"headline":13,"publisher":59,"fileFormat":62,"inLanguage":24,"description":14,"dateModified":63,"datePublished":64,"encodingFormat":62,"isAccessibleForFree":65,"interactionStatistic":66},"DigitalDocument",{"name":9,"@type":58},"Person",{"url":42,"name":60,"@type":61},"DocShare","Organization","application/pdf","2026-09-02","2026-08-21",true,{"@type":67,"interactionType":68,"userInteractionCount":20},"InteractionCounter",{"@type":69},"ViewAction",{"@type":71,"mainEntity":72},"FAQPage",[73,79,83,87],{"name":74,"@type":75,"acceptedAnswer":76},"为什么冷 MoE 模型会导致 GPU 显存利用率低？","Question",{"text":77,"@type":78},"权重是模型决定且稳定的，而 KV-cache 随请求的上下文长度、并发和速率而变化且在解码后回收。冷模型很少同时达到 KV-cache 峰值，导致按最坏情况为每个模型预留 KV 容量会浪费显存。","Answer",{"name":80,"@type":75,"acceptedAnswer":81},"CrossPool 如何解决仅做 KV-cache 共享仍不够的问题？",{"text":82,"@type":78},"CrossPool 将 FFN 权重和 KV-cache 分到不同的 GPU 内存池：权重池汇聚多个冷模型的 FFN 权重，KV-cache 池动态共享用于活动请求的 KV-cache，并通过池边界交换隐藏状态而非 KV 张量以保持注意力本地访问。",{"name":84,"@type":75,"acceptedAnswer":85},"CrossPool 的哪些机制降低开销并提升长上下文能力？",{"text":86,"@type":78},"它包含 KV-cache planner 与 virtualizer，并使用层级流水调度隐藏隐藏状态传输，同时采用持久化内核与降低控制开销的策略来减少 CPU-GPU 控制开销，从而在支持突发长上下文请求时保持竞争性能。",{"name":88,"@type":75,"acceptedAnswer":89},"CrossPool 的性能提升体现在什么指标上？",{"text":90,"@type":78},"文中指出在效率的 GPU 内存池支持下，CrossPool 能在突发长上下文请求场景中表现优于基于 kvcached 的多-LLM 服务系统，并将 P99 TBT 降低最多 10.4 倍。","https://schema.org",{"og:url":53,"og:type":93,"og:title":13,"og:site_name":60,"og:description":14},"article",{"robots":95,"canonical":53},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":98},[99,103,107,111,116,121,125,128,132,135,139],{"id":21,"doc_module":4,"doc_module_name":47,"category_name":100,"show_sort_weight":101,"slug":102},"Story & Novel",90,"story-novel",{"id":48,"doc_module":4,"doc_module_name":47,"category_name":104,"show_sort_weight":105,"slug":106},"Literature",80,"literature",{"id":54,"doc_module":4,"doc_module_name":47,"category_name":108,"show_sort_weight":109,"slug":110},"Exam",70,"exam",{"id":112,"doc_module":4,"doc_module_name":47,"category_name":113,"show_sort_weight":114,"slug":115},5,"Comic",60,"comic",{"id":117,"doc_module":4,"doc_module_name":47,"category_name":118,"show_sort_weight":119,"slug":120},6,"Technology",50,"technology",{"id":20,"doc_module":4,"doc_module_name":47,"category_name":122,"show_sort_weight":123,"slug":124},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":47,"category_name":12,"show_sort_weight":126,"slug":127},30,"research-report",{"id":22,"doc_module":4,"doc_module_name":47,"category_name":129,"show_sort_weight":130,"slug":131},"Religion & Spirituality",20,"religion-spirituality",{"id":130,"doc_module":4,"doc_module_name":47,"category_name":133,"show_sort_weight":130,"slug":134},"World Cup","world-cup",{"id":136,"doc_module":4,"doc_module_name":47,"category_name":137,"show_sort_weight":136,"slug":138},10,"Lifestyle","lifestyle",{"id":140,"doc_module":4,"doc_module_name":47,"category_name":141,"show_sort_weight":112,"slug":142},19,"General","general"]