CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

Jul 16, 2026

Upload by Violet

Online large language model (LLM) serving underpins modern AI services, yet dynamic request patterns often create severe workload skewness, where a small subset of model instances handles most traffic. Traditional instance-level scaling is coarse-grained, causing large cold-start delays when scaling up and leaving systems exposed during sudden bursts when scaling down. CoCoScale introduces layer-wise dynamic scaling that expands hot-layer parallelism onto reclaimed idle resources for elastic data parallelism without changing architectures. Evaluations show 97.9%–99.3% lower cold-start latency, and 20.7%–28.1% lower average latency in production traces.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Introduction

Workload volatility and skewness

Limits of instance-wise scaling

CoCoScale layer-wise scaling approach

arXiv :2607 .04 18 1v 1 [ cs .DC] 5 Jul 2026 CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving JINGFENG WU∗ , YIYUAN HE∗ , MINXIAN XU†, and XITONG GAO, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China CHONG MA, LE CHEN, MIN SHEN, and LIN QU, Alibaba Group Inc, China KEJIANG YE, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China CHENGZHONG XU, State Key Lab ofIOTSC, University of Macau, China Online large language model (LLM) serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for finegrained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%–99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7%–28.1% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency. CCS Concepts: • Computing methodologies → Computer Systems; • Computer systems organization → Distributed Systems. Additional Key Words and Phrases: LLM, Inference Serving, Module Scaling, Replication, Migration ACM Reference Format: Jingfeng Wu, Yiyuan He, Minxian Xu, Xitong Gao, Chong Ma, Le Chen, Min Shen, Lin Qu, Kejiang Ye, and Chengzhong Xu. 2026. CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving. ACM Trans. Arch. Code Optim. 1, 1, Article 1 (January 2026), 22 pages. [https://doi.org/XXXXXX](https://doi.org/XXXXXX) 1 Introduction The advent of state-of-the-art LLMs, such as GPT 5.2 [1], Qwen 3 [36], and DeepSeek V3 [7], has reshaped user experiences and driven the rapid growth of online LLM serving. Consequently, major cloud platforms [3, 11, 20] have emerged as the foundational infrastructure for diverse downstream applications. Despite these advancements, online serving environments contend with significant operational challenges stemming from extreme workload volatility and imbalance. Specifically, request patterns exhibit unpredictable dynamics, leading to frequent load fluctuations across instances. Furthermore, there is a severe workload disparity among heterogeneous models. As demonstrated by the OpenRouter [22] leaderboard, the top 25% of model instances process 75% of the total traffic, ∗ Both authors contributed equally to this research. †Minxian Xu is the corresponding author. Authors’ Contact Information: Jingfeng Wu, [jf.wu2@siat.ac.cn](jf.wu2@siat.ac.cn); Yiyuan He, [yy.he2@siat.ac.cn](yy.he2@siat.ac.cn); Minxian Xu, [mx.xu@siat.ac.cn](mx.xu@siat.ac.cn); Xitong Gao, [xt.gao@siat.ac.cn](xt.gao@siat.ac.cn), Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China; Chong Ma, [machong.mc@alibaba-inc.com](machong.mc@alibaba-inc.com); Le Chen, [donghuai.cl@taobao.com](donghuai.cl@taobao.com); Min Shen, [shenmin.sm@taobao.com](shenmin.sm@taobao.com); Lin Qu, [xide.ql@taobao.com](xide.ql@taobao.com), Alibaba Group Inc, Hangzhou, China; Kejiang Ye, [kj.ye@siat.ac.cn](kj.ye@siat.ac.cn

FAQs

How does CoCoScale improve scaling without changing model architectures?

CoCoScale performs layer-wise dynamic scaling by selectively expanding the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism while avoiding architecture changes or added hardware overhead.

Jul 16, 2026

Upload by Violet