[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82795-en":3,"doc-seo-82795-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82795,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving","Online large language model (LLM) serving underpins modern AI services, yet dynamic request patterns often create severe workload skewness, where a small subset of model instances handles most traffic. Traditional instance-level scaling is coarse-grained, causing large cold-start delays when scaling up and leaving systems exposed during sudden bursts when scaling down. CoCoScale introduces layer-wise dynamic scaling that expands hot-layer parallelism onto reclaimed idle resources for elastic data parallelism without changing architectures. Evaluations show 97.9%–99.3% lower cold-start latency, and 20.7%–28.1% lower average latency in production traces.","arXiv :2607 .04 18 1v 1 [ cs .DC] 5 Jul 2026  \nCoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving  \nJINGFENG WU∗ , YIYUAN HE∗ , MINXIAN XU†, and XITONG GAO, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China  \nCHONG MA, LE CHEN, MIN SHEN, and LIN QU, Alibaba Group Inc, China KEJIANG YE, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China CHENGZHONG XU, State Key Lab ofIOTSC, University of Macau, China  \nOnline large language model (LLM) serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for finegrained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%–99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7%–28.1% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency.  \nCCS Concepts: • Computing methodologies → Computer Systems; • Computer systems organization → Distributed Systems.  \nAdditional Key Words and Phrases: LLM, Inference Serving, Module Scaling, Replication, Migration  \nACM Reference Format:  \nJingfeng Wu, Yiyuan He, Minxian Xu, Xitong Gao, Chong Ma, Le Chen, Min Shen, Lin Qu, Kejiang Ye, and Chengzhong Xu. 2026. CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving. ACM Trans. Arch. Code Optim. 1, 1, Article 1 (January 2026), 22 pages. [https://doi.org/XXXXXX](https://doi.org/XXXXXX)  \n1 Introduction  \nThe advent of state-of-the-art LLMs, such as GPT 5.2 [1], Qwen 3 [36], and DeepSeek V3 [7], has reshaped user experiences and driven the rapid growth of online LLM serving. Consequently, major cloud platforms [3, 11, 20] have emerged as the foundational infrastructure for diverse downstream applications.  \nDespite these advancements, online serving environments contend with significant operational challenges stemming from extreme workload volatility and imbalance. Specifically, request patterns exhibit unpredictable dynamics, leading to frequent load fluctuations across instances. Furthermore, there is a severe workload disparity among heterogeneous models. As demonstrated by the OpenRouter [22] leaderboard, the top 25% of model instances process 75% of the total traffic,  \n∗ Both authors contributed equally to this research.  \n†Minxian Xu is the corresponding author.  \nAuthors’ Contact Information: Jingfeng Wu, [jf.wu2@siat.ac.cn](jf.wu2@siat.ac.cn); Yiyuan He, [yy.he2@siat.ac.cn](yy.he2@siat.ac.cn); Minxian Xu, [mx.xu@siat.ac.cn](mx.xu@siat.ac.cn);  \nXitong Gao, [xt.gao@siat.ac.cn](xt.gao@siat.ac.cn), Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen,  \nChina; Chong Ma, [machong.mc@alibaba-inc.com](machong.mc@alibaba-inc.com); Le Chen, [donghuai.cl@taobao.com](donghuai.cl@taobao.com); Min Shen, [shenmin.sm@taobao.com](shenmin.sm@taobao.com);  \nLin Qu, [xide.ql@taobao.com](xide.ql@taobao.com), Alibaba Group Inc, Hangzhou, China; Kejiang Ye, [kj.ye@siat.ac.cn](kj.ye@siat.ac.cn","cbCaiogv0BvvhOtB","https://ap.wps.com/l/cbCaiogv0BvvhOtB","pdf",1805332,1,22,"English","en",105,"# Introduction\n## Workload volatility and skewness\n## Limits of instance-wise scaling\n## CoCoScale layer-wise scaling approach","[{\"question\":\"How does CoCoScale improve scaling without changing model architectures?\",\"answer\":\"CoCoScale performs layer-wise dynamic scaling by selectively expanding the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism while avoiding architecture changes or added hardware overhead.\"}]",1784182978,55,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"cocoscale-leveraging-layer-wise-scaling-to-unlock-the-potential-of-online-llm-serving","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/cocoscale-leveraging-layer-wise-scaling-to-unlock-the-potential-of-online-llm-serving/82795/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"How does CoCoScale improve scaling without changing model architectures?","Question",{"text":75,"@type":76},"CoCoScale performs layer-wise dynamic scaling by selectively expanding the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism while avoiding architecture changes or added hardware overhead.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,115,120,123,127],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":45,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":45,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":113,"slug":114},30,"research-report",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":98,"slug":130},19,"General","general"]