[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84639-en":3,"doc-seo-84639-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84639,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving","Disaggregated LLM serving separates prefill and decode onto distinct GPU pools, which avoids phase interference but introduces load imbalance under bursty, heavy-tailed workloads. In a production-style A100 2P2D cluster, prefill accounts for only 2–23% of P95 time-to-first-token (TTFT), with the remainder due to prefill queuing and inter-node GPU-GPU KV-cache transfer. The work proposes a proactive, load-aware prefill-deflecting scheduler (Kairos) that routes queued requests to decode nodes by searching chunk schedules that preserve decode time-between-tokens (TBT) SLO headroom and eliminate KV transfer. On vLLM with DeepSeek-V2-Lite traces, P95 TTFT improves by up to 81% and SLO attainment by up to 79%, with sub-millisecond routing overhead.","Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving  \nShrikara Arun  \nMicrosoft  \n[t-sarun@microsoft.com](t-sarun@microsoft.com)  \nAnjaly Parayil  \nMicrosoft  \n[aparayil@microsoft.com](aparayil@microsoft.com)  \nSrikant Bharadwaj  \nMicrosoft  \n[srikant.bharadwaj@microsoft.com](srikant.bharadwaj@microsoft.com)  \narXiv :2607 .02043v 1 [ cs .DC] 2 Jul 2026  \nRenee St. Amant  \nMicrosoft [reneestamant@microsoft.com](reneestamant@microsoft.com)  \nAbstract  \nDisaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavytailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Timeto-First-Token (TTFT) . Queuing and inter-node GPU-GPU KV-cache transfer account for the rest.  \nWe present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunkedprefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO, and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the internode KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.  \nACM Reference Format:  \nShrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, and Victor Rühle. 2026. Towards Load-Aware Prefill Deflection for DisaggregatedLLM Serving. In . ACM, New York, NY, USA, 12 pages. [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn)  \n1 Introduction  \nLarge language model (LLM) inference has become a dominant datacenter workload, and serving it efficiently at scale demands careful management of both latency and throughput. Modern production systems increasingly use disaggregated inference: prefill and decode are executed on separate GPU node pools, allowing each phase to scale independently and eliminating the GPU contention that arises when both phases compete on the same hardware [15, 16, 21] .  \nVictor Rühle  \nMicrosoft  \n[virueh@microsoft.com](virueh@microsoft.com)  \nPrefill nodes become the critical bottleneck. In a disaggregated cluster, prefill nodes bear the full cost of processing incoming prompts before any token is generated. Under realistic workloads where prompt lengths are heavy-tailed and request arrivals are bursty, prefill nodes saturate and requests queue while decode nodes have idle compute capacity. Meanwhile, decode nodes sit underutilized. The decode phase itself is memory-bound, leaving some compute on the table and disaggregated serving never utilizes this headroom. Previous works such as Splitwise [15], DistServe [21], Mooncake [16], and TetriInfer [6] all share this design. On a cluster with 2 prefill and 2 decode nodes (2P2D), running representative production-style workloads, we find that prefill execution itself accounts for only 2-23% of P95 TTFT with the rest being prefill queuing wait and inter-node GPU-GPU KV-cache transfer (§3, Table 3) . The result is that queue size grows on prefill nodes, especially during bursts of arrivals, while decode node compute which could be used to alleviate the pressure is underutilized. Rerouting the prefill phase of requests to the decode node also has the added benefit of removing the need for KV cache transfer, since the cache is built directly on the decode node. This is especially effective during bursts of reques","cbCailfrdwdHvxVJ","https://ap.wps.com/l/cbCailfrdwdHvxVJ","pdf",1092359,1,12,"English","en",105,"# Introduction\n## Load imbalance in disaggregated serving\n## Prefill deflection with TBT-awareness\n## Scheduler decision criteria\n# Kairos: time-aware prefill deflection","[{\"question\":\"What problem does disaggregated LLM serving face under bursty workloads?\",\"answer\":\"Prefill nodes saturate while decode nodes remain underutilized, causing requests to queue. This leads to a large portion of P95 TTFT being dominated by prefill waiting and KV-cache transfer rather than prefill compute itself.\"},{\"question\":\"How does the proposed scheduler decide whether to deflect a request to a decode node?\",\"answer\":\"For each queued prefill request, it estimates the TTFT if kept on the prefill node, computes the decode-node prefill completion time across candidate safe chunk sizes, and checks available TBT headroom. It deflects only when doing so reduces tail latency without violating TBT SLO constraints.\"},{\"question\":\"What benefit comes from running prefill of deflected requests on the decode node?\",\"answer\":\"Because the KV cache is built directly on the decode node, inter-node GPU-GPU KV-cache transfer is eliminated. This is especially effective during request bursts when transfer can become a bottleneck.\"}]",1784197399,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"towards-load-aware-prefill-deflection-for-disaggregated-llm-serving","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/towards-load-aware-prefill-deflection-for-disaggregated-llm-serving/84639/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does disaggregated LLM serving face under bursty workloads?","Question",{"text":75,"@type":76},"Prefill nodes saturate while decode nodes remain underutilized, causing requests to queue. This leads to a large portion of P95 TTFT being dominated by prefill waiting and KV-cache transfer rather than prefill compute itself.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed scheduler decide whether to deflect a request to a decode node?",{"text":80,"@type":76},"For each queued prefill request, it estimates the TTFT if kept on the prefill node, computes the decode-node prefill completion time across candidate safe chunk sizes, and checks available TBT headroom. It deflects only when doing so reduces tail latency without violating TBT SLO constraints.",{"name":82,"@type":73,"acceptedAnswer":83},"What benefit comes from running prefill of deflected requests on the decode node?",{"text":84,"@type":76},"Because the KV cache is built directly on the decode node, inter-node GPU-GPU KV-cache transfer is eliminated. This is especially effective during request bursts when transfer can become a bottleneck.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]