[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83665-en":3,"doc-seo-83665-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83665,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","PEEK Predictive Queue-Informed KV Cache Management for LLM Serving","PEEK presents a lightweight scheduling and eviction framework for large language model serving, targeting online streaming scenarios while remaining applicable to offline batch. It builds an incremental radix tree over the pending request queue to expose prefix-sharing clusters that existing engines do not surface. A low-overhead dual-walk performs longest-prefix matching against the engine’s prefix cache, admits cluster pioneers first, and uses a queue-aware eviction hook to protect blocks tied to queued demand. A multi-lane stride scheduler bounds starvation and adapts fairness as the queue composition changes. Experiments on SGLang and vLLM across five workloads show up to 4×H100-scale throughput and latency gains under high KV pressure.","arXiv :2607 .02525v1 [ cs .DC] 10 May 2026  \nPEEK: Predictive Queue-Informed KV Cache Management for LLM Serving ∗  \nBing Xie† Zhipeng Wang Masahiro Tanaka Zheng Zhen  \nAbstract  \nWe present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine’s prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation.  \nOn SGLang and vLLM across five workloads up to 4×H100 (DP=2 over TP=2), PEEK delivers up to 3.0 × /2 .6 × cache hit, 7.9 × /7 .1 × TTFT, 6.7 × /5 .5 × E2E, and 3.6 × /4 .5 × throughput gains over each engine’s strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.  \n1 Introduction  \nModern LLM serving engines optimize KV cache reuse: SGLang’s RadixAttention [Zheng et al., 2024a] provides longest-prefix matching over cached blocks; vLLM’s PagedAttention [Kwon et al., 2023] enables block-level sharing via OS-style paging. Yet both stop at the cache—neither reasons about which waiting requests share prefixes with each other, leaving the predictive signal in the request queue unexploited (Table 1) . SGLang’s prefix-aware policies (LPM, DFS_WEIGHT) remain bound to current cache state and degrade once the cache is cold; vLLM defaults to FCFS, which under memory pressure devolves into cache thrashing as interleaved prefix groups evict one another (§2) . In both engines, scheduling and eviction are decoupled, precluding joint optimization—inefficiencies magnified in long-context workloads such as RAG and agentic pipelines.  \nWe present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on online. Our motivation is a simple observation: the waiting queue is itself a structured workload whose prefix-sharing relationships can directly drive KV-cache reuse. PEEK retains LPM’s prefix-match objective but feeds it richer signals at lower overhead from an incremental radix tree via three mechanisms (Appendix Table 3):  \n1. See the queue’s structure for cheap (§3.1). PEEK maintains an incremental radix tree over the pending queue—no per-cycle rebuild, O (D) per insert/remove, O(1) per query on scheduling/eviction signals. To compute LPM hits against the engine cache, PEEK’s dual-walk codescends the pending tree against SGLang’s radix cache in one pass, returning all per-request hits in O (C×D) vs. a naive O (N×D); for high-sharing workloads C ≪ N, an N/C amortization3. On vLLM’s flat block-hash cache, PEEK falls back to per-request probes but still reads cluster signals off the tree—structure stock vLLM cannot see.  \n∗ Code: [https://github.com/xiexbing/peek](https://github.com/xiexbing/peek)[ ](https://github.com/xiexbing/peek)†Corresponding author.  \n3 N = queue size; C = number of prefix-sharing clusters; D = prompt depth in the tree (∼ O(log N)) . Preprint.  \n\n| FCFS\u003Cbr>(vLLM/SGLang) |  | LPM DFS_W\u003Cbr>(SGLang) (SGLang) | PEEK\u003Cbr>(Ours) |\n| --- | --- | --- | --- |\n| Scheduling |  |  |  |\n| Prefix-aware | ✗ | Cache tree‡ Cache tree‡ | Queue trie |\n| Ordering | Arrival | Per-request Batch | Global |\n| Caching & Eviction\u003Cbr>Mechanism APC / RadixCache RadixCache RadixCache Pending trie\u003Cbr>Eviction LRU/LFU LRU/LFU LRU/LFU Queue-aware |  |  |  |\n\nTable 1: Scheduling and eviction mechanisms in current serving engines. PEEK is the only engine that jointly optimizes scheduling and eviction. ‡LPM/DFS_W see only cache state, not ","cbCaiqvpT05wNe6V","https://ap.wps.com/l/cbCaiqvpT05wNe6V","pdf",1286120,6,1,26,"English","en",105,"# Introduction\n## Cluster-aware admit and evict\n## Bound starvation, adaptively\n# Evaluation","[{\"question\":\"What problem does PEEK address in LLM serving engines?\",\"answer\":\"PEEK targets inefficiencies caused by decoupled scheduling and eviction, where current engines exploit KV cache state but ignore structured prefix-sharing relationships inside the waiting queue, leading to cache thrashing under memory pressure and long-context workloads.\"},{\"question\":\"How does PEEK obtain predictive signals from the request queue?\",\"answer\":\"PEEK maintains an incremental radix tree over the pending queue and uses a dual-walk procedure to match this structure against the engine’s prefix cache, producing longest-prefix matches for waiting requests and identifying prefix-sharing clusters.\"},{\"question\":\"How does PEEK prevent starvation when prioritizing prefix clusters?\",\"answer\":\"PEEK adds a fairness lane based on arrival order with stride-interleaving against cluster-first admission. A dynamic-lane controller increases fairness when the queue becomes singleton-heavy or as the oldest singleton’s wait approaches its SLO using EMA-smoothed estimates.\"}]",1784189616,66,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"peek-predictive-queue-informed-kv-cache-management-for-llm-serving","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/peek-predictive-queue-informed-kv-cache-management-for-llm-serving/83665/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does PEEK address in LLM serving engines?","Question",{"text":76,"@type":77},"PEEK targets inefficiencies caused by decoupled scheduling and eviction, where current engines exploit KV cache state but ignore structured prefix-sharing relationships inside the waiting queue, leading to cache thrashing under memory pressure and long-context workloads.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does PEEK obtain predictive signals from the request queue?",{"text":81,"@type":77},"PEEK maintains an incremental radix tree over the pending queue and uses a dual-walk procedure to match this structure against the engine’s prefix cache, producing longest-prefix matches for waiting requests and identifying prefix-sharing clusters.",{"name":83,"@type":74,"acceptedAnswer":84},"How does PEEK prevent starvation when prioritizing prefix clusters?",{"text":85,"@type":77},"PEEK adds a fairness lane based on arrival order with stride-interleaving against cluster-first admission. A dynamic-lane controller increases fairness when the queue becomes singleton-heavy or as the oldest singleton’s wait approaches its SLO using EMA-smoothed estimates.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]