[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83611-en":3,"doc-seo-83611-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83611,16904993612988,"Olivia Brown","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Lynx Progressive Speculative Quantization for accelerating KV Transfer in Long-context Inference","Long-context inference for large language models relies on disaggregated execution where Key-Value (KV) cache must be transferred across the network before decoding can start, creating serious Time-to-First-Token (TTFT) latency. Existing KV quantization lowers transfer volume but keeps a rigid “receive-all-then-decode” barrier, limiting overlap between communication and computation. Lynx removes this assumption by using progressive, split-stream KV transfer: an Anchor stream carries high-priority bits to start decoding early while a Residual stream refines accuracy via verification.","Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference  \nWenchen Han Gingfung Matthew Yeung Marco Barletta  \nUniversity College London Huawei Huawei  \nWilliam Toner Amory Hoste Adam Barker  \nHuawei Huawei Huawei  \narXiv :2607 .0 183 1v 1 [ cs .DC] 2 Jul 2026  \nABSTRACT  \nLong-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy.  \nWe challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding.  \nAcross multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to 1.43× and improving accuracy overstate-of-the-art by up to 5. 1% .  \n1 INTRODUCTION  \nLarge Language Model (LLMs) inference is increasingly dominated by long-context workloads, where prompts of tens to hundreds of thousands of tokens are common in retrivalaugmented generation [54], agentic systems [6], and code intelligence [52] . State-of-the-art models such as Gemini 3.0 [13], Qwen 3 [51], and DeepSeek 3.2 [30] already support context windows of more than 1 million tokens.  \nIn order to sustain throughput under these workloads, modern serving systems disaggregate inference into a computebound prefill stage and a memory-bound decode stage, placing them on separate accelerator instances [29, 37, 61]. While prefill-decode disaggregation improves hardware utilization, it introduces a fundamental bottleneck: the transfer of KeyValue (KV) cache1 between instances [55] .  \nThe KV cache grows linearly with context length and model depth, reaching tens of gigabytes for a single longcontext request [40, 53] . Even on high-bandwidth interconnects, transferring this state introduces substantial latency that directly inflates Time-to-First-Token (TTFT) and delays decoding. As context lengths continue to scale, KV transfer latency, not compute, becomes the dominant limiter of end-to-end inference performance in disaggregated deployments [56, 60] .  \nExisting systems mitigate this cost by quantizing the KV cache prior to transfer, which compresses KV cache 2 to INT4 or INT8 [16, 33, 35, 56] . This effectively reduces the KV volume, which yields accelerated KV transfer over the network. As illustrated in Figure 1, while these approaches reduce data volume, they preserve a strict serialization barrier: the decode stage cannot begin until the full (compressed) KV cache has arrived and been reconstructed. Layer-wise pipelines partially overlap communication and computation, but when network bandwidth is the bottleneck, decoding still stalls at the boundary of each layer. As a result, current KV compression schemes reduce ","cbCaipbRYLee8kz9","https://ap.wps.com/l/cbCaipbRYLee8kz9","pdf",1975868,3,1,15,"English","en",105,"# Abstract\n# Introduction\n## Disaggregated long-context bottleneck\n## Limitations of existing KV quantization\n## Core insight and progressive split-stream transfer\n## Lynx design and verification","[{\"question\":\"为什么长上下文推理在解耦部署中会出现明显的 TTFT 延迟？\",\"answer\":\"在解耦推理中，解码阶段必须等到网络完成 KV cache 的跨实例传输并可用后才能开始，因此 KV 传输延迟会直接推高 TTFT。\"},{\"question\":\"Lynx 反对了哪项传统假设来加速 KV 传输？\",\"answer\":\"Lynx 挑战“KV cache 必须作为不可分割整体接收完毕才能使用”的假设，指出 KV 不同比特对注意力计算与精度的贡献并不均等。\"},{\"question\":\"Lynx 的 Anchor/Ressidual 两流如何帮助在不降低准确率的情况下提前开始解码？\",\"answer\":\"Anchor 流携带最主要的高优先级比特以触发解码，并在 Residual 流并发传输的过程中进行推测解码；随后通过验证来确保结果等价于更高精度的解码。\"}]",1784189261,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"lynx-progressive-speculative-quantization-for-accelerating-kv-transfer-in-long-context-inference","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/lynx-progressive-speculative-quantization-for-accelerating-kv-transfer-in-long-context-inference/83611/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么长上下文推理在解耦部署中会出现明显的 TTFT 延迟？","Question",{"text":75,"@type":76},"在解耦推理中，解码阶段必须等到网络完成 KV cache 的跨实例传输并可用后才能开始，因此 KV 传输延迟会直接推高 TTFT。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Lynx 反对了哪项传统假设来加速 KV 传输？",{"text":80,"@type":76},"Lynx 挑战“KV cache 必须作为不可分割整体接收完毕才能使用”的假设，指出 KV 不同比特对注意力计算与精度的贡献并不均等。",{"name":82,"@type":73,"acceptedAnswer":83},"Lynx 的 Anchor/Ressidual 两流如何帮助在不降低准确率的情况下提前开始解码？",{"text":84,"@type":76},"Anchor 流携带最主要的高优先级比特以触发解码，并在 Residual 流并发传输的过程中进行推测解码；随后通过验证来确保结果等价于更高精度的解码。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]