[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82221-en":3,"doc-seo-82221-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82221,1374391974468,"Eden","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","KV-PRM Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling","Process Reward Models (PRMs) effectively guide multi-agent test-time scaling (TTS) for LLM reasoning, yet existing text-based PRMs re-encode full trajectory text for every candidate. In long rollouts this scoring cost grows quadratically with sequence length, becoming a major bottleneck for long-context scenarios. KV-PRM eliminates redundant re-encoding by reading the key-value (KV) cache produced during generation. Verifying a single token against the pre-existing KV cache reduces scoring from O(L2) to O(L) and yields theoretical and empirical efficiency and performance gains across MATH, GSM8K, and AIME.","arXiv :2607 .09 153v 1 [ cs .AI] 10 Jul 2026  \nKV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling  \nPeng Kuang 1 , Haibo Jin 1 , Xiaoyu Han 1 , Yanli Wang2 , Xiaopeng Yuan 1 , Ye Yu 1 , Kaidi Xu3 , Haohan Wang 1  \n1University of Illinois Urbana-Champaign, 2Imperial College London,  \n3 City University of Hong Kong  \n[pengk2@illinois.edu](pengk2@illinois.edu) , [kaidixu@cityu.edu.hk](kaidixu@cityu.edu.hk) , [haohanw@illinois.edu](haohanw@illinois.edu)  \nAbstract  \nProcess Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLMbased multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs’ application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM’s generation phase. By processing a single\"verify token\" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O (L2 ) to O(L) . We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KVPRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000 × reduction in scoring FLOPs, a 37 × reduction in latency, and a 34 × reduction in per-sequence memory footprint compared to text-based PRMs.  \n1 Introduction  \nScaling test-time compute has recently emerged as a highly effective direction for improving the capabilities of Large Language Models (LLMs) on complex reasoning tasks [1, 2, 3] . Instead of relying on a single greedy decoding pass, advanced reasoning pipelines leverage multi-agent systems (MAS) [4] and test-time search (TTS) algorithms [5], such as beam search and Monte Carlo Tree Search (MCTS) [6], to explore diverse solution trajectories. A critical component guiding this search is the Process Reward Model (PRM) [7, 8, 9] . By evaluating the correctness of intermediate reasoning steps, PRMs guide the search algorithm to allocate compute toward more promising partial solutions.  \nHowever, scaling test-time search exposes a critical computational bottleneck: the cost of the PRM itself. Existing PRMs are fundamentally text-based. For every candidate trajectory proposed by the generator, a text-PRM must encode the entire sequence of text tokens from scratch to output a reward. As self-attention scales quadratically with sequence length, this introduces an O (L2 ) computational cost per scoring call, where L is the sequence length. In modern MAS workflows, reasoning trajectories routinely span thousands or tens of thousands of tokens. When searching over dozens of candidates at multiple agent handoffs, the redundant text re-encoding by the PRM becomes an even heavier burden, often matching the FLOPs required for the generation itself. Such an architectural bottleneck limits the application of PRMs in long-context reasoning scenarios.  \nPreprint.  \nIn this work, we identify a missed opportunity in current test-time scaling systems: the generation process of agents inherently computes a rich, high-dimensional representation of the trajectory, the Key-Value (KV) cache. As autoregressive generation requires caching historical states to predict the next token, a complete record of the model’s intermediate representations across all layers and positions is produced naturally during the LLM’s generation phase. In Section 3, we further theoretically prove the representational advantage of KV cache and its efficient expl","cbCaibiwtNbcHcmD","https://ap.wps.com/l/cbCaibiwtNbcHcmD","pdf",588670,2,1,15,"English","en",105,"# Introduction\n## Process Reward Models and the Quadratic Bottleneck\n## Missed Opportunity: KV Cache From Generation\n# KV-PRM Method\n## KV-Cache Transfer With a Verify Token\n# Experiments\n## Benchmarks and Results\n# Theoretical Insights and Efficiency Analysis\n## Information Capacity of KV Cache","[{\"question\":\"为什么现有的文本式 PRM 在多智能体长推理中会成为瓶颈？\",\"answer\":\"文本式 PRM 需要对每个候选轨迹从头重编码完整文本，且自注意力计算随序列长度按 O(L^2) 增长。在长上下文、候选数量多、交接多智能体时，重复重编码会显著增加整体计算成本。\"},{\"question\":\"KV-PRM 具体如何减少评分阶段的计算量？\",\"answer\":\"KV-PRM 不再处理解码后的文本，而是通过 KV-cache transfer 利用生成过程中自然产生的 KV cache。它在序列末尾追加一个“verify token”，并基于基础模型的预存 KV cache 用轻量 LoRA 适配进行读取验证。评分复杂度从 O(L^2) 降到 O(L)。\"},{\"question\":\"实验结果在什么任务与指标上体现了 KV-PRM 的优势？\",\"answer\":\"KV-PRM 在 MATH、GSM8K、AIME 等基准上与文本式 PRM 相当甚至优于其表现，并报告了最多 5,000× 的评分 FLOPs 降幅、15–37× 的延迟加速，以及约 34.2× 的单序列内存占用降低。\"}]",1784178942,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"kv-prm-efficient-process-reward-modeling-via-kv-cache-transfer-for-multi-agent-test-time-scaling","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/kv-prm-efficient-process-reward-modeling-via-kv-cache-transfer-for-multi-agent-test-time-scaling/82221/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-20","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么现有的文本式 PRM 在多智能体长推理中会成为瓶颈？","Question",{"text":75,"@type":76},"文本式 PRM 需要对每个候选轨迹从头重编码完整文本，且自注意力计算随序列长度按 O(L^2) 增长。在长上下文、候选数量多、交接多智能体时，重复重编码会显著增加整体计算成本。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"KV-PRM 具体如何减少评分阶段的计算量？",{"text":80,"@type":76},"KV-PRM 不再处理解码后的文本，而是通过 KV-cache transfer 利用生成过程中自然产生的 KV cache。它在序列末尾追加一个“verify token”，并基于基础模型的预存 KV cache 用轻量 LoRA 适配进行读取验证。评分复杂度从 O(L^2) 降到 O(L)。",{"name":82,"@type":73,"acceptedAnswer":83},"实验结果在什么任务与指标上体现了 KV-PRM 的优势？",{"text":84,"@type":76},"KV-PRM 在 MATH、GSM8K、AIME 等基准上与文本式 PRM 相当甚至优于其表现，并报告了最多 5,000× 的评分 FLOPs 降幅、15–37× 的延迟加速，以及约 34.2× 的单序列内存占用降低。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]