[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82733-en":3,"doc-seo-82733-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82733,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","Spork Self-Speculative Forking to Accelerate Agentic LLM Inference","LLM agents commonly follow a Thought–Action–Observation loop, but the action step is often handled serially, forcing the model to idle while waiting for external tool results. In many workloads, tool execution consumes 16–37% of wall time, with prior reports reaching 35–61%. Spork introduces self-sPeculative fORKing: a training-free controller that forks a probe early to predict the upcoming tool name and dispatches the speculated tool call in parallel with remaining decoding, then safely falls back on rejection. A cost model and components such as prefix-cache forking, confidence gating, and partial-token acceptance optimize break-even and keep correctness within 1pp of baseline, improving GAIA performance by 18%.","SPORK: Self-Speculative Forking to Accelerate  \nAgentic LLM Inference  \nHuajun Bai∗ Tsinghua University  \nYouyou Lu  \nTsinghua University  \nWeiwei Lv Huichuan Zheng  \nMeituan Tsinghua University  \nJiwu Shu† Tsinghua University  \narXiv :2607 .03333v 1 [ cs .DC] 3 Jul 2026  \nAbstract  \nLLM agents are becoming a common interface for research, coding, and question answering, yet their Thought–Action– Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16–37% of wall time in our workloads and 35–61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B’s upcoming tool name with 74.6–99.6% accuracy across five benchmarks. We present Spork (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-ofthought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, Spork falls back to serial execution with no correctness penalty. On real-tool benchmarks, Spork cuts Qwen3-32B’s GAIA 􀀥95 by 18% (131.9→108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. Spork deploysas a thin controller over standard completion APIs (noretraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. Spork is open source at [https://github.com/baihuajun24/spork](https://github.com/baihuajun24/spork).  \nKeywords: LLM agents, tool use, speculative execution, inference acceleration  \n1 Introduction  \n“Treat things before they exist. Regulate things before disorder begins.” —Lao Tzu  \nLLM-powered agents are becoming a common interface for complex work such as deep research, coding, and  \n∗ [Contact: baihuajun24@gmail.com](Contact: baihuajun24@gmail.com)  \n†Corresponding author.  \nQ: How many studio albums did Mercedes Sosa publish 2000–2009?  \n| name:\u003Cbr>args:\u003Cbr>\"search\"\u003Cbr>\"Sosa albums\u003Cbr>2000-2009\" |\n| --- |\n|  ✓ correct prediction |\n| name: \"search\"\u003Cbr>args: \"Sosa albums 2000-2009\" |\n\nFigure 1. Tool intent is visible early. A speculative thread, forked at the main thread’s first token with a forced tool-call prefix, predicts the exact tool call that the main thread emits ∼2,000 CoT tokens later.  \ntool-assisted question answering [3, 9, 20, 25, 29, 39] . In these systems, the LLM acts as the controller: it reasons over the user request, emits a structured tool call, waits for the external tool to finish, and then resumes generation using the tool result. This Thought–Action–Observation loop [39] is simple and safe, but it is also often serial: each tool result conditions all subsequent tokens, so the serving system cannot resume generation until the result arrives. While the tool is running, the GPU often has no useful work to do for that agent turn.  \nThis tool wait is hard to engineer away because it is structurally bounded by network round-trips, container cold-starts, and database I/O that resist the exponential improvements seen in model serving. It is also large. Our profiling across three benchmarks in Figure 2 shows that tool execution accounts for 16–37% of end-to-end wall time, and on turns with real network tools the GPU is idle for the majority of the turn; Sui et al. [32] report even higher fractions, 35–61% of total agent latency across agent workloads","cbCaik4s0lz0n5wZ","https://ap.wps.com/l/cbCaik4s0lz0n5wZ","pdf",1740013,6,1,16,"English","en",105,"# Abstract\n# Introduction\n## Background: Thought–Action–Observation and tool wait\n## Motivation: tool latency as the bottleneck\n## Related work and deployment gap\n## Paper contribution overview","[{\"question\":\"What problem does Spork address in agentic LLM inference?\",\"answer\":\"Spork targets the serial nature of the Thought–Action–Observation loop, where the GPU often idles during external tool execution, consuming a significant portion of wall time.\"},{\"question\":\"How does Spork predict and dispatch tool calls without training?\",\"answer\":\"Spork forks a probe at the start of generation to predict the upcoming tool name, then dispatches the speculative tool call early while the main thread continues decoding.\"},{\"question\":\"What happens when Spork’s speculative tool call prediction is wrong?\",\"answer\":\"If speculation is rejected, Spork falls back to serial execution, avoiding correctness penalties and preserving baseline accuracy behavior.\"}]",1784182569,40,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"spork-self-speculative-forking-to-accelerate-agentic-llm-inference","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/spork-self-speculative-forking-to-accelerate-agentic-llm-inference/82733/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does Spork address in agentic LLM inference?","Question",{"text":76,"@type":77},"Spork targets the serial nature of the Thought–Action–Observation loop, where the GPU often idles during external tool execution, consuming a significant portion of wall time.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does Spork predict and dispatch tool calls without training?",{"text":81,"@type":77},"Spork forks a probe at the start of generation to predict the upcoming tool name, then dispatches the speculative tool call early while the main thread continues decoding.",{"name":83,"@type":74,"acceptedAnswer":84},"What happens when Spork’s speculative tool call prediction is wrong?",{"text":85,"@type":77},"If speculation is rejected, Spork falls back to serial execution, avoiding correctness penalties and preserving baseline accuracy behavior.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":29,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":107,"slug":137},19,"General","general"]