[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81575-en":3,"doc-seo-81575-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},81575,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",8,"Research & Report","AugServe Adaptive Request Scheduling for Augmented Large Language Model Inference Serving","Augmented large language models (LLMs) that invoke external tools are increasingly used in inference serving, but external-call behavior creates major efficiency problems under strict Service-Level Objectives (SLOs). Existing serving systems ignore these dynamic execution patterns and rely on fixed batch token budgets, causing severe Head-of-Line blocking and reduced effective throughput (goodput). AugServe introduces state-aware request scheduling and dynamic batch token budgets to match heterogeneous requests and evolving execution states. Experiments show 6.5× and 4.7× higher effective throughput than vLLM and INFERCEPT.","AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving  \nYing Wang 1 Zhen Jin 1 Zhenqian Chen 1 Jiexiong Xu 1 Wenhai Lin 2 Yiquan Chen 2 Wenzhi Chen 1  \narXiv :2512 .040 13v 3 [ cs .CL] 10 Jul 2026  \nAbstract  \nAugmented large language models (LLMs) that invoke external calls are increasingly prevalent in inference serving. However, such augmentations pose significant challenges to inference efficiency under strict Service-Level Objectives (SLOs) . Existing inference systems are agnostic to the dynamic execution behaviors induced by external calls and rely on fixed batch-level token budget, which leads to severe Head-of-Line (HoL) blocking and substantially reduced effective throughput. We present AugServe, an efficient augmented LLM inference serving framework that mitigates request queuing latency and improves effective throughput under external-call-augmented workloads. AugServe integrates state-aware request scheduling with dynamic batch-level token budgets to adapt to heterogeneous requests and their dynamically changing execution states. Experimental results show that AugServe achieves 6.5 × and 4.7 × higher effective throughput than vLLMand INFERCEPT, respectively.  \n1. Introduction  \nAugmented Large Language Models (LLMs) have rapidly emerged as a promising paradigm (Abhyankar et al., 2024 ; Hao et al., 2023) for modern LLM inference serving. Compared with traditional text-only LLMs, which rely on fixed pretrained parameters and lack real-time knowledge (Schick et al., 2023 ; Gade et al., 2025), augmented LLMs extend their capabilities by invoking external tools (e.g., web APIs, databases, or specialized models) during inference (Chenet al., 2024b ; Go & Park, 2025 ; Qin et al., 2024a ; Lu et al., 2024) . This approach enables augmented LLMs to perform more complex tasks such as arithmetic computation (Chenet al., 2024a ; Yao & Yadav, 2025), real-time information  \n1College of Computer Science and Technology, Zhejiang University, Hangzhou, China 2Alibaba Group, Hangzhou, China. Correspondence to: Jiexiong Xu \u003C[jasonxu@zju.edu.cn](jasonxu@zju.edu.cn) >, Wenzhi Chen \u003C[chenwz@zju.edu.cn](chenwz@zju.edu.cn) >.  \nPreprint. July 13, 2026.  \nPrompt: \"Which team won the 2025 NBA Finals?\"  \nPrefill Decode  Paused  Resume Decode  \n(i)  \nAPI Call  \n\"winner of 2025 NBA Finals\"  \n(iii)  \nTime  \nFigure 1. Augmented LLM inference process.  \nretrieval (Su et al., 2024 ; Gade et al., 2025), and web interactions (Qi et al., 2025 ; Zhang et al., 2025) .  \nAugmented LLM inference service systems are becoming the key infrastructure for AI-centric cloud computing, with inference efficiency directly impacting user experience. Figure 1 illustrates the workflow of the augmented LLM inference service (Abhyankar et al., 2024 ; Gim et al., 2024): (i) During inference, the augmented LLM identifies the need for real-time information and triggers the corresponding tool calls. (ii) The inference process is paused while awaiting the response from the external augmentation module. (iii) Upon the response being returned, the serving system appends it to the sequence generated and resumes normal generation.  \nIdeally, inference systems must simultaneously deliver high throughput and low latency. In this context, Service-Level Objectives (SLOs) serve as strict latency boundaries (e.g., requiring Time-to-First-Token (TTFT) below a fixed threshold) (Gao et al., 2025 ; Zhong et al., 2024 ; Wu et al., 2023 ; Patel et al., 2025) . Accordingly, the system’s efficiency is best characterized by effective throughput (or goodput), defined as the volume of requests processed per unit time that successfully satisfy these SLO requirements (Wanget al., 2025 ; Karthik et al., 2024 ; Zhang et al., 2023) .  \nState-of-the-art inference systems focus on improving inference performance. vLLM (Kwon et al., 2023) has emerged as the de facto standard for efficient LLM serving. However, in augmented LLM inference, vLLM treats external calls as ","cbCaiuDCYIPVPghY","https://ap.wps.com/l/cbCaiuDCYIPVPghY","pdf",1561336,2,1,21,"English","en",105,"# Introduction\n## Augmented LLM inference workflow and SLOs\n## Existing systems (vLLM, INFERCEPT) and limitations\n## Challenges: HoL blocking and fixed batch token budgets","[{\"question\":\"What efficiency challenge does AugServe address in augmented LLM inference?\",\"answer\":\"AugServe targets efficiency degradation under strict SLOs caused by external-call-induced execution pauses, which existing systems mishandle in scheduling and batching.\"},{\"question\":\"How does AugServe improve effective throughput compared with vLLM and INFERCEPT?\",\"answer\":\"It uses state-aware request scheduling plus dynamic batch-level token budgets that adapt to heterogeneous requests and their changing execution states, reducing queuing latency and improving goodput.\"},{\"question\":\"Why do fixed batch-level token budgets hurt throughput when external calls are used?\",\"answer\":\"External calls pause requests whose contexts can occupy GPU memory in varying ways, so static token budgets cannot reflect dynamic memory availability, limiting concurrency and lowering throughput.\"}]",1784174412,53,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"augserve-adaptive-request-scheduling-for-augmented-large-language-model-inference-serving","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/augserve-adaptive-request-scheduling-for-augmented-large-language-model-inference-serving/81575/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What efficiency challenge does AugServe address in augmented LLM inference?","Question",{"text":75,"@type":76},"AugServe targets efficiency degradation under strict SLOs caused by external-call-induced execution pauses, which existing systems mishandle in scheduling and batching.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does AugServe improve effective throughput compared with vLLM and INFERCEPT?",{"text":80,"@type":76},"It uses state-aware request scheduling plus dynamic batch-level token budgets that adapt to heterogeneous requests and their changing execution states, reducing queuing latency and improving goodput.",{"name":82,"@type":73,"acceptedAnswer":83},"Why do fixed batch-level token budgets hurt throughput when external calls are used?",{"text":84,"@type":76},"External calls pause requests whose contexts can occupy GPU memory in varying ways, so static token budgets cannot reflect dynamic memory availability, limiting concurrency and lowering throughput.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]