[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83930-en":3,"doc-seo-83930-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83930,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Adaptive Inference Batching using Policy Gradients","Machine learning inference serving systems must balance high throughput and low latency under bursty, heterogeneous workloads. Static batching policies struggle to react to changing traffic, leading to inefficient GPU utilization and increased request waiting time. This work studies reinforcement learning for adaptive inference batching, using REINFORCE and PPO to learn dynamic batch size selection and request routing. A validated discrete-event simulator and extensive evaluation show that policy-gradient routing improves multi-GPU performance and reduces head-of-line blocking.","arXiv :2607 .05272v 1 [ cs .LG] 6 Jul 2026  \nAdaptive Inference Batching using Policy Gradients  \nRuslan Sharifullin  \nDepartment of Computer Science  \nStanford University  \n[rshar@stanford.edu](rshar@stanford.edu)  \nAbstract  \nMachine learning inference serving systems face the challenge of balancing high throughput with low latency, especially under bursty and heterogeneous workloads. Static batching policies often fail to adapt to dynamic traffic patterns. In this work, we explore Reinforcement Learning (RL) approaches, specifically REINFORCE and Proximal Policy Optimization (PPO), for adaptive inference batching. We develop a custom discrete-event simulator validated against standard queuing models and realworld traces (Azure Functions, BurstGPT) . Our extensive evaluation reveals that while simple static heuristics are surprisingly robust for standard single-GPU scenarios, RL provides significant value in multi-GPU environments. Specifically, we demonstrate that a Policy Gradient-based routing agent achieves a 3.5x performance improvement over Round-Robin scheduling by dynamically segregating heterogeneous workloads to minimize Head-of-Line blocking. This highlights the potential of RL not just for temporal batching, but for the joint optimization of request routing and batch composition in distributed inference systems.  \n1 Introduction  \nInference serving is a critical component of the machine learning lifecycle, bridging the gap between trained models and end-user applications. As Deep Learning models grow in size and complexity (e.g. Large Language Models), the computational cost of inference becomes a bottleneck. Batching requests improves throughput by amortizing fixed overheads (kernel launch, memory transfer) across multiple inputs. However, batching introduces a fundamental trade-off: larger batches increase throughput but also increase latency for individual requests waiting in the queue.  \nStatic batching policies, such as ”wait 10ms or until 32 requests accumulate,” are the industry standard (NVIDIA Corporation, 2023) . While simple to implement and robust in stable conditions, they are rigid and fail to adapt to dynamic traffic patterns (Crankshaw et al., 2017; Ali et al., 2020) . During periods of low traffic, a static timeout adds unnecessary latency; during high traffic bursts, a fixed batch size may underutilize the hardware if set too conservatively. Furthermore, in multi-model or multi-GPU environments, the complexity of scheduling increases exponentially. Heterogeneous requests (e.g., a mix of short ResNet inferences and long GPT gen-  \nerations) can cause Head-of-Line (HoL) blocking, where fast requests are stuck behind slow ones.  \nThis project addresses the question: Can reinforcement learning learn adaptive batching policies that outperform static heuristics? We formulate the batching problem as a Markov Decision Process (MDP) and apply Policy Gradient algorithms, specifically REINFORCE and PPO, to learn dynamic batch size selection and request routing. The agent observes the real-time system state (queue length, request types, and GPU availability) and learns a policy that balances competing objectives (throughput vs. latency) without manual tuning.  \n2 Related Work  \nThe problem of adaptive serving has been studied from both systems and learning perspectives.  \nHeuristic-based Systems: Clipper (Crankshaw et al., 2017) introduced a modular serving architecture that uses an additive-increasemultiplicative-decrease (AIMD) scheme to adjust batch sizes. While effective, AIMD is a reactive heuristic that oscillates and may not converge to the optimal policy for complex distributions. Triton Inference Server (NVIDIA Corporation,  \n2023) supports dynamic batching but relies on users to manually specify ”preferred batch sizes”and timeout windows, which requires extensive tuning for each new model and workload.  \nContinuous Batching: Recent advancements in LLM serving, such as Orca (Yu et al., 2022) and vL","cbCaikZ4uM1BY2jn","https://ap.wps.com/l/cbCaikZ4uM1BY2jn","pdf",1986897,3,1,5,"English","en",105,"# Introduction\n# Related Work\n# Environment (Dataset and Features)\n## Simulator Dynamics","[{\"question\":\"What problem does adaptive inference batching address?\",\"answer\":\"It addresses the throughput–latency trade-off in inference serving under bursty and heterogeneous request patterns, where static batching can become rigid and inefficient.\"},{\"question\":\"How is the batching task formulated for reinforcement learning?\",\"answer\":\"The work formulates batching as a Markov Decision Process and applies policy-gradient methods to learn batch size selection and request routing from observed system state.\"},{\"question\":\"What role does the policy-gradient routing agent play in performance gains?\",\"answer\":\"The routing agent dynamically segregates heterogeneous workloads to reduce head-of-line blocking, achieving up to a 3.5x performance improvement over round-robin scheduling in multi-GPU settings.\"}]",1784191515,13,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"adaptive-inference-batching-using-policy-gradients","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/adaptive-inference-batching-using-policy-gradients/83930/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does adaptive inference batching address?","Question",{"text":75,"@type":76},"It addresses the throughput–latency trade-off in inference serving under bursty and heterogeneous request patterns, where static batching can become rigid and inefficient.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is the batching task formulated for reinforcement learning?",{"text":80,"@type":76},"The work formulates batching as a Markov Decision Process and applies policy-gradient methods to learn batch size selection and request routing from observed system state.",{"name":82,"@type":73,"acceptedAnswer":83},"What role does the policy-gradient routing agent play in performance gains?",{"text":84,"@type":76},"The routing agent dynamically segregates heterogeneous workloads to reduce head-of-line blocking, achieving up to a 3.5x performance improvement over round-robin scheduling in multi-GPU settings.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":22,"slug":137},19,"General","general"]