SMETRIC Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

Jul 16, 2026

LLM scheduling is critical to serving, yet existing designs are unclear for agentic serving where requests originate from agents rather than humans. Agentic workloads change priorities: cluster TPS becomes primary, while per-token latency (TTFT/TPOT) can be relaxed but not dropped, and KV$ reuse dominates with over 80% reuse in a BAILIAN production trace. This study analyzes two real-world agent traces and shows KV$-aware routing can overload few instances, capping TPS. SMETRIC introduces balanced session-centric scheduling to preserve load balance and local reuse while limiting global-tier demand, improving cluster TPS and prefill throughput.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Abstract

Introduction

SMETRIC: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling Jiahao Wang 1 , Kaizhan Lin†1 , Kaixi Zhang 1 , Jinbo Han 1 , Xingda Wei 􀀌1 , Sijie Shen2 , Chenguang Fang2 , Wenyuan Yu2 , Rong Chen 1 , Haibo Chen 1 1Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University 2Alibaba Group arXiv :2607 .08565v 1 [ cs .DC] 9 Jul 2026 Abstract LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving—with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster’s tokens per second (TPS) the primary goal and relaxing—not eliminating—per-token latency requirements; and (2) requests share much of their KV$—reuse exceeds 80 % of request tokens in a production trace from BAILIAN, versus 54–62 % in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV$ reuse, thanks to the global-tier KV$ store and (2) by utilizing the workload’s intra-session locality, balancing a small fraction of requests—the first request in each agent session—suffices to balance the cluster without sacrificing most KV$ reuse on local instances. SMETRIC realizes these insights with balanced sessioncentric scheduling: it routes each session’s first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10–16 % under prefill-decode colocation with a global store and prefill TPS by 2–34 % under disaggregation over state-of-the-art schedulers, also with a better per-token latency. 1 Introduction Large language models (LLMs) now power agents like Claude Code [6] and OpenClaw [36], which accomplish complex tasks such as writing [5], debugging [14], and tuning the †Kaizhan Lin is affiliated with ShanghaiTech University; this work was done while he was an intern at Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University. 􀀌Xingda Wei is the corresponding author ([wxdwfc@sjtu.edu.cn](wxdwfc@sjtu.edu.cn)) . performance [39, 32] of the code that drives the computing world. An agent solves a task through a session of LLM requests, and generated tokens drive the agent toward completing its task. Therefore, how efficiently a model provider serves these tokens determines both the speed and the cost of running agents. To that end, existing providers deploy clusters dedicated to serving requests from many users’ sessions, each cluster running many instances, each able to serve a complete request. We term this cluster-scale, agent-dedicated LLM serving agentic serving in this paper. Note that agentic serving is still LLM serving: the provider serves each request through the standard LLM API and neither runs the agents nor sees their internal logic. What changes is the workload, because the requests now come from agents. Request scheduling—how the cluster routes each request to an instance—is a key pillar of LLM serving. Though wellstudied recently [54, 59, 42, 53, 17, 2], we argue that current scheduling designs must be revisited for agentic serving, for two reasons: First, agentic serving requires a different performance objective: the cluster’s tokens per second (TPS) becomes the primary goal, while the per-token latencies targeted by prior work [54, 59, 58]—time-to-first-token (TTFT) and time-peroutput-toke

FAQs

Why must LLM request scheduling be revisited for agentic serving?

Agentic serving changes the workload objective: cluster TPS becomes the primary goal, and latency constraints are only relaxed (not removed). It also shifts efficiency toward dominant KV$ reuse patterns unlike chat.

What problem do existing KV$-centric schedulers have in agentic serving?

They overly prioritize routing requests to instances that already cache the KV$, overloading a few instances while leaving others idle, which caps overall TPS.

How does SMETRIC achieve balanced session-centric scheduling?

SMETRIC routes each agent session’s first request primarily for load balance, then schedules follow-up requests in a cache-aware manner to preserve local reuse while keeping pressure on the global KV$ tier low.

Jul 16, 2026

Upload by Valentina