Trusted Floors Under Untrusted Learners A Runtime Assured-SLO Guard for ML Serving

Jul 16, 2026

Upload by Aria

Modern ML serving lets learned, unbounded components such as routers and latency-SLO admitters decide a tenant’s QoS at runtime. When one component is incorrect, the promised assured SLO can silently fail across Kubernetes layers, creating cross-layer surprises. The work replaces trust in the learner with a small trusted guard that wraps it, enforcing request-level safety projections via reservation and priority dispatch, while conservatively screening aggregate percentile obligations. The guard is evaluated on real 2×V100 systems and simulators, achieving assured-class miss 0.0 under miscalibrated admitters and contrasting baseline failures.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Composition without assurance

Two obligations, two epistemics

Guarded composition for assured QoS

Trusted Floors Under Untrusted Learners: A Runtime Assured-SLO Guard for ML Serving Hsiu-Chi Tsai National Yang Ming Chiao Tung University [hctsai1006@cs.nctu.edu.tw](hctsai1006@cs.nctu.edu.tw) NSDI ’27 Frontiers Track arXiv :2607 .09992v1 [ cs .DC] 10 Jul 2026 Abstract Modern ML serving increasingly lets learned, unbounded components (routers, latency-SLO admitters, admit ladders) decide a tenant’s quality of service; when one is wrong, the assured SLO can silently break, and the Kubernetes layers beneath (Kueue, DRA, the Gateway-API Inference Extension, GAIE) only add cross-layer surprises. Rather than trust the learner to be right, we bound the damage a wrong one can do: a small trusted guard wraps the untrusted learner—learned proposes, the guard disposes. We observe that a tenant’s assured-SLO obligation splits into two parts with different epistemics. Its safety projection (per-request dispatch feasibility and a per-class, per-window service floor) is a controllable obligation a small guard enforces at runtime. The guard holds it regardless of an arbitrarily-wrong learned admitter, by reservation and priority dispatch, with dropping infeasible work a safety net. Its guarantees are conditional, scoped by strength: the admission floor and drop rules hold by construction, the service floor modulo backend capacity, the worst-case sojourn only in a characterized envelope under the stated assumptions (§2) . Its aggregate obligation (a taillatency percentile over the workload ensemble) is a statistical residual with no per-request enforcement point; it can only be conservatively screened, and a cheap deterministic (􀀿50) screen is optimistically unsound not only near saturation (􀁤 → 1) but, we measure, well below it. We build the guard (a Simplex-style assured-floor gate plus assured-first priority dispatch) and show on real 2×V100 that it holds assured-class miss 0.0 (admitted-basis; reps 10, worst upper Wilson CI 0 .0053) across every miscalibration of a learned admitter under two shed policies that fail in opposite directions (overload, and rejecting assured), with priority dispatch shown load-bearing. Against the strongest 2026 baseline (a real, deployed GAIE Flow Control on a serving simulator), amislabelling router flips the same assured requests from miss 0.0 to 1.0; our guard instead reserves by the true class, so the label cannot break its floor. We characterize when a cheap config screen can be trusted (in overload away from the knee, where a 􀀿50 accept is already sound; a conservative percentile extends this) and when it cannot (near saturation, the tightest deadlines, and, for a 􀀿 50 screen, even below saturation) . Assurance is thus a property of the guard’s reservation+priority floor (given its preconditions), not of the learned component, which is hard to evaluate traditionally since one cannot enumerate all wrong learners. As a Frontiers submission we evaluate the stance on commodity 2×V100 anda serving simulator, and scope datacenter scale, real-model Flow Control, and a closed worst-case theorem as the agenda. 1 Composition without assurance A tenant’s quality of service is no longer set in one place. Modern ML-serving stacks increasingly let learned, unbounded components (smart routers, latency-SLO admitters, admit ladders) decide it at runtime. That QoS is itself composed across four independently-configured layers: cluster quota and priority (Kueue’s ClusterQueue/WorkloadPriorityClass), GPU device claims (DRA, [resource.k8s.io](resource.k8s.io)), inferencegateway priority and (experimental) Flow-Control admission (GAIE’s InferenceObjective), and runtime scheduler knobs. When one component is wrong, the tenant’s assured floor can silently collapse. Three facts make this dangerous.(i) Cross-layer surprises are real: we catalogue eight hazards where the composed semantics surprise the operator (e.g. a DRA alias double-counts a device; a Kueue quotaCheckStrategy flag alone flips admission of th

FAQs

What problem does the paper address in modern ML serving?

Learned, unbounded components in the serving stack can make runtime decisions that cause an assured SLO to break silently, especially when cross-layer interactions occur. Existing tooling cannot reliably validate composed behavior across layers.

How does the proposed runtime guard work?

A trusted guard wraps the untrusted learned component: the learned part proposes, and the guard disposes. It enforces the safety part of the assured SLO using reservation and priority dispatch, and drops work that is infeasible to meet deadlines.

What limitations exist for enforcing the aggregate part of an assured SLO?

The aggregate obligation (e.g., a tail latency percentile over the workload ensemble) cannot be enforced at a per-request point with a finite guarantee. The guard can only conservatively screen it, and the paper characterizes when a cheap configuration screen remains trustworthy versus when it fails near saturation and under tight deadlines.

Jul 16, 2026

Upload by Aria