OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters

Jul 16, 2026

Serving large language models on shared heterogeneous GPU clusters requires selecting GPU type, tensor-parallel degree, and numerical precision before consuming valuable node-hours. These choices are difficult because throughput, launch success, and demand fluctuate, while static recipes miss interactions such as quantization-dependent performance, KV-cache size-by-precision trade-offs, and >2× failure-rate differences across TP degrees. OmniPilot predicts feasible serving costs and abstains outside its measured support envelope. Using a conformally calibrated quantile cost model with OOD abstention, it achieves low prediction error and 95% top-1 accuracy on benchmarks, while correctly flagging unsupported scenarios.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Abstract

1. Introduction

1.1 First-attempt failures on shared clusters are expensive

1.2 The gap

arXiv :2607 .0 1579v 1 [ cs .DC] 2 Jul 2026 OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters D. Balamurugan∗1 and Thomas W. Bush 1 1 Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University July 3, 2026 Abstract Serving large language models (LLMs) on a shared, heterogeneous GPU cluster requires users and operators to select the GPU type, tensor-parallel degree, and precision before committing valuable node-hours. Making these choices is challenging because effective throughput, launch-success rates, and cluster demand and utilization continuously fluctuate. Furthermore, static configuration recipes miss critical interactions: quantization effects depend heavily on the model family, key-value cache pressure creates size-by-precision trade-offs, and failure rates vary by more than twofold across different tensorparallel degrees. Additionally, cluster resources are frequently constrained by unpredictable hardware failures. To address these challenges, we present OmniPilot, a launch advisor that predicts serving costs for feasible configurations and abstains when requests fall outside its measured support envelope. OmniPilot pairs a conformally calibrated quantile cost model (spanning eight serving targets) with anout-of-distribution (OOD) abstention layer. It ranks configurations using an economic utility metric calibrated to an operator’s revealed preferences. In evaluations across 460 benchmark runs on A100, H100, and H200 hardware across four precisions, OmniPilot predicts aggregate throughput with a 6.2% mean absolute percentage error (MAPE) and a log-space R2 = 0 .92. Compared to operator-relevant, model-free baselines, the advisor achieves 95% top-1 accuracy with a mean utility regret of just 0 .003. When tested on an OOD holdout of unsupported cells, prediction error climbs to 24–46% and conformal intervals cover 0 of 5 points; however, the abstention layer successfully flags all five as low-confidence. Over time, these OOD scenarios will be integrated into the training dataset to continuously expand the advisor’s support envelope. Training and recovery workloads remain out of scope for this work, but are slated for future research. Keywords: LLM inference serving, GPU clusters, cost modeling, conformal prediction, abstention, uncertainty quantification, quantization, MLOps. 1. Introduction 1.1 First-attempt failures on shared clusters are expensive Shared GPU clusters are highly contended, multi-tenant, and heterogeneous environments. Before an inference server can deliver its first token, an operator 1 must select the underlying hardware configuration: the specific GPU type, the tensor-parallel (TP) degree, and the numerical precision. Making these decisions blindly is costly. Out of our 561 inference benchmark runs, 80 runs (14%) ended in immediate failure due to hard hardware incompatibilities or out-of-memory resource exhaustion. Static, one-size-fits-all deployment recipes are brittle in this complex decision space. Intricate hardwaresoftware interactions create highly counterintuitive behaviors: a 4-bit quantized weight format can actually run slower than a 16-bit format on a given runtime, key-value (KV) cache pressure introduces dynamic size-byprecision trade-offs, and multi-GPU launch success rates degrade sharply as the TP degree increases. Because ∗ Corresponding author: [bala_desinghu@harvard.edu](bala_desinghu@harvard.edu) 1 Hereafter, we use the term operator as a convenient shorthand to represent any entity submitting a workload, including individual end-users, system administrators, or delegated AI agents managing MLOps pipelines. these low-level runtime nuances are entirely invisible to traditional cluster orchestration, operators require a pre-submission advisor. This system must provide performance predictions with calibrated uncertainty, alongside an explicit abstention signal to block unsupported or high

FAQs

What problem does OmniPilot address when serving LLMs on heterogeneous GPU clusters?

Operators must choose GPU type, tensor-parallel degree, and precision before committing node-hours. Static recipes are brittle due to fluctuating throughput, launch success rates, and hardware interactions like quantization and KV-cache trade-offs.

How does OmniPilot decide which configurations to recommend or reject?

OmniPilot uses a conformally calibrated quantile cost model to predict costs for feasible configurations and an out-of-distribution abstention layer to refuse requests outside its measured support envelope.

How accurate is OmniPilot’s prediction and how does it handle unsupported OOD cases?

Across 460 benchmark runs it predicts aggregate throughput with 6.2% MAPE and log-space R2 of 0.92, achieving 95% top-1 accuracy. On an OOD holdout, prediction error increases to 24–46%, but conformal intervals flag low confidence and the abstention layer identifies all unsupported points.

Jul 16, 2026

Upload by Olivia Brown