Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs

Jul 16, 2026

Reported serving speedups from quantized kernels often combine weight format, the kernel, and the inference runtime into one undifferentiated result. This study performs an attribution analysis on four NVIDIA RTX A5000 GPUs (24 GiB each) connected via NVLink-paired hosts, using a matched FP16 intermediate to keep the faster runtime while dropping the quantized kernel. Under matched greedy decoding, the end-to-end stack reaches 2.58×, with runtime changes contributing about two thirds of the gain on a log scale; the kernel+quantization contribution varies by at most 1.5% across related model families.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

Introduction

Attribution via matched intermediate

Tensor parallel sharding results

Multi-instance routing and workload/model dependence

arXiv :2607 . 11368v1 [ cs .DC] 13 Jul 2026 Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs Weijia Han Lisha Qu University of Washington Abstract Reported serving speedups from quantized kernels typically bundle the weight format, the kernel, and the inference runtime into one number. We present an attribution study on four NVIDIA RTX A5000 GPUs, 24 GiB each, on a single host with NVLink-bridged pairs. A matched intermediate stack that keeps the faster runtime without the quantized kernel splits the full speedup into a runtime part and a kernel and quantization part. Under matched greedy decoding the full stack reaches 2. 58× end to end, with the runtime change accounting for about two thirds of that gain on a logarithmic scale; across three similar model families the kernel and quantization part moves by at most 1.5% . Sharding one instance across all four cards falls well below doubling: a profiler trace attributes about 80% of the per token shortfall to coordination, and an NVLink versus PCIe control on the same hardware shows similar realized bandwidth on both links, pointing away from link bandwidth as the cause. Whether to run one sharded instance or several independent ones depends on the workload and the model, with the ranking reversing on the larger model: the smaller model splits between sharding and multiple instances by workload, while the larger model favors two paired instances onevery workload. Quantization extends sustainable concurrent users roughly four times past a reproducible half precision memory cliff. Differences in sampling mode and prompt pool between the two stacks are documented as threats to validity. 1 Introduction The deployment of large language models in production has motivated a cascade of systems optimizations: weight quantization to reduce memory footprint, paged attention to increase key value cache utilization, continuous batching to keep accelerators saturated, and tensor parallel sharding to spread computation across multiple accelerators. Each technique is typically evaluated as a complete stack replacement. A common framing is: “switch from HF Transformers to vLLM-Marlin and throughput grows ten times.” This conflates weight format, GEMM kernel, and inference runtime at once. These conflations make it hard to answer practical questions: which part of the stack is doing the work, and whether tensor parallel sharding still helps once a model fits on a single accelerator. We present an attribution study for the eight billion parameter Llama model on four NVIDIA RTX A5000 cards, bridged into pairs and drawn from a single host, a mid range configuration common in academic and industrial clusters. By introducing a vLLM-FP16 intermediate, we obtain a multiplicative decomposition of the total speedup into a runtime factor and a kernel and quantization factor. Their product equals the overall speedup exactly by construction; the matched intermediate cancels algebraically. Figure 1 previews the central result. We use the decomposition as a bookkeeping device, and make no causal independence claim (Section 3.9) . Our four main contributions are: 1. Attribution decomposition through a matched intermediate. Matched greedy decoding yields a 2.58× end to end speedup at the wide batch operating point. The runtime swap accounts for roughly Four A5000 GPUs Matched control (n = 3) Takeaway Figure 1: Overview of the study. Four A5000 GPUs bridged into NVLink pairs serve the same model through three stacks. A production swap measured in a single run reports one conflated 10. 62× end to end speedup, which bundles the runtime change, the kernel and quantization change, and differences in batching and sampling. Inserting an intermediate stack that keeps the faster runtime and drops the quantized kernel, under matched greedy decoding with three runs, splits the gain into a

FAQs

How does the matched FP16 intermediate enable separating runtime from kernel and quantization effects?

The study introduces a vLLM-FP16 intermediate stack that preserves the faster inference runtime while removing the quantized kernel split. This matched design allows the overall speedup to be decomposed into a runtime factor and a kernel+quantization factor by construction.

What speedup is reported under matched greedy decoding, and how is it distributed?

With matched greedy decoding at the wide batch operating point, the full stack achieves 2.58× end-to-end speedup. The runtime component accounts for roughly two thirds of the gain on a logarithmic scale, while the kernel and quantization component contributes the remaining portion.

Why does the paper conclude that tensor parallel sharding does not effectively stack on this hardware?

When sharding an instance across all four cards, throughput improves well below the practical doubling threshold in measured cells. Profiling attributes about 80% of the per-token shortfall to coordinating the shards, and an NVLink-versus-PCIe control shows similar realized bandwidth, pointing to synchronization/launch overhead rather than link bandwidth as the bottleneck.

Jul 16, 2026

Upload by Genevieve