PTXBench - Benchmark and Adapt LLMs for GPU Kernel Optimization - with Architecture-specific PTX

Sep 15, 2026

Uploaded by Hazel

PTXBench is introduced as a benchmark that evaluates and adapts large language models (LLMs) to generate architecture-specific PTX for GPU kernel optimization. It measures functional correctness, whether required target instruction families execute at runtime, and peak/typical speedup against frontier libraries on GEMM and attention workloads across H100 and B200 GPUs. Results show uneven capability, especially reduced success on complex attention backward tasks, and target-instruction execution does not guarantee competitive performance. The paper further applies repair-conditioned supervised fine-tuning to Qwen3.6-27B, improving several tasks while revealing sensitivity to data coverage, balance, and the reasoning teacher.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Introduction

PTX and the challenge of architecture-specific optimization

PTXBench benchmark design and evaluation metrics

Contributions and adaptation study overview

arXiv : 2608 . 17379v2 [cs .CL] 19 Aug 2026 PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX Genghan Zhang1 ∗ Yixin Dong2 Chengze Fan4 Zhichen Zeng3 Yueming Yuan3 Shaowei Zhu4 Kunle Olukotun1 1 Stanford University 2Carnegie Mellon University 3RadixArk 4Independent Researcher ABSTRACT We introduce PTXBench, a benchmark for evaluating and adapting large language models (LLMs) to use architecture-specific PTX for GPU kernel optimization. PTXBench measures functional correctness, whether selected target instructions execute at runtime, and speedup over frontier libraries across GEMM and attention workloads on H100 and B200 GPUs. Our evaluation shows that architecture-specific PTX capability remains uneven: success rates fall substantially on complex attention backward workloads, and executing the target instructions does not necessarily translate into competitive performance. No evaluated model consistently matches frontier libraries across the suite. We further adapt Qwen3.6-27B using supervised fine-tuning. Repair-conditioned training improves several tasks, but generalization remains uneven; data coverage, balance, and the quality of the reasoning teacher matter in addition to dataset size. PTXBench provides an auditable testbed for measuring and improving LLMs’ ability to exploit evolving GPU architectures. 1 INTRODUCTION PTX (Parallel Thread Execution) is the lowest-level programmable interface that CUDA developers can explicitly control. High-performance kernels often require PTX-level optimization to efficiently leverage architecture-specific mechanisms [1] . Recent NVIDIA GPU generations have been introducing new architecture-specific PTX instructions for tensor cores, memory units, and synchronization mechanisms [2–6] . A portable CUDA kernel can remain functionally correct while leaving the defining capabilities of newer hardware unused. Higher-level kernel languages aim to recover performance portability [7, 8], which ultimately lower to the same PTX interface. However, keeping these kernel languages aligned with rapidly evolving hardware requires continued compiler engineering [9], while validating an optimizing compiler entails a much larger input and configuration space and thus is much harder than validating an individual kernel [10] . This raises a practical question for kernel developers: can an LLM directly exploit architecture-specific features, and can targeted post-training improve this ability? To answer this question, we introduce PTXBench, an architecture-specific GPU kernel benchmark that evaluates whether LLMs can exploit specified low-level GPU mechanisms. PTXBench incorporates MiniPTXAgent, a multiturn agent loop in which models are given an accurate architecture-specific knowledge pack and generate CUDA kernels with inline PTX (which we call CUDA–PTX) and revise them using structured execution feedback. Across each trajectory, PTXBench separately measures functional correctness, whether the required instruction family executes at runtime under the evaluated workload, and peak and typical performance. PTXBench is built on FlashInfer-Trace [11], a unified schema for kernel workloads, solutions, and evaluations. Separating the capability probe from the problem collection makes PTXBench extensible to suites with compatible interfaces, such as SOL-ExecBench [12] . Existing GPU kernel benchmarks, beginning with KernelBench [13], ask models to replace reference PyTorch operators with functionally correct, faster GPU kernels [14, 15] . These end-to-end outcomes are essential, but they do not isolate whether a model can directly program a specified architecture mechanism: performance may instead come from generic CUDA code or calls to existing vendor libraries. PTXBench complements these suites with a controlled capability probe: models must directly produce efficient low-level kernels using a specified family of architecture-specific PTX instructions

FAQs

What does PTXBench measure when evaluating LLMs?

PTXBench measures functional correctness, whether required architecture-specific PTX instruction families execute at runtime, and peak/typical performance relative to frontier libraries.

What workload patterns does PTXBench reveal about current models?

Models often succeed on forward workloads but struggle substantially with complex attention backward workloads. Even when target instructions execute, performance may still lag behind frontier libraries.

How is Qwen3.6-27B adapted in the study?

The study applies supervised fine-tuning conditioned on repair signals (repair-conditioned training) to Qwen3.6-27B, improving several tasks while showing uneven generalization.

Sep 15, 2026

Uploaded by Hazel