LLVM-Bench Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

Jul 16, 2026

LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. While large language models (LLMs) have shown strong performance in software issue resolution, their effectiveness on complex system-level LLVM compiler issues is insufficiently studied. This work introduces LLVMBench, a large-scale benchmark with 423 validated LLVM tasks, and LLVM-Gym, an automated evaluation platform. Experiments across multiple LLMs, retrieval settings, and agents reveal dominant failure modes and motivate a lightweight ensemble, LLVM-Ens.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Motivation: challenges in LLVM issue resolution

Limits of existing benchmarks

Contributions: LLVMBench and LLVM-Gym

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution Zhao Tian School of Computer Software, Tianjin University, Tianjin, China [tianzhao@tju.edu.cn](tianzhao@tju.edu.cn) Yingquan Zhao School of Cybersecurity, Tianjin University, Tianjin, China [zhaoyingquan@tju.edu.cn](zhaoyingquan@tju.edu.cn) Chenyao Suo School of Computer Software, Tianjin University, Tianjin, China [chenyaosuo@tju.edu.cn](chenyaosuo@tju.edu.cn) arXiv :2607 .00700v 1 [ cs . SE] 1 Jul 2026 Meng Wang School of Computer Science, University of Bristol, Bristol, UK [meng.wang@bristol.ac.uk](meng.wang@bristol.ac.uk) Junjie Chen* School of Computer Software, Tianjin University, Tianjin, China [junjiechen@tju.edu.cn](junjiechen@tju.edu.cn) Abstract—LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVMBench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVMBench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution. I. INTRODUCTION LLVM compiler infrastructure is one of the most influential and widely adopted software systems in modern computing [1]–[3] . It provides a collection of modular and reusable compiler and toolchain technologies that support a broad range of programming languages [4]–[7] . Owing to its flexibility, extensibility, and high performance, LLVM has become a foundational component in both industrial and academic compiler and program analysis projects [8]–[13] . As LLVM continues to evolve, however, developers must continuously address newly discovered bugs and feature requests. To date, LLVM community has accumulated nearly 100,000 issues on GitHub, *Junjie Chen is the corresponding author. of which approximately 30,000 remain unresolved [14] . Resolving these issues often requires understanding a largescale and highly complex codebase, implementing non-trivial code changes, and preserving existing functionality while addressing the target issue [15]–[17] . Consequently, LLVM issue resolution remains labor-intensive and costly, motivating the development of automated issue resolution techniques to improve both developer productivity and software quality. Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in software issue resolution [18]–[20] . Building upon the planning, reasoning, and tool-use abilities of LLMs, coding agents can autonomously navigate software repositories, execute commands, and generate patches for real-world issues. These capabilities have led to substantial progress on issue resolution benchmarks, most notably SWE-bench [15], a widely used benchmark that evaluates the ability of LLMs to resolve application-level software issues. Specifically, LLMs are provided only with an issue description and the corresponding codeb

FAQs

What problem does LLVM-Bench address?

LLVM-Bench targets the gap in evaluating LLM-based techniques on complex, system-level LLVM compiler issue resolution, which existing benchmarks do not adequately cover.

What are the main components introduced in the paper?

The paper introduces LLVMBench, a benchmark with 423 validated LLVM tasks, and LLVM-Gym, a platform that automates issue reproduction, patch application, compiler building, and test execution.

What failure modes are reported as dominant on LLVM-Bench?

Experiments indicate patch invalidity and build failures are the dominant failure modes for current LLM-based issue resolution approaches.

Jul 16, 2026

Upload by Olivia Brown