Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions

Jul 16, 2026

Upload by Ivy

Large language models can generate convincing quantum programming code, yet their ability to target the specific SDK version requested by users remains uncertain. The study frames this as API drift and introduces quantum-api-drift, a benchmark capturing version fidelity via execution success on the requested SDK, cross-version compatibility, observed failure modes, and documentation-guided repairs. Instantiated on Qiskit across v0.43, v1.3, and v2.0, evaluations measure 17 models over 50 tasks with extensive sampling.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Abstract

1 Introduction

arXiv :2607 .04072v 1 [ cs . SE] 5 Jul 2026 Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions Mohammad Arif Rasyidi and Syahirul Faiz Department of Computer Science Khalifa University Abu Dhabi, United Arab Emirates {100066916, [100065736}@ku.ac.ae](100065736}@ku.ac.ae) Abstract Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user. We study this problem as API drift and introduce quantum-api-drift, a benchmark for measuring version fidelity, defined here as execution success on the requested SDK version, cross-version compatibility, failure modes, and documentation-guided repair in LLMgenerated quantum SDK code. We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0 . We evaluate 17 models on 50 tasks with 3 samples per prompt, yielding 450 generated samples and 1,350 executions per model. Sixteen models are tested in a matched REST API setting with a 1024-token output cap, while GPT-5.4 (Codex CLI) is reported separately as a reference configuration. Across the 16 matched REST models, diagonal Pass@1 ranges from 0.02 to 0.85 . Claude Opus 4.7 is strongest on v0.43 and v2.0, while Grok 4.20 is strongest on v1.3 at 0.513 . Error profiles differ systematically by model strength: weaker models fail mainly with broken imports, while stronger models more often reach deprecation-level failures. Documentation-guided repair succeeds for 0.19 to 0.59 of repair attempts overall and is consistently much more effective for migration to v2.0 than to v1.3 . The benchmark artifacts are publicly available at [https://github.com/arasyi/quantum-api-drift](https://github.com/arasyi/quantum-api-drift. These)[. These](https://github.com/arasyi/quantum-api-drift. These) results show that version alignment is a distinct evaluation axis for quantum code generation and that API drift remains only partly recoverable even with migration guidance. 1 Introduction Large language models are increasingly used as programming assistants for scientific software, including quantum software development kits (SDKs) . In practice, however, users rarely ask for code against an abstract SDK. They ask for code that must run against a specific installed version. This creates a failure mode that is easy to recognize but still poorly measured: a model may generate code that appears sensible, yet targets the wrong API surface for the environment in which it will be executed. We study this problem as API drift in LLM-generated quantum code. The problem is broader than any single framework. Quantum software stacks evolve rapidly, tutorials age quickly, and generated code often sits at the boundary between static model knowledge and dynamic tool environments. As a result, version mismatches can arise even when the model appears strong on general code generation. Despite this practical importance, existing quantum-code evaluation mostly asks whether a generated solution passes a task, not whether it targets the requested SDK version or remains robust across version transitions. This gap sits at the intersection of two mature literatures. In code generation, executionbased benchmarks commonly evaluate correctness with Pass@k and related success metrics [4 , 2] . In software evolution, API change is a long-established source of downstream breakage [5 , 3 , 7] . Prior work on quantum code generation, including Qiskit HumanEval, shows that executionbased evaluation is feasible for quantum programming tasks [20] . What remains underexplored is the versioned setting: if a model is explicitly asked to write code for one SDK release, how often does it actually align with that release, how often does the code remain executable in other releases, and how often can version-induced failures be repaired from migr

FAQs

What problem does the paper study in LLM-generated quantum code?

The paper studies API drift: when a model is asked to target a specific quantum SDK version, the generated code may still fail because it targets the wrong API surface for the execution environment.

What is quantum-api-drift and how is version fidelity measured?

quantum-api-drift is a benchmark for measuring version fidelity, defined by execution success on the requested SDK version, cross-version compatibility, failure modes, and documentation-guided repair performance in LLM-generated quantum SDK code.

How do model strengths relate to failure modes and repair results?

Weaker models more often fail due to broken imports, while stronger models more frequently reach deprecation-level failures. Documentation-guided repair succeeds for a fraction of attempts overall and is substantially more effective for migrating to v2.0 than to v1.3.

Jul 16, 2026

Upload by Ivy