[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82140-en":3,"doc-seo-82140-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82140,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",8,"Research & Report","Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q","Quantum programs rely on Bugs4Q, a widely used benchmark of real Qiskit defects, yet prior evaluation assumes benchmark labels remain valid and that generated fixes run in the target environment. This study revalidates two Bugs4Q releases containing 67 unique real Qiskit defects by adding missing executable tests and rerunning all entries across six pinned Qiskit versions. Results reveal silent label inversion, so correctness depends on the (benchmark, version) pair. Four LLMs are tested, with GPT-5.4 attaining the best pass@10 (48.8%).","Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q  \nSaumya Brahmbhatt 1,†, Mitali Hukkeri 1,†, Dongchan Kim 1 , M.V. Panduranga Rao2 , and Lei Zhang 1  \n1Department of Information Systems, University of Maryland, Baltimore County, USA  \n2Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India  \n1 {saumyab1,mhukker1,dkim26,[leizhang](leizhang}@umbc.edu)[}](leizhang}@umbc.edu)[@umbc.edu](leizhang}@umbc.edu)  \n[2](2 {mvp}@cse.iith.ac.in)[ {](2 {mvp}@cse.iith.ac.in)[mvp](2 {mvp}@cse.iith.ac.in)[}](2 {mvp}@cse.iith.ac.in)[@cse.iith.ac.in](2 {mvp}@cse.iith.ac.in)  \n†These authors contributed equally to this work.  \narXiv :2607 .09007v 1 [ cs . SE] 10 Jul 2026  \nAbstract—In quantum programs, Bugs4Q is a widely used benchmark containing real quantum defects. However, its evaluation assumes that benchmark labels remain valid and that generated fixes execute in the target environment. We evaluate two Bugs4Q versions containing 67 unique real Qiskit defects, adding executable tests where missing, andre-run all entries across six pinned Qiskit releases (0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, and 2.3.1). We find that quantum benchmarks can suffer from silent label inversion: entries become invalid without errors when reference fixes stop executing or buggy programs no longer reproduce failures. Thus, correctness depends on the (benchmark, version) pair rather than the benchmark alone. We evaluate four LLMs (GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini), generating up to 10 repair candidates per defect and testing them across all versions. GPT-5.4 achieves the highest pass@10 (48.8%), followed by GPT-5.4-mini (47.3%), GPT- 5o-mini (30.3%), and GPT-4o-mini (22.6%). All models perform best on Qiskit 0.45.0 and decline after the Qiskit 1.0 transition. Many failures arise from deprecated or incompatible APIs rather than incorrect repairs, and 64% of successful repairs occur on entries invalid under the target version. We release a re-validated, version-pinned Bugs4Q benchmark and show that benchmark validation must precede repair evaluation.  \nIndex Terms—quantum software debugging, automated program repair, large language models  \nI. INTRODUCTION  \nWhile testing and debugging quantum software remain challenging due to quantum mechanics, such as superposition and entanglement [1]–[3], automated program repair (APR) utilizing large language models (LLMs) has significantly advanced this agenda in classical software. Prior work has shown that large pre-trained models can outperform earlier APR techniques under several settings [4]–[7] .  \nIn contrast, LLM-based repair of quantum software is still in its early stages. Existing studies provide promising but fragmented evidence. Guo et al. evaluated ChatGPT on Bugs4Q and showed that ChatGPT-based repair is feasible on real defects [8], [9] . QBugLM [10] studies quantum debugging with taxonomy-driven bug injec-  \ntion in OpenQASM 3.0 . QuanBench+ evaluates multiframework quantum code generation with executable tests and a feedback-based repair setting [11] . However, these APR studies rely on benchmarks whose labelsand executability are stable. A recent reproducibility study showed that Bugs4Q degrades substantially across Qiskit versions, with reproducibility dropping sharply on modern releases [12] .  \nMotivated by this gap, we benchmark four contemporary LLMs (GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini) on repairing Qiskit programs using the two Bugs4Q releases with a version-pinned evaluation protocol. We further author executable tests for entries that originally lacked them, and we execute both benchmark references and model-generated fixes under six controlled Qiskit environments (0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, and 2.3.1) spanning the major-version transition around Qiskit 1.0 . Our goal is not only to measure how often LLMs repair real quantum defects, but also to determine how much of the reported success o","cbCainlTRCJ4g1FR","https://ap.wps.com/l/cbCainlTRCJ4g1FR","pdf",166817,1,4,"English","en",105,"# Introduction\n# Our Method\n## Benchmark Construction\n## Test Construction","[{\"question\":\"Why can Bugs4Q benchmark labels become invalid during evaluation?\",\"answer\":\"Bugs4Q can suffer from silent label inversion when reference fixes stop executing or buggy programs no longer reproduce failures. In that case, entries become invalid without obvious errors.\"},{\"question\":\"What evaluation protocol is used across Qiskit versions?\",\"answer\":\"The study reruns benchmark references and model-generated fixes across six pinned Qiskit releases (0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, 2.3.1), treating validity as a property of the (case, version) pair.\"},{\"question\":\"Which LLM performs best and how do pass rates change by version?\",\"answer\":\"GPT-5.4 achieves the highest pass@10 at 48.8%, followed by GPT-5.4-mini (47.3%). Models perform best on Qiskit 0.45.0 and decline after the Qiskit 1.0 transition.\"}]",1784178414,10,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"benchmarking-large-language-models-on-repairing-qiskit-programs-using-bugs4q","",{"@graph":35,"@context":84},[36,52,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":21},"https://docshare.wps.com/document/benchmarking-large-language-models-on-repairing-qiskit-programs-using-bugs4q/82140/",{"url":51,"name":13,"@type":53,"author":54,"headline":13,"publisher":56,"fileFormat":59,"inLanguage":23,"description":14,"dateModified":60,"datePublished":61,"encodingFormat":59,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":55},"Person",{"url":40,"name":57,"@type":58},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why can Bugs4Q benchmark labels become invalid during evaluation?","Question",{"text":74,"@type":75},"Bugs4Q can suffer from silent label inversion when reference fixes stop executing or buggy programs no longer reproduce failures. In that case, entries become invalid without obvious errors.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What evaluation protocol is used across Qiskit versions?",{"text":79,"@type":75},"The study reruns benchmark references and model-generated fixes across six pinned Qiskit releases (0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, 2.3.1), treating validity as a property of the (case, version) pair.",{"name":81,"@type":72,"acceptedAnswer":82},"Which LLM performs best and how do pass rates change by version?",{"text":83,"@type":75},"GPT-5.4 achieves the highest pass@10 at 48.8%, followed by GPT-5.4-mini (47.3%). Models perform best on Qiskit 0.45.0 and decline after the Qiskit 1.0 transition.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":57,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":28,"slug":132},"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]