[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83571-en":3,"doc-seo-83571-105":30,"detail-sidebar-cat-0-en-105":95},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83571,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents","Repository-level performance-optimization benchmarks such as GSO, SWE-Perf, and SWE-fficiency assess coding agents by patching real repositories and comparing runtime to unoptimized baselines and official reference patches. Leaderboard scores increasingly serve as evidence of progress, yet they may conflate runtime instability, benchmark-specific scoring rules, and solved-task composition. An audit across three benchmarks replays official reference patches, analyzes scoring-driven ranking disagreements, and quantifies how often public submissions match or exceed reference and baseline performance.","Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?  \nZhi Chen Zhensu Sun* Yuling Shi  \nSingapore Management University Singapore Management University Shanghai Jiao Tong University  \nSingapore, Singapore Singapore, Singapore Shanghai, China  \n[zhi.chen.2023@smu.edu.sg](zhi.chen.2023@smu.edu.sg) [zssun@smu.edu.sg](zssun@smu.edu.sg) [yuling.shi@sjtu.edu.cn](yuling.shi@sjtu.edu.cn)  \nDavid Lo Lingxiao Jiang  \nSingapore Management University Singapore Management University  \nSingapore, Singapore Singapore, Singapore  \n[davidlo@smu.edu.sg](davidlo@smu.edu.sg) [lxjiang@smu.edu.sg](lxjiang@smu.edu.sg)  \narXiv :2607 .0 12 1 1v 1 [ cs . SE] 1 Jul 2026  \nAbstract—Repository-level performance-optimization benchmarks such as GSO, SWE-Perf, and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, wereplay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every crossmachine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency’s leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%–82.8% . Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3%(384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8%(449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.  \nIndex Terms—software engineering benchmarks, performance optimization, coding agents, benchmark validity  \nI. INTRODUCTION  \nImproving software performance is a practical and commercially important software-engineering task. Given a codebase and a workload, a coding agent is expected to produce a patch that preserves correctness while improving the program’s performance [1]–[3] . Code-efficiency benchmarks have also grown in scope. Earlier benchmarks focus on function-level edits [4], standalone programs [5], [6], and function-level and file-level code generation tasks [7]–[9]. Recent repository-level  \n*Corresponding author.  \nbenchmarks such as GSO [10], SWE-Perf [11], and SWEfficiency [12] require agents to edit real projects and evaluate those edits with executable workloads, reference patches, and benchmark-specific scoring rules.  \nSuch performance benchmarks differ significantly from functional repair benchmarks because they measure a nonfunctional property. Functional repair benchmarks such as SWE-Bench mainly judge whether the relevant tests pass, a binary and usually reproducible outcome [13] . Performance benchmarks must run the base program, the reference patch, and submitted patches under workloads, then compare noisy runtime measurements. These measurements are not fixed quantities: they fluctuate across runs because of CPU scheduling, cache state, memory bandwidth contention, and machinelevel microarchitectural effects [14]–[16] . The same patch can there","cbCaihtoeu0n3nR6","https://ap.wps.com/l/cbCaihtoeu0n3nR6","pdf",475406,4,1,12,"English","en",105,"# Introduction\n## Background: Repository-level performance benchmarks\n## Core questions and audit goals\n# Method and evaluation focus\n## Reference patch cross-machine validity\n## Scoring-rule impact on leaderboards\n## Public submissions vs reference and baseline","[{\"question\":\"论文研究的核心问题是什么？\",\"answer\":\"研究在于评估这些代码性能优化基准在多机器回放与评分规则影响下，是否仍能可靠反映编码代理的性能改进，并检验公开排行榜是否掩盖真实差距。\"},{\"question\":\"官方参考补丁在跨机器回放时的有效性结果如何？\",\"answer\":\"回放显示参考补丁并非总能满足原基准的有效性规则：GSO为39/102、SWE-Perf为11/140、SWE-fficiency为411/498；其中SWE-Perf更脆弱，许多补丁带来的运行时间变化接近零。\"},{\"question\":\"公开提交的排行榜是否会因评分规则而产生显著差异？\",\"answer\":\"是的。论文指出同一组公开提交在GSO与SWE-fficiency的官方排名存在较多两两对比不一致，并且SWE-fficiency的排行榜计分会把部分任务赋予过高权重，从而影响总体名次。\"},{\"question\":\"公开提交在性能上与参考补丁和未优化基线相比表现如何？\",\"answer\":\"对每个任务的10个公开提交统计显示：在可回放任务中，至少一个提交能在85.3%（384/450）任务上达到或超过参考补丁，并在99.8%（449/450）任务上优于未优化基线代码。\"}]",1784188915,30,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":90,"head_meta":92,"extra_data":94,"updated_unix":28},"are-performance-optimization-benchmarks-reliably-measuring-coding-agents","",{"@graph":36,"@context":89},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/are-performance-optimization-benchmarks-reliably-measuring-coding-agents/83571/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81,85],{"name":72,"@type":73,"acceptedAnswer":74},"论文研究的核心问题是什么？","Question",{"text":75,"@type":76},"研究在于评估这些代码性能优化基准在多机器回放与评分规则影响下，是否仍能可靠反映编码代理的性能改进，并检验公开排行榜是否掩盖真实差距。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"官方参考补丁在跨机器回放时的有效性结果如何？",{"text":80,"@type":76},"回放显示参考补丁并非总能满足原基准的有效性规则：GSO为39/102、SWE-Perf为11/140、SWE-fficiency为411/498；其中SWE-Perf更脆弱，许多补丁带来的运行时间变化接近零。",{"name":82,"@type":73,"acceptedAnswer":83},"公开提交的排行榜是否会因评分规则而产生显著差异？",{"text":84,"@type":76},"是的。论文指出同一组公开提交在GSO与SWE-fficiency的官方排名存在较多两两对比不一致，并且SWE-fficiency的排行榜计分会把部分任务赋予过高权重，从而影响总体名次。",{"name":86,"@type":73,"acceptedAnswer":87},"公开提交在性能上与参考补丁和未优化基线相比表现如何？",{"text":88,"@type":76},"对每个任务的10个公开提交统计显示：在可回放任务中，至少一个提交能在85.3%（384/450）任务上达到或超过参考补丁，并在99.8%（449/450）任务上优于未优化基线代码。","https://schema.org",{"og:url":52,"og:type":91,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":93,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":96},[97,101,105,109,114,119,124,126,131,134,138],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},"Exam",70,"exam",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},5,"Comic",60,"comic",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},6,"Technology",50,"technology",{"id":120,"doc_module":4,"doc_module_name":46,"category_name":121,"show_sort_weight":122,"slug":123},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":29,"slug":125},"research-report",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":129,"slug":130},9,"Religion & Spirituality",20,"religion-spirituality",{"id":129,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":129,"slug":133},"World Cup","world-cup",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":135,"slug":137},10,"Lifestyle","lifestyle",{"id":139,"doc_module":4,"doc_module_name":46,"category_name":140,"show_sort_weight":110,"slug":141},19,"General","general"]