[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83445-en":3,"doc-seo-83445-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83445,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","ALGOBENCH Benchmarking Algorithmic Adaptation in Code Generation","Established programming benchmarks like HumanEval and Live CodeBench yield high pass rates but fail to measure whether models can truly reason about algorithms. As problem statements and reference solutions enter public training corpora, later models may succeed via exposure rather than adaptation. ALGOBENCH builds traceable, constraint-shifted algorithmic variants from competitive-programming problems so the original algorithm fails, and evaluates complexity suitability with OPTT, OPTS, TRAPRATE, GAPT, and CONSENS.","ALGOBENCH: Benchmarking Algorithmic Adaptation in Code Generation  \nXinyuan Song 1 Zekun Cai2 ,3 Liang Zhao 1  \n1Emory University, Atlanta, GA, USA 2The University of Tokyo, Tokyo, Japan  \n3LocationMind, Tokyo, Japan  \n{[xinyuan.song](xinyuan.song) , [liang.zhao}@emory.edu](liang.zhao}@emory.edu) , [caizekun@csis.u-tokyo.ac.jp](caizekun@csis.u-tokyo.ac.jp)  \narXiv :2607 .00062v1 [ cs . SE] 30 Jun 2026  \nAbstract  \nHigh pass rates on established programming benchmarks such as HumanEval and Live  \nCodeBench do not always show whether a model can reason about algorithms. Many fixed benchmarks eventually become part of the public training ecosystem through released problem statements, editorials, and generated solutions, allowing later models to improve partly by exposure rather than by stronger algorithmic ability. We introduce ALGOBENCH, a framework that automatically builds novel algorithmic problems from known competitiveprogramming problems through structured constraint-shifting transformations. Each accepted ALGOBENCH variant is traceable to a source problem, but must make the original reference algorithm fail. Beyond pass@k, we introduce complexity-aware metrics—including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS—to test whether a solution is not only functionally correct but also asymptotically suitable for the generated problem. Experiments across multiple LLMs and prompting strategies show that performance drops sharplyon ALGOBENCH variants, retrieval can increase reuse of the old algorithm, and many correct-looking solutions fail to meet the required complexity. Error analysis shows that failures are mainly algorithmic rather than implementation-level, suggesting that ALGOBENCH evaluates adaptation beyond functional correctness. Code is available at [https://](https://)[ ](https://)[github.com/Hik289/algobench.git](github.com/Hik289/algobench.git).  \n1 Introduction  \nLarge language models (LLMs) now perform well on many programming and algorithmic benchmarks. Recent systems obtain high pass@1 scoreson standard datasets such as HumanEval (Chenet al., 2021) and LiveCodeBench (Jain et al., 2024) . These scores, however, do not necessarily measure algorithmic reasoning. Many programming problems, editorials, and reference solutions are  \npublicly available, and web-scale pretraining corpora may contain exact or near-duplicate problem– solution pairs. A model can therefore pass a benchmark by recalling a known solution pattern, rather than by deriving the required algorithm from the stated constraints (Golchin and Surdeanu, 2024 ; Shi et al., 2024) .  \nRecent benchmarks reduce direct contamination by using newer or harder problems. LiveCodeBench (Jain et al., 2024) and LiveBench (White et al., 2025) collect released tasks, while ProBench evaluates models on competitiveprogramming problems with online submissions, difficulty grading, and algorithm-tag analysis (Yang et al., 2025) . Humanity’s Last Code Exam (HLCE) further uses IOI and ICPC World Finals problems to test advanced reasoning models on difficult contest tasks (Li et al., 2025) . These benchmarks improve over older static datasets, but they remain fixed once released. Their problem statements, editorials, and model-generated solutions can later enter training corpora, allowing future models to improve partly by exposure rather than by stronger algorithmic reasoning. Thus, a benchmark for algorithmic ability should not rely only on a static set of hard or recently collected problems.  \nThis motivates a different benchmark-design question: how can we automatically generate new algorithmic problems so that LLMs cannot improve by memorizing existing problem statements and solutions? Such a benchmark satisfy two conditions. First, each generated problem should be traceable to a known source problem, so that old-template reuse can be measured rather than only suspected. Second, the generated problem should require a new algorithmic treatment ora different asymptotic comple","cbCaimMFM1xKKCpx","https://ap.wps.com/l/cbCaimMFM1xKKCpx","pdf",8819976,4,1,23,"English","en",105,"# Abstract\n# Introduction\n## Motivation: limitations of fixed benchmarks\n## Design goal: prevent memorization and exposure effects\n## ALGOBENCH framework and transformations\n## Quality gates and complexity verification","[{\"question\":\"Why can fixed programming benchmarks overestimate algorithmic reasoning?\",\"answer\":\"Because released problem statements and reference solutions can become part of public pretraining corpora, letting models pass by recalling known patterns rather than deriving algorithms from constraints.\"},{\"question\":\"How does ALGOBENCH ensure a generated problem tests adaptation rather than reuse?\",\"answer\":\"It generates variants through structured constraint-shifting transformations that remain traceable to a source problem, while requiring the original reference algorithm to fail and a new algorithmic solution to work.\"},{\"question\":\"What does ALGOBENCH measure beyond pass@k?\",\"answer\":\"It adds complexity-aware metrics (including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS) and uses a deterministic complexity verifier to distinguish asymptotically suitable solutions from slower ones that still pass permissive limits.\"}]",1784187822,58,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"algobench-benchmarking-algorithmic-adaptation-in-code-generation","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/algobench-benchmarking-algorithmic-adaptation-in-code-generation/83445/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why can fixed programming benchmarks overestimate algorithmic reasoning?","Question",{"text":75,"@type":76},"Because released problem statements and reference solutions can become part of public pretraining corpora, letting models pass by recalling known patterns rather than deriving algorithms from constraints.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does ALGOBENCH ensure a generated problem tests adaptation rather than reuse?",{"text":80,"@type":76},"It generates variants through structured constraint-shifting transformations that remain traceable to a source problem, while requiring the original reference algorithm to fail and a new algorithmic solution to work.",{"name":82,"@type":73,"acceptedAnswer":83},"What does ALGOBENCH measure beyond pass@k?",{"text":84,"@type":76},"It adds complexity-aware metrics (including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS) and uses a deterministic complexity verifier to distinguish asymptotically suitable solutions from slower ones that still pass permissive limits.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]