[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83586-en":3,"doc-seo-83586-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83586,8796095360427,"Lucas Martin","https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d",8,"Research & Report","PAIR-BENCH Progressive, Adaptive, and Interactive Feedback for Code Improvement Benchmarking","Large language models are commonly evaluated for code generation and program repair using binary functional correctness, which only records pass or fail and overlooks intermediate progress, feedback usage, regressions, and the refinement trajectory. PAIR-BENCH introduces a progressive and adaptive benchmark for code improvement via feedback-guided refinement. It applies progressive hinting with two controls: failureregion scenarios and hint-depth. The benchmark assesses targeted failure repairs, hint generalization, preservation of correct behavior, and the assistance required, using trajectory-level metrics for finer evaluation than final outcomes.","Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback  \nCuong Chi Le  \nUniversity of Texas at Dallas Texas, USA [cuong.le@utdallas.edu](cuong.le@utdallas.edu)  \nAashish Yadavally  \nUniversity of Central Florida Florida, USA[aashish.yadavally@ucf.edu](aashish.yadavally@ucf.edu)  \nMinh Le-Anh  \nFPT Software AI Center Vietnam [minhla4@fpt.com](minhla4@fpt.com)  \nTien N. Nguyen  \nUniversity of Texas at Dallas Texas, USA  \n[tien.n.nguyen@utdallas.edu](tien.n.nguyen@utdallas.edu)  \narXiv :2607 .0 1360v 1 [ cs . SE] 1 Jul 2026  \nAbstract—Large language models (LLMs) are typically evaluated on code generation and program repair using binary functional correctness: a generated program or patch either passes or fails a test suite. This protocol is simple but coarse, as it ignores partial progress, feedback use, regressions, and the refinement trajectory through which models often improve code. We introduce PAIR-BENCH, a progressive and adaptive benchmark for evaluating code improvement: transforming an incorrect or incomplete program into a more correct one through feedback-guided refinement. PAIR-BENCH uses progressive hinting, a structured feedback protocol with two controls. Failureregion control determines what the feedback targets by grouping hidden failing tests into failure scenarios, while hint-depth control determines how much repair-relevant information is revealed, from coarse symptoms to implementation-level guidance. This design enables PAIR-BENCH to measure whether a model repairs targeted failures, generalizes beyond the hint, preserves alreadycorrect behavior, and how much assistance it requires. By evaluating repair trajectories progressive metrics rather than only final pass/fail outcomes, PAIR-BENCH provides a finergrained assessment of LLM code-improvement capability.  \nI. INTRODUCTION  \nLarge language models (LLMs) have demonstrated strong capabilities across diverse software engineering (SE) tasks, ranging from code generation [1], [2] to automated program repair (APR) [3], [4], [5] . In code generation, models arepresented with a natural language problem statement and evaluated on whether the generated program is functionally correct. In the case of APR, they are evaluated on whether the generated patch for a given buggy program restores correctness without introducing regressions. Despite their apparent differences in problem formulation, both settings share the same success measurement criterion: a binary, instance-level pass/fail outcome determined by the associated test suite.  \nWhile a good first-approximation, such binary measures are sparse and provide a narrow view of overall performance. A model is considered successful only if its generated solution passes all tests in the associated test suite. In practice, however, test suites are often weak and rarely exhaustive [6], [7] . Thus, a model-generated solution that fixes the reported failure but introduces subtle regressions may still be counted as correct, leading to an inflated assessment of model performance [8] . Even when a model is unsuccessful, pass/fail outcomes are uninformative: a solution that fails one edge case is treated  \nthe same as one that fails all tests, nor does a partially correct solution receive any credit for progress.  \nThe limitations of binary measures are further exposed in workflows where models operate in iterative refinement loops, progressively improving upon an initial solution through compiler feedback [9], [10], execution information [11], [12],[2], or model-generated diagnostics [13],[10] . We refer to this process of transforming an incorrect or incomplete program into a more correct one through feedback-guided refinement as code improvement. Despite its inherently incremental nature, evaluation still remains limited to the binary outcome of the final solution, ignoring the refinement trajectory altogether. Such an assessment can therefore be misleading: a stronger model may produce an init","cbCaid17N23TGFHe","https://ap.wps.com/l/cbCaid17N23TGFHe","pdf",1288074,5,1,11,"English","en",105,"# Abstract\n# Introduction\n## Binary measures and their limitations\n## Code improvement as feedback-guided refinement\n## Designing progress-centric evaluation\n# PAIR-BENCH overview\n## Progressive hinting mechanism","[{\"question\":\"What problem does PAIR-BENCH address in existing LLM code evaluation?\",\"answer\":\"Existing evaluations rely on binary pass/fail test-suite outcomes, which ignore partial progress, feedback utilization, regressions, and the refinement trajectory. PAIR-BENCH evaluates whether models make incremental progress toward correctness using trajectory-level metrics.\"},{\"question\":\"How does PAIR-BENCH control the content of feedback?\",\"answer\":\"PAIR-BENCH uses progressive hinting with two controls: failureregion groups hidden failing tests into failure scenarios, and hint-depth controls how detailed repair-relevant information is revealed, from coarse symptoms to implementation-level guidance.\"},{\"question\":\"What does PAIR-BENCH measure beyond final pass/fail results?\",\"answer\":\"PAIR-BENCH measures whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires—assessed through progressive repair trajectories instead of only final outcomes.\"}]",1784189021,28,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"pair-bench-progressive-adaptive-and-interactive-feedback-for-code-improvement-benchmarking","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/pair-bench-progressive-adaptive-and-interactive-feedback-for-code-improvement-benchmarking/83586/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-26","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does PAIR-BENCH address in existing LLM code evaluation?","Question",{"text":76,"@type":77},"Existing evaluations rely on binary pass/fail test-suite outcomes, which ignore partial progress, feedback utilization, regressions, and the refinement trajectory. PAIR-BENCH evaluates whether models make incremental progress toward correctness using trajectory-level metrics.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does PAIR-BENCH control the content of feedback?",{"text":81,"@type":77},"PAIR-BENCH uses progressive hinting with two controls: failureregion groups hidden failing tests into failure scenarios, and hint-depth controls how detailed repair-relevant information is revealed, from coarse symptoms to implementation-level guidance.",{"name":83,"@type":74,"acceptedAnswer":84},"What does PAIR-BENCH measure beyond final pass/fail results?",{"text":85,"@type":77},"PAIR-BENCH measures whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires—assessed through progressive repair trajectories instead of only final outcomes.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":20,"slug":138},19,"General","general"]