[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83916-en":3,"doc-seo-83916-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83916,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Is Three the Magic Number An Empirical Evaluation of LLM Based Repair Loops","Iterative repair loops are a central design pattern in LLM-based software engineering workflows, where models repeatedly generate artifacts, validate them via signals like compiler errors or failing tests, and produce repaired versions. An underexplored issue is how the chosen iteration limit affects effectiveness. Across multiple software engineering tasks—code generation, test generation, and code translation—results show diminishing returns: most gains arise from the first three to four iterations, while later repairs add only marginal improvement. Workflow orchestration and feedback design dominate model choice, so repair budgets should be treated as an explicit experimental variable to ensure cost, runtime, and reproducibility are properly measured.","Is Three the Magic Number?  \nAn Empirical Evaluation of LLM-Based Repair Loops  \nTobias Kiecker  \nHumboldtUniversität zu Berlin Berlin, Germany  \nEik Reichmann  \nHumboldtUniversität zu Berlin Berlin, Germany  \nHosung Kang  \nKorea University Seoul, Korea  \nGabin An  \nKorea University Seoul, Korea  \nLars Grunske  \nHumboldtUniversität zu Berlin Berlin, Germany  \narXiv :2607 .05 197v 1 [ cs . SE] 6 Jul 2026  \nAbstract  \nIterative repair loops have become a core design pattern in LLMbased software engineering systems. These workflows repeatedly generate, validate, and repair artifacts using feedback such as compiler errors or test failures. Despite their widespread use, the impact of repair-loop iteration limits remains poorly understood, as most prior work adopts fixed, often arbitrary, repair budgets.  \nWe study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation. Across several representative workflows, datasets, and contemporary low-cost LLMs, we observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements. We further find that repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself. These results suggest that repair budgets should be treated as an explicit experimental variable, as they directly affect evaluation outcomes, computational cost, runtime, and reproducibility in LLM-based software engineering research.  \n1 Introduction  \nLarge Language Models (LLMs) are far from perfect. Nevertheless, they are increasingly used in software engineering (SE) research and practice [5]. A common pattern when using LLMs for programming tasks is iterative refinement: users generate a solution, validate the output, identify remaining issues such as compiler errors, failing test cases, or missing edge cases, and feed these back to the model for repair. This process is repeated until the generated artifact either satisfies the requirements or the user abandons the attempt and reformulates the prompt. This iterative repair behavior has also been adopted by many LLM-based SE tools [9–14, 18, 20, 21, 24] .  \nInstead of relying on manual feedback, such systems automatically validate generated artifacts using mechanisms such as compilation, static analysis, or test execution. When validation fails, the observed errors are provided to the model in order to generate a repaired version automatically. To prevent infinite repair cycles or oscillating behaviors in which one repair introduces new faults, these systems typically enforce an arbitrary fixed iteration limit [16, 23]. Furthermore, with the emergence of autonomous LLM agents [2, 19, 22] repair loops are becoming even more relevant. Such agents internally perform iterative cycles of tool calls, generation, validation, testing, and repair in order to accomplish complex SE tasks. As agent-based workflows become increasingly common, the choice of repair-loop limits directly affects the practicality and efficiency of these systems.  \nInput  \nSolution  \nError  \nFigure 1: Generalized SE workflow with LLM-repair loop  \nChoosing the right limit for repair loop iterations is important and impactful for several reasons. First, each additional repair attempt increases API usage costs and energy consumption, contributing both to higher economic as well as environmental impact [4] . Second, repeated inference calls increase runtime which can lead to slower development workflows. Third, iteration limits directly influence reproducibility and comparability across studies. Different repair budgets may lead to substantially different performance outcomes, making it difficult to fairly compare approaches when iteration limits are selected arbitrarily or are insufficiently documented [6] .  \nDespite their importance, iteration limits","cbCaitWpq2bzUxaJ","https://ap.wps.com/l/cbCaitWpq2bzUxaJ","pdf",520580,1,5,"English","en",105,"# Introduction\n# Repair Loops in LLM-Based SE-Tools","[{\"question\":\"What are LLM-based repair loops in software engineering systems?\",\"answer\":\"They iteratively generate, validate (e.g., via compilation, static analysis, or test execution), and repair artifacts until correctness is achieved or a preset iteration limit is reached.\"},{\"question\":\"How do repair-loop iteration limits affect performance across tasks?\",\"answer\":\"Across representative workflows for code generation, test generation, and code translation, benefits diminish after a small number of iterations, with most improvements coming from the first three to four repair attempts.\"},{\"question\":\"What determines repair behavior more: the model or the workflow design?\",\"answer\":\"Repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself.\"}]",1784191428,13,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"is-three-the-magic-number-an-empirical-evaluation-of-llm-based-repair-loops","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/is-three-the-magic-number-an-empirical-evaluation-of-llm-based-repair-loops/83916/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What are LLM-based repair loops in software engineering systems?","Question",{"text":75,"@type":76},"They iteratively generate, validate (e.g., via compilation, static analysis, or test execution), and repair artifacts until correctness is achieved or a preset iteration limit is reached.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How do repair-loop iteration limits affect performance across tasks?",{"text":80,"@type":76},"Across representative workflows for code generation, test generation, and code translation, benefits diminish after a small number of iterations, with most improvements coming from the first three to four repair attempts.",{"name":82,"@type":73,"acceptedAnswer":83},"What determines repair behavior more: the model or the workflow design?",{"text":84,"@type":76},"Repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":21,"slug":137},19,"General","general"]