[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84136-en":3,"doc-seo-84136-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84136,2336464648746,"Skyler","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Specification Grounding Drives Test Effectiveness for LLM Code","Large language models often produce code that seems correct on standard inputs but fails on edge cases, invalid inputs, and spec-defined corner conditions, such as missing input checks. A common remedy is iterative self-generated testing and repair, yet the source of the benefit is unclear. A controlled experiment isolates whether gains come from test existence or from grounding tests in a specification. Grounding yields markedly more correct code across multiple Claude tiers and held-out sets, with reduced false rejections and robust sensitivity/precision improvements.","arXiv :2607 .06636v 1 [ cs . SE] 7 Jul 2026  \nSPECIFICATION GROUNDING DRIVES TEST EFFECTIVENESS FOR  \nLLM CODE  \nAmin Haeri  \nModel Development Innovation TD Bank, Toronto, Canada [amin.haeri@td.com](amin.haeri@td.com)  \nMahdi Ghelichi  \nModel Development Innovation TD Bank, Toronto, Canada [mahdi.ghelichi@td.com](mahdi.ghelichi@td.com)  \nABSTRACT  \nLarge language models frequently generate code that appears correct on typical inputs yet fails on edge cases, invalid inputs, and specification-defined corner conditions, such as a missing input check or an unhandled boundary case. A popular fix has the model write its own tests and repair until they pass. This helps, but the source of the gain is unclear. Does it come from the tests merely existing, or from their grounding in a specification of what the code should do? We isolate this factor through a controlled experimental design. Holding fixed everything that normally varies (test count, which model writes the tests, and the repair loop), we change a single line of the test-writer’s prompt that controls whether it receives the spec as a checklist of rules. The baseline is no strawman, since it is already told to probe invalid inputs and edge cases. Grounding the tests in the spec produces correct code +38 percentage points more often than this strong baseline on each of three Claude tiers (Haiku 4.5, Sonnet 4.6, Opus 4.8), and +36 points more often on a held-out set. These results indicate that specification grounding, rather than test quantity, is the primary driver of test effectiveness.  \nDoubling the baseline’s budget barely helps, and even combining eight independent ungrounded test suites plateaus far below grounding. An ablation pins the cause to the spec’s content rather than its format. Given the spec as a plain paragraph, the tester still recovers 27 of 30 bugs; asked to plan tests without the spec, it recovers only 2 of 30 . The effect remains robust under stronger baselines.  \nA property-based generator catches 28 of 30 bugs but invents out-of-spec requirements, and a full AlphaCodium-style loop only matches the baseline. It also holds across three model families run full-stack. Beyond Claude, GPT-5.3-codex gains +28 points and Gemini 3.5 Flash gains +19 . Atask-level sign test over 18 tasks is significant at p = 0 .002 (only about a 0.2% chance of a gap this consistent if grounding made no difference) . Specification grounding improves both sensitivity and precision: more real bugs are caught and far less correct code is wrongly rejected. The false-alarm rate is 0% for the grounded tests versus 33% for the baseline, rising to 68% for the baseline when the oracle is the Python standard library itself. On well-specified algorithmic problems it neither helps nor hurts.  \n1 Introduction  \nA modern language model, asked to write a small function, almost always returns something that compiles and handles the common case. Failures frequently arise in boundary conditions and under-specified behaviours. Consider a one-line ticket: “parse a compact integer range: ‘a-b’ expands to the inclusive list [a, . . ., b]; a bare ‘n’ returns [n].” Claude Opus 4.8 writes:  \ndef parse_range(spec: str) -> list [int]:  \nspec = spec.strip()  \nstart, sep, end = spec.partition( \"- \" )  \nif sep == \"\" : \\# bare ’n’  \nreturn [int(spec)]  \na, b = int(start), int(end)  \nFigure 1: The experiment in one picture. A code model writes one implementation from a prose ticket. A fixed tester then writes tests under one of two prompts differing in a single line: SPEC gets the spec as a checklist and writes one test per rule (blue); FREE+, our fair baseline, gets only the ticket but is told to “cover the edges”(orange) . The tests drive an identical repair loop, and an independent gold oracle the models never see judges the final code. Only the test prompt changes across arms; §4.2 pins the effect on its content, not its enumeration.  \nif a > b:  \nraise ValueError(f \"start {a} > end {b} \" ) return list (r","cbCaibn6pCJ2BnQB","https://ap.wps.com/l/cbCaibn6pCJ2BnQB","pdf",356718,2,1,27,"English","en",105,"# Introduction\n## Experiment design and variable isolation\n## Results across model tiers and held-out sets\n## Ablation and oracle comparisons\n## Broader evaluation with other generators","[{\"question\":\"What problem does the paper address in LLM-generated code?\",\"answer\":\"LLMs can generate code that compiles and works on typical inputs but fails on edge cases, invalid inputs, and corner conditions defined by specifications, such as missing input checks and unhandled boundaries.\"},{\"question\":\"How does the experiment isolate the reason test-and-repair improves effectiveness?\",\"answer\":\"It holds test count, which model writes the tests, and the repair loop fixed, while changing a single line in the test-writer prompt that determines whether tests are grounded in the specification as a checklist of rules.\"},{\"question\":\"What are the main findings on test effectiveness from specification grounding?\",\"answer\":\"Specification grounding produces correct code significantly more often than a strong baseline that probes invalid inputs and edge cases without receiving the spec. The improvement also shows reduced false-alarm rates and remains robust under stronger baselines and across multiple model families.\"}]",1784193248,68,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"specification-grounding-drives-test-effectiveness-for-llm-code","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/specification-grounding-drives-test-effectiveness-for-llm-code/84136/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address in LLM-generated code?","Question",{"text":75,"@type":76},"LLMs can generate code that compiles and works on typical inputs but fails on edge cases, invalid inputs, and corner conditions defined by specifications, such as missing input checks and unhandled boundaries.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the experiment isolate the reason test-and-repair improves effectiveness?",{"text":80,"@type":76},"It holds test count, which model writes the tests, and the repair loop fixed, while changing a single line in the test-writer prompt that determines whether tests are grounded in the specification as a checklist of rules.",{"name":82,"@type":73,"acceptedAnswer":83},"What are the main findings on test effectiveness from specification grounding?",{"text":84,"@type":76},"Specification grounding produces correct code significantly more often than a strong baseline that probes invalid inputs and edge cases without receiving the spec. The improvement also shows reduced false-alarm rates and remains robust under stronger baselines and across multiple model families.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]