[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83526-en":3,"doc-seo-83526-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83526,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",6,"Technology","ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation","Large Language Models (LLMs) act as programming assistants, yet code generation quality is limited by input requirements that are often ambiguous, incomplete, or underspecified in real software development. While current one-shot synthesis excels, proactive clarification of user intent remains insufficiently benchmarked. ClarifyCodeBench is introduced as an interactive benchmark built from real-world tasks, with annotated ambiguity types, clarification questions, and ground-truth answers. Two metrics assess interaction quality: Turn-discounted Key Question Rate and Optimal Round Adherence, supported by evaluations of six leading LLMs.","ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous  \nRequirements for Code Generation  \narXiv :2607 .007 1 1v 1 [ cs . SE] 1 Jul 2026  \nZheng Fang Peking University  \nBeijing, China [fangz@pku.edu.cn](fangz@pku.edu.cn)  \nDongming Jin  \nPeking University Beijing, China [dmjin@stu.pku.edu.cn](dmjin@stu.pku.edu.cn)  \nYihong Dong  \nPeking University Beijing, China [dongyh@stu.pku.edu.cn](dongyh@stu.pku.edu.cn)  \nYongmin Li  \nPeking University Beijing, China [liyongmin@pku.edu.cn](liyongmin@pku.edu.cn)  \nKechi Zhang  \nPeking University Beijing, China [zhangkechi@pku.edu.cn](zhangkechi@pku.edu.cn)  \nZhi Jin  \nPeking University Beijing, China [zhijin@pku.edu.cn](zhijin@pku.edu.cn)  \nGe Li∗ Peking University Beijing, China [lige@pku.edu.cn](lige@pku.edu.cn)  \nAbstract  \nLarge Language Models (LLMs) have emerged as powerful programming assistants. However, the efficacy of code generation is fundamentally constrained by the quality of input requirements, which, in real-world software development, are frequently ambiguous, incomplete, or underspecified. While LLMs excel at one-shot code synthesis, their ability to proactively clarify intent remains underexplored, as a critical trait for robust software engineering. Existing benchmarks largely overlook this interactive bottleneck, assuming perfectly specified prompts that do not reflect the iterative nature of requirement elicitation. To bridge this gap, we introduce ClarifyCodeBench, a novel interactive benchmark specifically designed to evaluate LLMs’ capability in resolving requirement ambiguity. Constructed from real-world programming tasks, ClarifyCodeBench features high-quality manual annotations, including 􀀣 unique ambiguity types, associated clarification questions, and corresponding ground-truth answers. Furthermore, we formalize two rigorous metrics to assess the interaction quality: Turn-discounted Key Question Rate (TKQR), which penalizes inefficient questioning, and Optimal Round Adherence (ORA), which measures the precision of the elicitation process. We conduct a systematic evaluation of six state-of-the-art LLMs using ClarifyCodeBench. Our empirical results yield three critical insights: 1) Capability Decoupling: Strong code generation performance does not inherently translate to effective requirement clarification; 2) The Reasoning Paradox: While increased computational \"thinking\"(e.g., via reasoning models) enhances code correctness, it yields marginal gains in identifying ambiguities; 3) The Multi-ambiguity Ceiling: LLMs’ clarification performance degrades sharply as the density of ambiguities increases, revealing a significant bottleneck in handling complex, real-world specifications. Our work underscores the necessity for future AI4SE research to transition from static synthesis to interactive elicitation.  \n∗ Corresponding author.  \nProblem  \nYou are given two positive integers A and B. Output the square of A + B  \nInput  \nThe input is given from the Standard Input in the following format:  \nA B  \nOutput  \nPrint the answer.  \nConstraints  \n1 \u003C= A, B \u003C= 2025  \nAll input values are integers  \nCompute (􀜣 + 􀜤)2 or 􀜣2 +􀜤 ?  \nFigure 1: An ambiguous code generation requirement.  \nKeywords  \nCode Generation, Requirements Elicitation, Large Language Models  \n1 Introduction  \nLarge language models (LLMs) are increasingly used as programming assistants [1, 28] . Prior studies have reported productivity gains when developers use LLMs for software engineering tasks [27, 33] . Among these tasks, code generation is one of the most fundamental [5] . In this setting, users provide requirements in natural language, and LLMs generate executable code accordingly [3, 5] . However, the requirements presented to LLMs are often ambiguous or incomplete [26] . For example, Figure 1 shows a code generation requirement that admits at least two interpretations: whether the target expression is (􀀖+􀀗)2 or 􀀖2 +􀀗 . Such ambiguity can hurt code generation performance and, more importantly, pro","cbCaihDDtL3RHt2F","https://ap.wps.com/l/cbCaihDDtL3RHt2F","pdf",823515,1,11,"English","en",105,"# Introduction\n## Problem Motivation\n## Limitations of Existing Benchmarks\n## ClarifyCodeBench Overview\n# Metrics and Evaluation","[{\"question\":\"Why is requirement ambiguity a critical issue for LLM-based code generation?\",\"answer\":\"Because ambiguous or incomplete requirements can lead the LLM to generate outputs that do not match the user’s actual intent, harming both correctness and usability in real development workflows.\"},{\"question\":\"What gap do existing code generation benchmarks fail to cover?\",\"answer\":\"They mainly assume fully specified prompts and evaluate functional correctness, but they do not test whether a model can first detect ambiguity, ask the right clarification questions, and then generate correct code based on answers.\"},{\"question\":\"How does ClarifyCodeBench evaluate an LLM’s clarification performance?\",\"answer\":\"It uses an interactive setup with annotated ambiguity types, clarification questions, and ground-truth answers, and introduces two metrics: Turn-discounted Key Question Rate (penalizing inefficient questioning) and Optimal Round Adherence (measuring precision of the elicitation process).\"}]",1784188624,28,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"clarifycodebench-evaluating-llms-on-clarifying-ambiguous-requirements-for-code-generation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/clarifycodebench-evaluating-llms-on-clarifying-ambiguous-requirements-for-code-generation/83526/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is requirement ambiguity a critical issue for LLM-based code generation?","Question",{"text":75,"@type":76},"Because ambiguous or incomplete requirements can lead the LLM to generate outputs that do not match the user’s actual intent, harming both correctness and usability in real development workflows.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What gap do existing code generation benchmarks fail to cover?",{"text":80,"@type":76},"They mainly assume fully specified prompts and evaluate functional correctness, but they do not test whether a model can first detect ambiguity, ask the right clarification questions, and then generate correct code based on answers.",{"name":82,"@type":73,"acceptedAnswer":83},"How does ClarifyCodeBench evaluate an LLM’s clarification performance?",{"text":84,"@type":76},"It uses an interactive setup with annotated ambiguity types, clarification questions, and ground-truth answers, and introduces two metrics: Turn-discounted Key Question Rate (penalizing inefficient questioning) and Optimal Round Adherence (measuring precision of the elicitation process).","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":121,"slug":122},8,"Research & Report",30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]