[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-657538-105":59,"doc-detail-657538-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","evaluating-ill-defined-tasks-in-large-language-models-paper","Evaluating Ill-Defined Tasks in Large Language Models - Paper","","Many evaluations of Large Language Models (LLMs) focus on inherently ill-defined tasks where input/output spaces and success criteria remain unclear. This work analyzes why existing benchmarks and metrics fail to give reliable, diagnostic signals of model capability for such settings. Using two case studies—Complex Instruction Following (CIF) and Natural Language to Mermaid Sequence Diagrams (NL2Mermaid)—the study shows how benchmark coverage, metric design, and LLM-based judging introduce instability and conflate failure modes, while richer criteria enable actionable insights.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/evaluating-ill-defined-tasks-in-large-language-models-paper/657538/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/evaluating-ill-defined-tasks-in-large-language-models-paper/657538.png","ImageObject",300,407,{"name":92,"@type":93},"Connor ","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-30","2026-10-26",true,{"@type":102,"interactionType":103,"userInteractionCount":81},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"Why do existing evaluation benchmarks struggle with ill-defined tasks?","Question",{"text":112,"@type":113},"They often assume fixed, objective correctness, ignore ambiguity, and provide limited visibility into intermediate reasoning or failure modes. This makes scores unstable and non-diagnostic for tasks with subjective success criteria.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"What are the key failure sources identified for ill-defined task evaluation?",{"text":117,"@type":113},"The paper highlights three sources: insufficient real-world coverage and constraint diversity in benchmark data, metrics that collapse multiple diagnostic dimensions into single scores, and variance/bias introduced by LLM-as-a-Judge.",{"name":119,"@type":110,"acceptedAnswer":120},"How do the CIF and NL2Mermaid case studies contribute to the analysis?",{"text":121,"@type":113},"CIF reveals issues such as limited real-world instruction complexity coverage, sensitivity to phrasing, inconsistent non-comparable metrics, and instability from judge-based evaluation. NL2Mermaid shows that multi-faceted criteria can produce actionable insights beyond aggregate scores.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},657538,1792982552,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":81,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":52,"language":139,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":67,"update_tm":129,"read_time":143},687207022233,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","EVALUATING ILL-DEFINED TASKS IN LARGE LANGUAGE MODELS  \nYi Zhou∗  \nIBM Research San Jose, CA [yi.zhou@ibm.com](yi.zhou@ibm.com)  \nBasel Shbita∗ IBM Research San Jose, CA [basel@ibm.com](basel@ibm.com)  \narXiv :2603 . 17067v1 [ cs .CL] 17 Mar 2026  \nABSTRACT  \nMany evaluations of Large Language Models (LLMs) target tasks that are inherently ill-defined, with unclear input and output spaces and ambiguous success criteria. We analyze why existing evaluation benchmarks and metrics fail to provide reliable or diagnostic signals of model capability for such tasks. We examine two case studies: Complex Instruction Following (CIF), where we identify recurring issues including limited coverage of real-world instruction complexity, sensitivity to instruction phrasing, inconsistent and non-comparable metrics, and instability introduced by LLM-based judges; and Natural Language to Mermaid Sequence Diagrams (NL2Mermaid), where we show how multi-faceted evaluation criteria can yield actionable insights beyond aggregate scores. Together, these case studies show that current evaluations frequently conflate distinct failure modes, yielding scores that are unstable, non-diagnostic, and difficult to act upon. Our findings expose fundamental limitations in existing evaluation practices for ill-defined tasks and motivate more robust, interpretable evaluation designs.  \n1 INTRODUCTION  \nLarge language models (LLMs) have made rapid progress in artificial intelligence, driving significant advances in natural language understanding and reasoning, and demonstrating strong performance across a broad range of tasks (Naveed et al., 2025; Matarazzo & Torlone, 2025; Yang et al., 2024; Brown et al., 2020; Wang et al., 2019) . They now support a wide range of workflows, spanning domain-specific fine-tuning pipelines (Ouyang et al., 2022) to agent-based systems capable of multistep planning and tool use (Wu et al., 2025) . Despite their success, LLMs pose unique challenges due to probabilistic outputs, opaque behavior, and hard-to-isolate failure modes (Gao et al., 2025) . As their use grows, evaluation benchmarks must extend beyond performance metrics to diagnose failures and suggest directions for improvement.  \nIn this paper, we focus on a class of problems known as ill-defined tasks, in which goals, constraints, or evaluation criteria are under-specified or open to interpretation. Such tasks are pervasive in real-world applications, e.g., the case of Complex Instruction Following (CIF), where multiple solutions may be acceptable and the boundaries between success and failure are often very subjective. By studying LLM performance on ill-defined tasks, we can gain deeper insights into how they handle user instructions, why models fail, and what signals can guide the development of more robust and interpretable systems. However, most existing benchmarks are designed for well-defined tasks with a unique or narrowly specified notion of correctness (Hendrycks et al., 2021b;a; Wanget al., 2024) . Benchmarks for ill-defined tasks fail to account for ambiguity and the potentially infinite number of valid solutions, and provide limited visibility into intermediate reasoning or failure modes (Chang et al., 2024; Li et al., 2024; Wang et al., 2025) . To address this gap, we analyze why existing evaluation approaches for ill-defined tasks fail to provide reliable or diagnostic signals of model capability. We focus on three primary failure sources: (i) benchmark data lacking realworld coverage and constraint diversity, (ii) metrics that collapse multiple diagnostic dimensions into single scores, conflating failure modes, and (iii) variance and bias introduced by LLM-as-aJudge (LLMaJ) . We examine CIF as our primary use case, studying how representative benchmarks  \n∗Equal contribution.  \nfail to capture meaningful aspects of model capability. We complement this analysis with a second ill-defined task, Natural Language to Mermaid Sequence Diagrams (NL2Mermaid), demonstrat","cbCaiutIsZMJI3fs","https://ap.wps.com/l/cbCaiutIsZMJI3fs","pdf",205451,"English","# Abstract\n# Introduction\n# Challenges in Evaluating Ill-Defined Tasks\n# Complex Instruction Following (CIF)\n# Natural Language to Mermaid Sequence Diagrams (NL2Mermaid)","[{\"question\":\"Why do existing evaluation benchmarks struggle with ill-defined tasks?\",\"answer\":\"They often assume fixed, objective correctness, ignore ambiguity, and provide limited visibility into intermediate reasoning or failure modes. This makes scores unstable and non-diagnostic for tasks with subjective success criteria.\"},{\"question\":\"What are the key failure sources identified for ill-defined task evaluation?\",\"answer\":\"The paper highlights three sources: insufficient real-world coverage and constraint diversity in benchmark data, metrics that collapse multiple diagnostic dimensions into single scores, and variance/bias introduced by LLM-as-a-Judge.\"},{\"question\":\"How do the CIF and NL2Mermaid case studies contribute to the analysis?\",\"answer\":\"CIF reveals issues such as limited real-world instruction complexity coverage, sensitivity to phrasing, inconsistent non-comparable metrics, and instability from judge-based evaluation. NL2Mermaid shows that multi-faceted criteria can produce actionable insights beyond aggregate scores.\"}]","Evaluating Ill-Defined Tasks in Large Language Models - Paper | PDF",25]