[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84760-en":3,"doc-seo-84760-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84760,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","ToolFailBench Diagnosing Tool-Use Failures in LLM Agents","Tool calling is central to modern language model agents, yet aggregate benchmark scores can obscure where tool use fails. ToolFailBench provides a diagnostic benchmark covering 1,000 single-turn tasks across finance, medicine, law, cybersecurity, and real estate. It separates failure modes such as Tool-Skip, ResultIgnore, Output-Fabrication, and UnnecessaryTool-Use using a rule classifier and two LLM judges. Results across 19 models show a best Clean Tool-Use Rate of 86.33% and distinct failure profiles under similar aggregate scores.","ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents  \nHarsh Soni 1  \narXiv :2607 .04686v 1 [ cs .CL] 6 Jul 2026  \nAbstract  \nTool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn’t guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, ResultIgnore, Output-Fabrication, and UnnecessaryTool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across  \n19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an AlwaysCall pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tooluse evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed. [Code: github.com/SoHarshh/ToolFailBench](Code: github.com/SoHarshh/ToolFailBench)  \n1. Introduction  \nTool calling is now a core part of language model agents. Modern agents are expected to call external functions, APIs, search tools, databases, and domain-specific services, then use the returned information in their final answer (Schick et al., 2023 ; Patil et al., 2025) . This matters in settings such as customer support (Yao et al., 2024), finance (Bigeard  \n1UC Berkeley. Correspondence to: Harsh Soni \u003C[harsh.soni@berkeley.edu](harsh.soni@berkeley.edu) >.  \nPublished at ICML 2026 Workshops, 43rd International Conference on Machine Learning, Seoul, South Korea, 2026 . Copyright 2026 by the author(s) .  \net al., 2025), healthcare (Jiang et al., 2025), software engineering (Jimenez et al., 2024), and long-horizon multiapplication workflows (Li et al., 2025) . In these settings, reliability depends not only on whether a model calls a tool, but also on whether it uses the tool result correctly.  \nExisting benchmarks measure important parts of tool use, but they often collapse different failures into one score. BFCL evaluates function-call correctness (Patil et al., 2025);τ-bench evaluates whether an agent reaches the right end state in simulated customer-service tasks (Yao et al., 2024); ToolLLM measures tool-use success across realworld APIs (Qin et al., 2024); and Toolathlon evaluates long-horizon task execution across many software applications (Li et al., 2025) . These benchmarks are useful, but a single pass rate or accuracy number may hide ”why” a model failed. A model that never calls a needed tool, a model that calls the tool but ignores the result, and a model that calls the tool but invents extra information can all look similar under aggregate evaluation, even though they fail indifferent ways.  \nWe introduce ToolFailBench, a diagnostic benchmark for measuring where tool-use failures occur. ToolFailBench contains 1,000 single-turn tasks across five professional domains: finance, medicine, law, cybersecurity, and real estate. Tool-required tasks are built as parametric traps: the mock tool return contradicts a likely memorized prior, so a model must use the returned value instead of falling back on memory. Control tasks test the opposite behavior, asking whether a model can answer directly without calling a tool unnecessarily.  \nToolFailBench assigns each response a failure-mode label rather than only a final success score. For toolrequired tasks, we separate Tool-Skip, Result-Ignore, OutputFabrica","cbCaik8fmHMStVcz","https://ap.wps.com/l/cbCaik8fmHMStVcz","pdf",2549988,1,18,"English","en",105,"# Abstract\n# Introduction\n# Related Work","[{\"question\":\"What problem does ToolFailBench address in evaluating LLM tool calling?\",\"answer\":\"Aggregate scores can hide the reasons models fail, since different error types can lead to similar final task accuracy.\"},{\"question\":\"How does ToolFailBench structure tasks to expose different tool-use failures?\",\"answer\":\"Tool-required tasks use parametric traps that contradict memorized priors, forcing the model to trust tool outputs, while control tasks test whether the model can answer directly without unnecessary tools.\"},{\"question\":\"Which failure modes does ToolFailBench label, and how are labels produced?\",\"answer\":\"It labels Tool-Skip, ResultIgnore, Output-Fabrication, and UnnecessaryTool-Use, using a deterministic rule classifier plus two LLM judges aggregated by majority vote.\"}]",1784198077,45,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"toolfailbench-diagnosing-tool-use-failures-in-llm-agents","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/toolfailbench-diagnosing-tool-use-failures-in-llm-agents/84760/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does ToolFailBench address in evaluating LLM tool calling?","Question",{"text":75,"@type":76},"Aggregate scores can hide the reasons models fail, since different error types can lead to similar final task accuracy.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does ToolFailBench structure tasks to expose different tool-use failures?",{"text":80,"@type":76},"Tool-required tasks use parametric traps that contradict memorized priors, forcing the model to trust tool outputs, while control tasks test whether the model can answer directly without unnecessary tools.",{"name":82,"@type":73,"acceptedAnswer":83},"Which failure modes does ToolFailBench label, and how are labels produced?",{"text":84,"@type":76},"It labels Tool-Skip, ResultIgnore, Output-Fabrication, and UnnecessaryTool-Use, using a deterministic rule classifier plus two LLM judges aggregated by majority vote.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]