[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84795-en":3,"doc-seo-84795-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84795,5909877438554,"Maeve","https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272",8,"Research & Report","AgentGym2 Benchmarking Large Language Model Agents in De-Idealized Real-World Environments","Language agents and LLM agents are increasingly deployed in production, where uncertainty, noise, and incomplete specifications make evaluation significantly harder than in simplified benchmarks. AgentGym2 introduces a realistic, task-instance-driven evaluation framework that measures end-to-end procedure execution, proactive goal clarification, tool discovery via exploration, tool composition for unseen tasks, and robustness to noisy or underspecified inputs. Experiments on 15 proprietary and open-source models show even leading systems underperform on AgentGym2, indicating a substantial gap.","arXiv :2607 .05 174v 1 [ cs .AI] 6 Jul 2026  \nAgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments  \nZhiheng Xi1∗†, Dingwen Yang1 ∗ , Jiaqi Liu1 , Jixuan Huang1 , Honglin Guo1 , Baodai Huang1 , Tinggang Chen1 , Qi Zhang1 , Zhonghang Lu1 , Chenyu Liu1 , Jiajun Sun1 , Jiazheng Zhang1 , Dingwei Zhu1 , Xin Guo1 , Junzhe Wang1 , Zhihao Zhang1 , Yuming Yang1 , Junjie Ye1 , Minghe Gao2 , Dongrui Liu3 , Jiaming Ji4 , Guohao Li5 Tao Gui1†, Qi Zhang1†, Xuanjing Huang1†  \n1 Fudan University 2 Zhejiang University  \n[3](3 Shanghaijiaotong University4 Peking University5 CAMEL-AI.org)[ Shanghaijiaotong University](3 Shanghaijiaotong University4 Peking University5 CAMEL-AI.org)[4](3 Shanghaijiaotong University4 Peking University5 CAMEL-AI.org)[ Peking University](3 Shanghaijiaotong University4 Peking University5 CAMEL-AI.org)[5](3 Shanghaijiaotong University4 Peking University5 CAMEL-AI.org)[ CAMEL-AI.org](3 Shanghaijiaotong University4 Peking University5 CAMEL-AI.org)  \n[zhxi22@m.fudan.edu.cn](zhxi22@m.fudan.edu.cn) , {tgui,qz,[xjhuang}@fudan.edu.cn](xjhuang}@fudan.edu.cn)  \nLanguage agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents’ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications.  \nPerformance  \n30  \n25  \n20  \n15  \n10  \n5  \n(a) Performance vs Model Size  \nPerformance  \n0 200 400 600 800 1000  \nParameters (B)  \n40  \n30  \n20  \n10  \n(b) Performance vs Interaction Rounds  \n\n|  |  |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n|  |  |  |  |  | \u003Cbr> |  |  |\n|  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |\n\nPerformance  \n5 10 15 20 25 30 35  \nAvg Round/Task  \n(c) Performance vs Cost  \n40  \n30  \n20  \n10  \n16$ 32$ 64$ 128$ 256$ 512$ 1024$  \nTotal Cost (USD)  \nFigure 1 | Effect on performance across three dimensions: total cost, the average number of interaction rounds, and the parameter scale.  \n* Equal contribution.†Corresponding authors.  \n1 Our code and dataset are available at [https://github.com/hotdog-zz/Agentgym2](https://github.com/hotdog-zz/Agentgym2) and [https://](https://)[ ](https://)[huggingface.co/datasets/hotdogzz/Agentgym2](huggingface.co/datasets/hotdogzz/Agentgym2)  \n1. Introduction  \nAs language language models advance rapidly, their applications evolve from simple conversational chatbots to autonomous agents capable of complex tasks such as deep research and data analysis, with increasing deployment in production (Hong et al., 2025; Jin et al., 2025; Li et al., 2025a; Zheng et al., 2025; Zhu et al., 2025) . Consequently, it is essential to evaluate their ability in handling practical tasks in real-world (Garg et al., 2025; Ko et al., 2026; Patwardhan et al., 2025) .  \nDespite rapid growth in agent evaluation research, existing benchmarks remains simplified or idealized (Patil et al., 2025; Xu et al., 2024), failing to capture the complexity in real-world environment. They typically provide ","cbCaiuWZdjy12yJ1","https://ap.wps.com/l/cbCaiuWZdjy12yJ1","pdf",5550243,1,35,"English","en",105,"# Introduction\n## Limitations of Existing Agent Benchmarks\n## AgentGym2: Realistic Evaluation Framework\n## System Design and Toolbox Architecture\n## Task Grounding and Measurement Goals","[{\"question\":\"Why do existing agent benchmarks fail to reflect real-world deployment difficulty?\",\"answer\":\"They commonly use simplified, idealized settings with pre-packaged tool interfaces and pre-solved intermediate steps, while real deployments involve uncertainty, noise, ambiguous requests, incomplete or misleading information, and unavailable tools upfront.\"},{\"question\":\"What capabilities does AgentGym2 evaluate beyond reasoning and planning?\",\"answer\":\"AgentGym2 measures end-to-end procedure execution, proactive clarification and goal interpretation, tool discovery through exploration, composition of tools for unseen tasks, and robustness to noisy and underspecified information.\"},{\"question\":\"What does AgentGym2 change in its environment/tool setup compared with query-specific tools?\",\"answer\":\"AgentGym2 provides a basic, composable toolbox instead of query-specific pre-selected tools, creating a larger and more realistic action space and supporting multiple core action categories such as browsing, retrieval, file processing, multimodal understanding, and code execution.\"}]",1784198298,88,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"agentgym2-benchmarking-large-language-model-agents-in-de-idealized-real-world-environments","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/agentgym2-benchmarking-large-language-model-agents-in-de-idealized-real-world-environments/84795/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do existing agent benchmarks fail to reflect real-world deployment difficulty?","Question",{"text":75,"@type":76},"They commonly use simplified, idealized settings with pre-packaged tool interfaces and pre-solved intermediate steps, while real deployments involve uncertainty, noise, ambiguous requests, incomplete or misleading information, and unavailable tools upfront.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What capabilities does AgentGym2 evaluate beyond reasoning and planning?",{"text":80,"@type":76},"AgentGym2 measures end-to-end procedure execution, proactive clarification and goal interpretation, tool discovery through exploration, composition of tools for unseen tasks, and robustness to noisy and underspecified information.",{"name":82,"@type":73,"acceptedAnswer":83},"What does AgentGym2 change in its environment/tool setup compared with query-specific tools?",{"text":84,"@type":76},"AgentGym2 provides a basic, composable toolbox instead of query-specific pre-selected tools, creating a larger and more realistic action space and supporting multiple core action categories such as browsing, retrieval, file processing, multimodal understanding, and code execution.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]