[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84069-en":3,"doc-seo-84069-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84069,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation","Web agents are increasingly used to execute online tasks, creating a pressing need for robust evaluation of both navigation behavior and task completion quality. Prior benchmarks are limited by insufficient scale and weak domain diversity, and LLM-as-Judge methods often miss fine-grained interaction semantics such as exact query formulation and filtering. Many evaluations also overemphasize navigation success while ignoring deployment-critical end-to-end extraction needs. WebRetriever introduces 800 websites and 1,550 tasks across varied domains, plus NavEval and three complementary protocols for holistic assessment.","arXiv :2607 .06 1 18v 1 [ cs .CV] 7 Jul 2026  \nWebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation  \nWei Dong 1 ∗, Tianyu Fu 1 ∗, Zhe Yu 1, Hanning Wang 1, Anyang Su 1, Zhizhou Fang 1, Yuyang Chen 1, Shuo Wang 1, Minghui Wu 1,  \nPing Jiang 1, Zhen Lei2 ,3, and Chenxu Zhao 1†  \n1 Mininglamp Technology  \n2 School of Artificial Intelligence, University of Chinese Academy of Sciences  \n3 MAIS, Institute of Automation, Chinese Academy of Sciences  \n[zhaochenxu@mininglamp.com](zhaochenxu@mininglamp.com)[ ](zhaochenxu@mininglamp.com)[https://github.com/Mininglamp-AI/WebRetriever](https://github.com/Mininglamp-AI/WebRetriever)  \nAbstract. As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit several fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, thereby constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-asJudge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for realworld deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and  \n1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns.  \nWe propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction.  \nExtensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone serves as an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.  \nKeywords: Web Agent · Dataset and Benchmark · LLM-as-a-Judge · Large Language Models · Vision Language Action  \n∗ Equal contribution. †Corresponding author.  \n2 W. Dong, T. Fu et al.  \n1 Introduction  \nRecent advances in large language models have significantly improved language understanding, reasoning, and decision making, driving multimodal web agents to emerge as a key paradigm for automating complex online tasks [3, 14, 17, 19, 24, 35] . By jointly perceiving the visual content, structural information, and textual semantics of web pages, these agents can interact with real websites and have demonstrated strong potential in e-commerce [7,42], customer service [10], and enterprise automation [5, 11] . As model capability and system complexity continue to grow, accurate, reliable, and scalable evaluation of web agents has become a critical bottleneck for further progress.  \nWeb agent evaluation benchmarks are divided into offline and online settings. Offline benchmarks [6, 9, 16, 21, 23, 28–31, 42, 46] provide controlled, reproducible environments but suffer from limited fidelity to real-world complexity, creating gaps between benchmark scores and actual performance. Recent online benchmarks [5, 11, 12, 38, 41, 43] enable live website interaction for better real-world characterization. However, they remain limited in website scale, domain coverage, and ","cbCaiavJv2ov6m8T","https://ap.wps.com/l/cbCaiavJv2ov6m8T","pdf",11641329,1,32,"English","en",105,"# Introduction\n## Motivation and limitations of existing benchmarks\n## WebRetriever benchmark design\n## NavEval and deployment-oriented evaluation protocols\n## Experimental insights and protocol comparisons","[{\"question\":\"What key limitations affect existing web agent evaluation benchmarks?\",\"answer\":\"Existing benchmarks often lack sufficient scale and domain diversity, inadequately model fine-grained interaction semantics in LLM-as-Judge setups, and focus too much on navigation success while neglecting deployment-oriented requirements like end-to-end information extraction.\"},{\"question\":\"What is WebRetriever, and what resources does it provide?\",\"answer\":\"WebRetriever is a large-scale benchmark covering 800 websites and 1,550 tasks across diverse domains and user intents, aiming to improve coverage of real-world interaction needs.\"},{\"question\":\"How does NavEval improve automated evaluation compared with prior LLM-as-Judge methods?\",\"answer\":\"NavEval uses rich interaction context beyond visual screenshots, enabling better alignment with human judgment and supporting more faithful assessment of agent behavior.\"}]",1784192386,81,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"webretriever-a-large-scale-comprehensive-benchmark-for-efficient-web-agent-evaluation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/webretriever-a-large-scale-comprehensive-benchmark-for-efficient-web-agent-evaluation/84069/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What key limitations affect existing web agent evaluation benchmarks?","Question",{"text":75,"@type":76},"Existing benchmarks often lack sufficient scale and domain diversity, inadequately model fine-grained interaction semantics in LLM-as-Judge setups, and focus too much on navigation success while neglecting deployment-oriented requirements like end-to-end information extraction.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is WebRetriever, and what resources does it provide?",{"text":80,"@type":76},"WebRetriever is a large-scale benchmark covering 800 websites and 1,550 tasks across diverse domains and user intents, aiming to improve coverage of real-world interaction needs.",{"name":82,"@type":73,"acceptedAnswer":83},"How does NavEval improve automated evaluation compared with prior LLM-as-Judge methods?",{"text":84,"@type":76},"NavEval uses rich interaction context beyond visual screenshots, enabling better alignment with human judgment and supporting more faithful assessment of agent behavior.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]