[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85425-en":3,"doc-seo-85425-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85425,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",8,"Research & Report","Can Argus Judge Them All? Comparing VLMs Across Domains","Vision-Language Models (VLMs) are increasingly used in retrieval, content generation, and decision-support pipelines, where selection is often based on benchmark rankings. Yet similar benchmark capability can mask large behavioral differences across datasets and evaluation conditions, creating a Capability-Reliability Gap. The work introduces ARGUS-EVAL, evaluating VLM behavior via Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M). Experiments on multiple VLMs show capability- and reliability-oriented rankings diverge notably.","Can Argus Judge Them All? Comparing VLMs Across Domains  \nHarsh Joshi1 , Gautam Siddharth Kashyap2 , Rafiq Ali3 , Ebad Shabbir3 , Niharika Jain4 , Sarthak Jain5 , Jiechao Gao6 * , Usman Naseem2  \n1Bharati Vidyapeeth, New Delhi, India  \n2Macquarie University, Sydney, Australia  \n3DSEU-Okhla, New Delhi, India  \n4Vivekananda Institute of Professional Studies, New Delhi, India  \n5IIIT-Delhi, India  \n6 Center for SDGC, Stanford University, California, USA  \narXiv :2507 .0 1042v2 [ cs .IR] 13 Jul 2026  \nAbstract  \nVision-Language Models (VLMs) are increasingly used in industry VLM applications such as retrieval systems, content generation platforms, and decision-support workflows, where model selection is commonly guided by benchmark rankings. These rankings are largely determined by retrieval, captioning, and reasoning downstream tasks; however, models with similar task performance often show substantially different behavior across datasets.  \nThis creates a Capability-Reliability Gap between benchmark performance and observed model stability. We present ARGUS-EVAL, a capability-reliability-oriented evaluation framework for VLMs that characterizes model behavior through Benchmark Capability P(M), Cross-Dataset Consistency CDC (M), Robustness Retention RR (M), and Efficiency E (M) . We evaluate CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct across retrieval, captioning, and reasoning downstream tasks. The results reveal notable differences between capability-oriented and reliability-oriented rankings. Qwen-2.5VL-3BInstruct achieves the strongest overall capability (R@1 = 82 .7%, BLEU-4 = 47 .2%, CIDEr = 141 .6, CDC = 0 .91), whereas CLIP records the lowest latency (31ms) and memory footprint (0 .9GB) .  \n1 Introduction  \nVision-Language Models (VLMs) combine visual perception and language understanding within a unified architecture for machines to retrieve (Singh, 2026), caption (Liang et al., 2026), and reason (Nguyen et al., 2026) about multimodal content (Ma et al., 2026) . As VLMs become increasingly integrated into retrieval systems, content generation platforms, and decision-support workflows, model selection has emerged as an important practical challenge. Prior works (e.g., Brimont et al.  \n* Corresponding Author: [jiechao@stanford.edu](jiechao@stanford.edu)  \nFigure 1: Illustration of the Capability-Reliability Gap in industry VLM applications. Despite achieving benchmark capability above 80%, representative VLMs show up to 23.4% variation in cross-dataset performance stability, resulting in substantially different CDC (M) scores (see §5) .  \n2026) primarily evaluates VLMs through benchmark rankings derived from retrieval, captioning, and reasoning downstream tasks using metrics such as Recall@K, BLEU, CIDEr, SPICE, and task accuracy. Let P(M) denote the benchmark capability of a model M. Under evaluation protocols, models are ranked according to P (M), implicitly assuming that higher capability corresponds to higher reliability. However, models with similar values of P (M) often show substantially different performance distributions across datasets and evaluation conditions. As a result, benchmark rankings may provide limited insight into how consistently a model behaves beyond the evaluation benchmark. We refer to this discrepancy as the Capability-Reliability Gap. Let R(M) denote the reliability of a model under varying evaluation conditions. In practice, the assumption P (Mi) > P (Mj) ⇒ R(Mi) > R (Mj) frequently fails, implying that the model maximizing capability, arg maxM P(M), may not coincide with the model maximizing reliability, arg maxM R(M)(see Figure 1) .  \nTo study this gap, we introduce ARGUS-EVAL, a capability-reliability-oriented evaluation framework for VLMs. Unlike traditional evaluations that focus primarily on Benchmark Capability P(M),  \nFigure 2: Overview of ARGUS-EVAL. Dataset-level results from retrieval, captioning, and reasoning downstream tasks are aggregated into P (M) and subsequent","cbCaifWKxe1z4Uhf","https://ap.wps.com/l/cbCaifWKxe1z4Uhf","pdf",1399827,1,9,"English","en",105,"# Abstract\n# Introduction\n# Related Work","[{\"question\":\"What problem does the paper identify with current VLM benchmark rankings?\",\"answer\":\"It shows that benchmark capability rankings derived from retrieval, captioning, and reasoning do not reliably predict model stability across different datasets and evaluation conditions, leading to a Capability-Reliability Gap.\"},{\"question\":\"What is ARGUS-EVAL and what does it measure?\",\"answer\":\"ARGUS-EVAL is a capability-reliability-oriented evaluation framework that characterizes VLM behavior using Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M).\"},{\"question\":\"How do the reported model rankings differ when prioritizing capability vs reliability?\",\"answer\":\"The results reveal notable differences between capability-oriented and reliability-oriented rankings: the paper reports one model achieving the strongest overall capability while another records the lowest latency and memory footprint, and overall reliability-related measures vary across datasets.\"}]",1784203377,23,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"can-argus-judge-them-all-comparing-vlms-across-domains","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/can-argus-judge-them-all-comparing-vlms-across-domains/85425/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper identify with current VLM benchmark rankings?","Question",{"text":75,"@type":76},"It shows that benchmark capability rankings derived from retrieval, captioning, and reasoning do not reliably predict model stability across different datasets and evaluation conditions, leading to a Capability-Reliability Gap.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is ARGUS-EVAL and what does it measure?",{"text":80,"@type":76},"ARGUS-EVAL is a capability-reliability-oriented evaluation framework that characterizes VLM behavior using Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M).",{"name":82,"@type":73,"acceptedAnswer":83},"How do the reported model rankings differ when prioritizing capability vs reliability?",{"text":84,"@type":76},"The results reveal notable differences between capability-oriented and reliability-oriented rankings: the paper reports one model achieving the strongest overall capability while another records the lowest latency and memory footprint, and overall reliability-related measures vary across datasets.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]