[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84355-en":3,"doc-seo-84355-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84355,1099514068365,"Aurelia","https://ap-avatar.wpscdn.com/avatar/10000253d8d9f28188e?_k=1776742907772140068",8,"Research & Report","Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models","Modern AI models show strong performance on many established benchmarks, yet they still fail at tasks humans find nearly trivial, such as string manipulation or drawing simple images. Blind-spots-bench is introduced to expose these persistent blind spots using a set of 235 curated questions. Questions collected from an AI course are cleaned, annotated with structured reference solutions, and organized with a task taxonomy. An automated grading pipeline evaluates diverse models across language, vision-language, and image-generation, revealing notable gaps between closed-source and open-weight systems.","arXiv :2607 .083 17v 1 [ cs .AI] 9 Jul 2026  \nBlind-Spots-Bench: Evaluating Blind Spots in Multimodal Models  \nMatteo Santelmo∗ Xiuying Wei∗ Israa Fakih∗ Felix Bauer∗ Juan Garcia Giraldo∗ Chengkun Li∗ Etienne Bamas† Emmanuel Abb†  \n´Ecole Polytechnique Fdrale de Lausanne (EPFL), Switzerland  \n blind-spots-bench  eval-pipeline  \nAbstract  \nModern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs. These examples suggest that existing benchmarks may under-measure persistent blind spots in current systems. We introduce blind-spots-bench, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI. We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples. We further develop an automated grading pipeline to evaluate a wide range of models, including open-weight and closedsource language, vision-language, and image-generation models. Our analysis on blind-spots-bench reveals that closed-source frontier models can substantially outperform open-weight models with even ≈ 10% gap, even when they attain comparable performance on existing benchmarks. A more fine-grained analysis shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models. These results highlight the value of blind-spots-bench as a diagnostic stress test for identifying concrete weaknesses in current modern models.  \nFigure 1: Left: Accuracy on blind-spots-bench vs. Artificial Analysis Intelligence Index score for text-only problems. Right: Performance of four VLM models on several sub-tasks.  \n∗ Equal contribution. Correspondence to \u003C[matteo.santelmo@epfl.ch](matteo.santelmo@epfl.ch)>, \u003C[xiuying.wei@epfl.ch](xiuying.wei@epfl.ch)>. Matteo led the evaluation and Xiuying led dataset cleaning and labeling. Dagger† denotes advising roles.  \nPreprint.  \n1 Introduction  \nThe rapid ascent of large language models (LLMs) and multimodal models has been marked by strong performance across a wide range of tasks, including mathematics, coding, reasoning, and vision. Recent frontier models [1–3] have achieved impressive results, often nearly saturating several carefully designed benchmarks [4–8], in some cases rivaling or surpassing human experts. Similar progress has also been observed on vision-language benchmarks, where modern systems demonstrate strong capabilities [9–11] .  \nIn practice, however, these models still exhibit surprising failures on tasks that are straightforward for humans [12–16] . For instance, a model may struggle to generate a string of a specified length [17], produce an image of a cat with exactly four eyes [18, 19], draw a clock showing a requested time [20], or solve a relatively simple Sudoku puzzle [21] . These failures point to persistent blind spots across a range of abilities like spatial reasoning [22], logical consistency [23], and character-level manipulation [17] .  \nTo systematically study which questions remain easy for humans yet difficult for today’s state-ofthe-art models, we introduce blind-spots-bench, a benchmark of 235 manually curated questions across multiple input-output format, accompanied by a reproducible evaluation framework. The questions are collected from students in a graduate-level AI course, who were asked in October 2025 to propose problems frontier AI chatbots failed to answer correctly. Each problem is annotated with a structured reference solution, question format, and task label. To support fine-grained analysis, we introduce a task taxonomy with three broad categories and finer-grained subtask labels: objectcentric tasks, such as counting, recognition, and spatial reasoning; abstract reasoning tasks, mainly involving m","cbCailfgPqyfOLSx","https://ap.wps.com/l/cbCailfgPqyfOLSx","pdf",5183496,1,25,"English","en",105,"# Introduction\n## Benchmark design and dataset construction\n## Evaluation pipeline and model coverage\n## Results and analysis","[{\"question\":\"What problem does blind-spots-bench address in modern AI systems?\",\"answer\":\"It targets persistent weaknesses where contemporary multimodal models fail at tasks that humans consider almost trivial. The benchmark is designed to reveal these blind spots that common benchmarks may under-measure.\"},{\"question\":\"How is the blind-spots-bench dataset created and organized?\",\"answer\":\"Raw questions are collected from students in a graduate AI course, then cleaned and annotated with structured reference solutions. A task taxonomy groups problems into object-centric, abstract reasoning, and language-and-knowledge categories, with finer-grained subtask labels.\"},{\"question\":\"What does the evaluation show about different model types and task categories?\",\"answer\":\"Closed-source frontier models achieve substantially higher accuracy than open-weight models by about a 10% gap even when comparable on existing benchmarks. However, no single model dominates across all task types, and some fine-grained visual perception tasks remain challenging for every evaluated model.\"}]",1784195042,63,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"blind-spots-bench-evaluating-blind-spots-in-multimodal-models","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/blind-spots-bench-evaluating-blind-spots-in-multimodal-models/84355/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does blind-spots-bench address in modern AI systems?","Question",{"text":74,"@type":75},"It targets persistent weaknesses where contemporary multimodal models fail at tasks that humans consider almost trivial. The benchmark is designed to reveal these blind spots that common benchmarks may under-measure.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How is the blind-spots-bench dataset created and organized?",{"text":79,"@type":75},"Raw questions are collected from students in a graduate AI course, then cleaned and annotated with structured reference solutions. A task taxonomy groups problems into object-centric, abstract reasoning, and language-and-knowledge categories, with finer-grained subtask labels.",{"name":81,"@type":72,"acceptedAnswer":82},"What does the evaluation show about different model types and task categories?",{"text":83,"@type":75},"Closed-source frontier models achieve substantially higher accuracy than open-weight models by about a 10% gap even when comparable on existing benchmarks. However, no single model dominates across all task types, and some fine-grained visual perception tasks remain challenging for every evaluated model.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]