[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-662982-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-662982-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","aecv-bench-benchmarking-multimodal-models-on-architectural-and-engineering-drawing-understanding","AECV-Bench - Benchmarking Multimodal Models on Architectural and Engineering Drawing Understanding","","AEC drawings encode geometry and semantics through symbols, layout conventions, and dense annotation, yet it remains unclear whether modern multimodal and vision-language models can reliably interpret this graphical language. AECV-Bench evaluates multimodal and vision-language models on realistic AEC artefacts via object counting on 120 floor plans and drawing-grounded document QA with 192 question–answer pairs covering OCR, counting, spatial and comparative reasoning. Results use exact-match accuracy and MAPE for counting, plus overall accuracy with LLM-judge scoring and targeted human adjudication. Models show strong OCR/text QA (up to 0.95), moderate spatial reasoning, and unsolved symbol-centric counting (often 0.40–0.55), indicating document-assistant strengths but limited drawing literacy and motivating domain tools and human-in-the-loop AEC automation.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/aecv-bench-benchmarking-multimodal-models-on-architectural-and-engineering-drawing-understanding/662982/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/aecv-bench-benchmarking-multimodal-models-on-architectural-and-engineering-drawing-understanding/662982.png","ImageObject",300,407,{"name":42,"@type":43},"Asher","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-30","2026-10-28",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",5,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What tasks does AECV-Bench evaluate for AEC drawing understanding?","Question",{"text":63,"@type":64},"It evaluates object counting on floor plans (doors, windows, bedrooms, toilets) and drawing-grounded document QA that tests OCR-based text extraction, instance counting, spatial reasoning, and comparative reasoning across drawing regions.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How is performance measured in the benchmark?",{"text":68,"@type":64},"Object counting uses per-field exact-match accuracy and MAPE. Document-QA uses overall accuracy, per-category breakdowns, an LLM-as-a-judge scoring pipeline, and targeted human adjudication for edge cases.",{"name":70,"@type":61,"acceptedAnswer":71},"What do the results suggest about current multimodal models on architectural drawings?",{"text":72,"@type":64},"OCR and text-centric document QA perform best (up to 0.95 accuracy), spatial reasoning is moderate, but symbol-centric counting—especially doors and windows—remains largely unsolved (often 0.40–0.55 accuracy) with substantial proportional errors.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},662982,1793224108,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,100,105,110,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":97,"show_sort_weight":98,"slug":99},"Comic",60,"comic",{"id":101,"doc_module":4,"doc_module_name":25,"category_name":102,"show_sort_weight":103,"slug":104},6,"Technology",50,"technology",{"id":106,"doc_module":4,"doc_module_name":25,"category_name":107,"show_sort_weight":108,"slug":109},7,"Healthcare",40,"healthcare",{"id":111,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},8,30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":55,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":111,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144},687197207639,"https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd","AECV-Bench: Benchmarking Multimodal Models on Architectural and  \nEngineering Drawing Understanding  \nAuthors: Aleksei Kondratenko, PhD§; Mussie Birhane§; Houssame E. Hsain‡; Guido Maciocci§  \n§ AECFoundry  \n‡ University of New South Wales, Kensington, NSW, Australia  \n [https://github.com/AECFoundry/AECV-Bench](https://github.com/AECFoundry/AECV-Bench)  \nAbstract  \nAEC drawings encode geometry and semantics through symbols, layout conventions, and dense annotation, yet it remains unclear whether modern multimodal and vision-language models can reliably interpret this graphical language. We present AECV-Bench , a benchmark for evaluating multimodal and vision-language models on realistic AEC artefacts via two complementary use cases: (i) object counting on 120 high-quality floor plans (doors, windows, bedrooms, toilets), and (ii) drawing-grounded document QA spanning 192 question–answer pairs that test text extraction (OCR), instance counting, spatial reasoning, and comparative reasoning over common drawing regions. Object-counting performance is reported using per-field exact-match accuracy and MAPE results , while document-QA performance is reported using overall accuracy and percategory breakdowns with an LLM-as-a-judge scoring pipeline and targeted human adjudication for edge cases. Evaluating a broad set of state-of-the-art models under a unified protocol, we observe a stable capability gradient; OCR and text-centric document QA are strongest (up to 0.95 accuracy), spatial reasoning is moderate, and symbol-centric drawing understanding – especially reliable counting of doors and windows – remains unsolved (often 0.40–0.55 accuracy) with substantial proportional errors. These results suggest that current systems function well as document assistants but lack robust drawing literacy, motivating domain-specific representationsand tool-augmented, human-in-the-loop workflows for safe AEC automation.  \n1. Introduction  \nThe Architectural Engineering and Construction (AEC) industry heavily relies on 2D drawings as the primary medium for conveying design intent, coordinating disciplines, and documenting as-built conditions. Architectural and engineering drawings, plans, sections, details, and specifications encode not only geometry, but also rich semantic information in the form of symbols, annotations, tags, and crossreferences to specifications and standards. Reading, cross-checking, and updating this information consumes a large fraction of engineers’ and architects’ time, and errors in interpretation easily propagate directly into clashes, inaccurate estimates, rework, and safety risks.  \nCurrently, multimodal large language models (MMLMs) and vision language models (VLMs) have made impressive progress on generic visual understanding (Yin et al. , 2024; OpenAI, 2023; Gemini Team, 2023; Liu et al. , 2023) . Off-the-shelf APIs can now answer open-ended questions, read texts, extract information from images, scanned PDFs, and summarize complex documents. These advancements have sparked significant interest across the AEC industry; vendors and practitioners are actively experimenting with LLM-  \npowered assistants that promise to extract information (Zheng & Fischer, 2023; Prieto et al. , 2023), count doors and windows (Schönfelder et al. , 2024; Li & Wang, 2024; Zhang & Zhou, 2023), check code compliance (Ying & Sacks, 2024; Chen et al. , 2023), and automatically generate reports from design package (Chen & Fang, 2025; Liao & Zhou, 2024) .  \nHowever, despite this enthusiasm, it remains unclear how well current models really understand AEC drawings. Most widely used benchmarks for multimodal and vision models focus heavily on perceptionheavy tasks: vision question answering, captioning, OCR/document understanding, basic object/scene recognition, and relatively short-horizon reasoning grounded in single images or short clips (Fu et al. , 2024; Mathew et al. , 2021; Mathew et al. , 2022; Van Landeghem et al. , 2023; Masry et al. , 2","cbCainghHkZXumpq","https://ap.wps.com/l/cbCainghHkZXumpq","pdf",1311330,23,"English","# Introduction\n## Motivation and gap in existing benchmarks\n## Benchmark overview and evaluation protocol\n## Contributions","[{\"question\":\"What tasks does AECV-Bench evaluate for AEC drawing understanding?\",\"answer\":\"It evaluates object counting on floor plans (doors, windows, bedrooms, toilets) and drawing-grounded document QA that tests OCR-based text extraction, instance counting, spatial reasoning, and comparative reasoning across drawing regions.\"},{\"question\":\"How is performance measured in the benchmark?\",\"answer\":\"Object counting uses per-field exact-match accuracy and MAPE. Document-QA uses overall accuracy, per-category breakdowns, an LLM-as-a-judge scoring pipeline, and targeted human adjudication for edge cases.\"},{\"question\":\"What do the results suggest about current multimodal models on architectural drawings?\",\"answer\":\"OCR and text-centric document QA perform best (up to 0.95 accuracy), spatial reasoning is moderate, but symbol-centric counting—especially doors and windows—remains largely unsolved (often 0.40–0.55 accuracy) with substantial proportional errors.\"}]","AECV-Bench - Benchmarking Multimodal Models on Architectural and Engineering Drawing Understanding | PDF",58]