[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-244288-105":3,"detail-sidebar-cat-1-en-105":81,"doc-detail-244288-en":126},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","flowlearn-evaluating-large-vision-language-models-on-flowchart-understanding-research-paper","FlowLearn - Evaluating Large Vision-Language Models on Flowchart Understanding - research paper","","Flowcharts are concise visual representations used to communicate complex processes, yet their comprehension by large vision-language models (LVLMs) remains insufficiently studied. The paper presents the FlowLearn dataset, including 3,858 scientific flowcharts and 10,000 simulated flowcharts with rich annotations for visual components, OCR, Mermaid code, and VQA pairs. A FlowLearn test set evaluates state-of-the-art LVLMs, revealing task-specific limitations and no single model excelling across all flowchart understanding benchmarks.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/template/","Template",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/template/presentations/","Presentations",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/template/flowlearn-evaluating-large-vision-language-models-on-flowchart-understanding-research-paper/244288/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/flowlearn-evaluating-large-vision-language-models-on-flowchart-understanding-research-paper/244288.png","ImageObject",442,249,{"name":42,"@type":43},"Margaret","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-21","2026-09-12",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",14,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is the FlowLearn dataset designed to do?","Question",{"text":63,"@type":64},"FlowLearn is tailored to improve understanding of flowcharts by large vision-language models. It provides scientific and simulated flowcharts with detailed annotations and a dedicated test set.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What kinds of annotations does FlowLearn include?",{"text":68,"@type":64},"FlowLearn includes annotations for visual components, OCR, Mermaid code representation, and visual question-answering (VQA) pairs. These support both general and component-specific evaluation.",{"name":70,"@type":61,"acceptedAnswer":71},"How do different LVLMs perform on FlowLearn tasks?",{"text":72,"@type":64},"Performance varies by task: GPT-4V is highest for counting nodes on simulated flowcharts, while Claude is highest for OCR tasks. No single model achieves the best results across all tasks in the framework.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},244288,1791362727,{"code":4,"msg":82,"data":83},"success",[84,88,93,97,102,107,112,117,122],{"id":85,"doc_module":22,"doc_module_name":25,"category_name":29,"show_sort_weight":86,"slug":87},11,90,"presentations",{"id":89,"doc_module":22,"doc_module_name":25,"category_name":90,"show_sort_weight":91,"slug":92},12,"Resumes",80,"resumes",{"id":55,"doc_module":22,"doc_module_name":25,"category_name":94,"show_sort_weight":95,"slug":96},"Invoices",70,"invoices",{"id":98,"doc_module":22,"doc_module_name":25,"category_name":99,"show_sort_weight":100,"slug":101},15,"Posters",60,"posters",{"id":103,"doc_module":22,"doc_module_name":25,"category_name":104,"show_sort_weight":105,"slug":106},16,"Social Media",50,"social-media",{"id":108,"doc_module":22,"doc_module_name":25,"category_name":109,"show_sort_weight":110,"slug":111},17,"Forms",40,"forms",{"id":113,"doc_module":22,"doc_module_name":25,"category_name":114,"show_sort_weight":115,"slug":116},18,"Letters",30,"letters",{"id":118,"doc_module":22,"doc_module_name":25,"category_name":119,"show_sort_weight":120,"slug":121},21,"Paper Templates",5,"papers-templates",{"id":123,"doc_module":22,"doc_module_name":25,"category_name":124,"show_sort_weight":4,"slug":125},158,"General","general-158",{"code":4,"msg":82,"data":127},{"doc_id":79,"user_id":128,"nickname":42,"user_avatar":129,"doc_module":22,"category_id":85,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":135,"language":136,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":12,"update_tm":140,"read_time":30},137451207643,"https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2","ECAI 2024  \nU. Endriss et al. (Eds.)  \n© 2024 The Authors.  \nThis article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0). doi:10.3233/FAIA240473  \n73  \nFlowLearn: Evaluating Large Vision-Language Models  \non Flowchart Understanding  \nHuitong Pana, * , Qi Zhanga, Cornelia Carageab, Eduard Draguta and Longin Jan Lateckia  \naDept. of Computer and Information Sciences, Temple University, Philadelphia b Dept. of Computer Science, University of Illinois Chicago  \nAbstract. Flowcharts are graphical tools for representing complex concepts in concise visual representations. This paper introduces the FlowLearn dataset, a resource tailored to enhance the understanding of flowcharts. FlowLearn contains complex scientific flowcharts and simulated flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature and the simulated subset contains 10,000 flowcharts created using a customizable script. The dataset is enriched with annotations for visual components, OCR, Mermaid code representation, and VQA question-answer pairs. Despite the proven capabilities of Large Vision-Language Models (LVLMs) in various visual understanding tasks, their effectiveness in decoding flowcharts—a crucial element of scientific communication—has yet to be thoroughly investigated. The FlowLearn test set is crafted to assess the performance of LVLMs in flowchart comprehension. Our study thoroughly evaluates state-of-the-art LVLMs, identifying existing limitations and establishing a foundation for future enhancements in this relatively underexplored domain. For instance, in tasks involving simulated flowcharts, GPT-4V achieved the highest accuracy (58%) in counting the number of nodes, while Claude recorded the highest accuracy (83%) in OCR tasks. Notably, no single model excels in all tasks within the FlowLearn framework, highlighting significant opportunities for further development.  \n1 Introduction  \nFlowcharts are visual tools that simplify complex processes and concepts across various domains, condensing intricate information into concise visual representations that enhance both comprehension and communication. In this paper, a flowchart is defined as a diagram that outlines a sequence of operations using standardized symbols like rectangles for steps and arrows to indicate process flow, as demonstrated in Figure 1 .  \nFlowchart comprehension, particularly in the domain of computer vision and LVLMs, remains underexplored despite their extensive application. Resources like ACL-Fig [12] that include scientific flowcharts are limited and often provide only basic figure captions and sparse inline reference annotations. Flowcharts’ complex nature—requiring text recognition, identification of various visual elements (e.g., boxes, nodes, symbols), and understanding of node connections—demands more comprehensive annotations for effective evaluation, emphasizing the need for specialized resources.  \nFurther underscoring the inadequacy of current resources, our preliminary analysis of 208 flowcharts from ACL-Fig using Gemini-  \n∗ [Corresponding Author. Email: huitong.pan@temple.edu](Corresponding Author. Email: huitong.pan@temple.edu)  \nPro-Vision [8] yielded a low BLEU score of 0.006 (refer to supplementary material [21] for details) . However, this score doesn’t fully represent the model’s comprehension capabilities. The captions in ACL-Fig, sometimes as brief as ‘Figure 2: Alignment learning algorithm’, do not provide robust ground truths for meaningful evaluation. With a median caption length of just nine words, ACL-Fig is inadequate for evaluating flowchart understanding.  \nAddressing these gaps, we introduce the FlowLearn Dataset1 , which includes both scientific and simulated flowcharts. The scientific subset features 3,858 flowcharts sourced from scientific literature, annotated with captions (median length of 25 words) and infigure text. The simu","cbCaiqd3XQEdVIqM","https://ap.wps.com/l/cbCaiqd3XQEdVIqM","pdf",531523,8,"English","# Introduction\n## Flowchart comprehension and challenges\n## Related works","[{\"question\":\"What is the FlowLearn dataset designed to do?\",\"answer\":\"FlowLearn is tailored to improve understanding of flowcharts by large vision-language models. It provides scientific and simulated flowcharts with detailed annotations and a dedicated test set.\"},{\"question\":\"What kinds of annotations does FlowLearn include?\",\"answer\":\"FlowLearn includes annotations for visual components, OCR, Mermaid code representation, and visual question-answering (VQA) pairs. These support both general and component-specific evaluation.\"},{\"question\":\"How do different LVLMs perform on FlowLearn tasks?\",\"answer\":\"Performance varies by task: GPT-4V is highest for counting nodes on simulated flowcharts, while Claude is highest for OCR tasks. No single model achieves the best results across all tasks in the framework.\"}]","FlowLearn - Evaluating Large Vision-Language Models on Flowchart Understanding - research paper | PDF",1789215167]