[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83924-en":3,"doc-seo-83924-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83924,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","A Multimodal Reasoning Typology for Grounding Chart-Image Coherence in Science Communication","Charts and images co-occur in scientific publications, yet computational methods often fail to model how their coherence is established. This work treats the chart, associated image, and caption as a single multimodal unit, arguing that the inferential work required to interpret it varies predictably. A reasoning-gap typology R1–R5 is derived bottom-up from 79 traumatic brain injury papers and 32 chart–image pairs, grounded in communication grounding theory. A study shows the levels predict expert–non-expert agreement on vision-language model descriptions, pinpointing where contextual knowledge, not the figure, creates coherence.","© 2026 IEEE. This is the author’s version of the article that has been published in the proceedings of IEEE Visualization conference. The final version of this record is available at: xx.xxxx/TVCG.201x.xxxxxxx/  \nA Multimodal Reasoning Typology for  \nGrounding Chart-Image Coherence in Science Communication  \nAvina Nakarmi*  \nDepartment of Data Science New Jersey Institute of Technology  \nSohom Sen†  \nDepartment of Electrical & Computer Engineering New Jersey Institute of Technology  \nXun Song‡ Department of Computer Science New Jersey Institute of Technology  \nSreyashi Samaddar §  \nDepartment of Biology Brooklyn College  \nAritra Dasgupta¶  \nDepartment of Data Science New Jersey Institute of Technology  \narXiv :2607 .05222v 1 [ cs .CV] 6 Jul 2026  \nABSTRACT  \nCharts and images appear together throughout scientific publications, yet most computational work does not characterize their coherence. We argue that a chart, its accompanying image, and the caption that links them form a multimodal unit, and that the inferential work required to read it varies systematically. To capture this variation, we develop a typology of reasoning gaps, R1 through R5, that characterizes how chart, image, and text jointly convey a scientific claim, and the interpretive work this demands of the reader. Some pairs restate the same data, while in other pairs, charts are used to quantify a structure the image localizes, project image content onto an external variable, audit an image-based claim, or jointly construct a frame that neither panel can establish alone. The typology is anchored in the grounding theory of communication and was derived bottom-up, with a neuroscience expert, from a corpus of 79 traumatic brain injury papers and 32 chart-image pairs. Crucially, the levels provide a systematic mechanism for identifying where grounding succeeds or breaks down, rather than leaving it to subjective inference. We show this in a study in which a domain expert and three non-experts judge vision-language model (VLM) descriptions of 25 pairs: the level predicts where their judgments align and where they diverge, isolating the points at which contextual knowledge, not the figure, carries coherence. This typology thus offers figure designers a systematic way to balance text against chart-image pairs, bridging the expert-to-non-expert divide in reading a scientific takeaway.  \nIndex Terms: Multimodal reasoning, science communication, grounding, neuroscience, chart comprehension  \n1 INTRODUCTION  \nCharts and images often appear side by side in scientific papers. A connectivity matrix might sit next to a node-edge graph and a hierarchy tree of the same network (Figure 2a); a study-design timeline might run alongside a year of pain-rating results (Figure 2e); a brain map might be paired with an ROC curve showing how well a feature separates two groups (Figure 2d) . Often the pairing is the point: readers are meant to recover a claim from the combination that neither panel makes on its own. This setting remains largely unmodeled in computational work. Much chart understanding work extracts data from one chart and answers questions about it [18] . Multi-panel figure work tends to assume the relationships between  \n* e-mail: [an778@njit.edu](an778@njit.edu)[ ](an778@njit.edu)†e-mail: [ss4887@njit.edu](ss4887@njit.edu)[ ](ss4887@njit.edu)‡e-mail: [xs29@njit.edu](xs29@njit.edu)  \n§[e-mail:sreyashi.samaddar@brooklyn.cuny.edu](e-mail:sreyashi.samaddar@brooklyn.cuny.edu)[ ](e-mail:sreyashi.samaddar@brooklyn.cuny.edu)¶ e-mail: [aritra.dasgupta@njit.edu](aritra.dasgupta@njit.edu)  \npanels are fixed or labeled in advance [21, 13] . What goes unmodeled is the reader’s side of the exchange: the inference needed to integrate panels when no question is posed and no relation is given. That inference is our subject, and what we formalize as interpretive effort. We treat the chart, image, and caption as a single multimodal unit for science communication. The distinction between c","cbCaihm7hkH3ovHQ","https://ap.wps.com/l/cbCaihm7hkH3ovHQ","pdf",3207559,1,5,"English","en",105,"# Abstract\n# Introduction\n# Related Work","[{\"question\":\"What problem does the typology address in chart-image-based science communication?\",\"answer\":\"It addresses the lack of computational characterization of how charts and accompanying images become coherent through the caption, capturing the reader’s required inference rather than treating panels as independently interpretable.\"},{\"question\":\"How is the multimodal unit defined in this work?\",\"answer\":\"The work treats the chart, image, and the caption linking them as one multimodal unit, where the chart is read via quantitative encodings and the image is read via depicted structure, depending on the reading task.\"},{\"question\":\"What are the reasoning gaps R1–R5 used for?\",\"answer\":\"R1–R5 classify systematic variations in how chart, image, and text jointly convey a scientific claim, including cases where panels restate data or where they quantify, project, audit, or construct an interpretive frame together.\"}]",1784191464,13,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"a-multimodal-reasoning-typology-for-grounding-chart-image-coherence-in-science-communication","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/a-multimodal-reasoning-typology-for-grounding-chart-image-coherence-in-science-communication/83924/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the typology address in chart-image-based science communication?","Question",{"text":75,"@type":76},"It addresses the lack of computational characterization of how charts and accompanying images become coherent through the caption, capturing the reader’s required inference rather than treating panels as independently interpretable.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is the multimodal unit defined in this work?",{"text":80,"@type":76},"The work treats the chart, image, and the caption linking them as one multimodal unit, where the chart is read via quantitative encodings and the image is read via depicted structure, depending on the reading task.",{"name":82,"@type":73,"acceptedAnswer":83},"What are the reasoning gaps R1–R5 used for?",{"text":84,"@type":76},"R1–R5 classify systematic variations in how chart, image, and text jointly convey a scientific claim, including cases where panels restate data or where they quantify, project, audit, or construct an interpretive frame together.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":21,"slug":137},19,"General","general"]