[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273516-105":3,"doc-detail-273516-en":81,"detail-sidebar-cat-0-en-105":97},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","which-source-wins-task-dependent-reliance-in-vision-language-models-abstract","Which Source Wins? Task-Dependent Reliance in Vision-Language Models - Abstract","","Vision-language models (VLMs) integrate images and text, but when the modalities disagree—especially when one becomes harder to read—the mechanism of how reliance shifts remains unclear. The study evaluates modality reallocation under controlled legibility degradation, building conflicts by combining rendered arithmetic images from one problem with text from another. It further introduces ChartQA-Conflict (229 chart-report conflicts) and tests six open-weight VLMs plus frontier API models. Results show an evaluation- and task-dependent reversal in reliance across benchmarks.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/which-source-wins-task-dependent-reliance-in-vision-language-models-abstract/273516/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/which-source-wins-task-dependent-reliance-in-vision-language-models-abstract/273516.png","ImageObject",300,407,{"name":42,"@type":43},"Ivy","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-05","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",8,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"How do the authors study shifts in reliance between image and text in VLMs?","Question",{"text":63,"@type":64},"They create controlled conflicts by degrading either the image or the text across multiple legibility levels while keeping the other source clean, then track how model preference changes under disagreement.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What datasets are used to build image-text conflicts?",{"text":68,"@type":64},"The work uses GSM8K and SVAMP by rendering one arithmetic problem as an image and pairing it with text from a different problem. It also introduces ChartQA-Conflict, derived from ChartQA, with matched chart and table-image representations.",{"name":70,"@type":61,"acceptedAnswer":71},"What key pattern shows up across the arithmetic benchmarks versus ChartQA-Conflict?",{"text":72,"@type":64},"On GSM8K and SVAMP, five of six open-weight models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models show the opposite pattern, shifting more strongly away from the degraded visual source.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273516,1789471835,{"code":4,"msg":82,"data":83},"success",{"doc_id":79,"user_id":84,"nickname":42,"user_avatar":85,"doc_module":4,"category_id":55,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":86,"file_id":87,"file_url":88,"file_type":89,"file_size":90,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":91,"language":92,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":93,"faqs":94,"seo_title":95,"seo_description":12,"update_tm":80,"read_time":96,"preview_page_count":4},549758252649,"https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819","Which Source Wins? Task-Dependent Reliance in Vision–Language Models  \nRodela Ghosh*  \nUniversity of South Florida [rg21@usf.edu](rg21@usf.edu)  \nAviral Gupta*  \nUniversity of South Florida [aviralgupta@usf.edu](aviralgupta@usf.edu)  \nGuangjing Wang University of South Florida [guangjingwang@usf.edu](guangjingwang@usf.edu)  \narXiv : 2608 . 17205v 1 [cs .CL] 17 Aug 2026  \nAbstract  \nVision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them. We study this modality reallocation with a controlled setup: we degrade either the image or the text across four levels of legibility while keeping the other clean, and track how the model’s preference changes. We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers. We also introduce ChartQA-Conflict, a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations. We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin. On GSM8K and SVAMP, five of six models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models exhibit the opposite pattern, shifting more strongly away from the degraded visual source. This reversal persists after calibrating for unimodal accuracy loss and after replacing charts with plain table images. Two frontier API models, GPT-5.6-Luna and Gemini-3.5-Flash, behaviorally  \nreplicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction. These results show that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings. The source code is available at [https://github.com/Ro-netizen004/](https://github.com/Ro-netizen004/)[ ](https://github.com/Ro-netizen004/)[multimodal-arbitration-artifact](multimodal-arbitration-artifact.)[.](multimodal-arbitration-artifact.)  \n1 Introduction  \nHumans integrate multisensory cues and downweight noisier sources when cues conflict (Ernst and Banks, 2002 ; Ernst and Bülthoff, 2004) .  \n*Equal contribution.  \nÆ Model input  \nYou are given two sources describing a math problem. Source A is the attached image. Source B is the text below. Solve the problem step by step and end with \\#\\#\\#\\# \u003Canswer> .  \në Source A —attached image  \n􀀇 Source B —text  \nA robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?  \nFigure 1: The VLM is prompted to solve a problem based on two independent information sources. Source A is an image, which renders one problem; Source B is the text of a different problem.  \nVision-language models (VLMs) likewise integrate visual and textual inputs, yet it remains unclear whether they appropriately recalibrate their reliance on each modality when one becomes less reliable. Prior work shows that VLMs perform worse when text-based problems are shown in an image rather than provided as plain text (Yuan et al., 2025 ; Xu et al., 2026 ; Liu et al., 2026) . This performance gap has been attributed in part to the additional challenge of extracting information from images before reasoning over the recovered content (Sun et al., 2026) . However, existing evaluations typically assess visual and textual inputs in isolation, which cannot determine which modality a model relies on when both are presented but provide conflicting evidence. In this paper, we ask: when one source becomes harder to read, does a VLM shift its reliance toward the other, and does it shift by the same amount whether we degrade the image or  \nthe text? An asymmetric shift would indicate that modality reweighting depends not only on degradation levels but also on which modality is degraded and how the model is evaluated in a sp","cbCairRUfl5uDL3S","https://ap.wps.com/l/cbCairRUfl5uDL3S","pdf",1602630,20,"English","# Introduction\n## Problem motivation\n## Controlled conflicts and legibility degradation\n## Dataset construction\n## Evaluation setup and findings\n# Summary of contributions","[{\"question\":\"How do the authors study shifts in reliance between image and text in VLMs?\",\"answer\":\"They create controlled conflicts by degrading either the image or the text across multiple legibility levels while keeping the other source clean, then track how model preference changes under disagreement.\"},{\"question\":\"What datasets are used to build image-text conflicts?\",\"answer\":\"The work uses GSM8K and SVAMP by rendering one arithmetic problem as an image and pairing it with text from a different problem. It also introduces ChartQA-Conflict, derived from ChartQA, with matched chart and table-image representations.\"},{\"question\":\"What key pattern shows up across the arithmetic benchmarks versus ChartQA-Conflict?\",\"answer\":\"On GSM8K and SVAMP, five of six open-weight models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models show the opposite pattern, shifting more strongly away from the degraded visual source.\"}]","Which Source Wins? Task-Dependent Reliance in Vision-Language Models - Abstract | PDF",50,{"code":4,"msg":82,"data":98},[99,103,107,111,116,120,125,128,132,135,139],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":100,"show_sort_weight":101,"slug":102},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":104,"show_sort_weight":105,"slug":106},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},"Exam",70,"exam",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":113,"show_sort_weight":114,"slug":115},5,"Comic",60,"comic",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":118,"show_sort_weight":96,"slug":119},6,"Technology","technology",{"id":121,"doc_module":4,"doc_module_name":25,"category_name":122,"show_sort_weight":123,"slug":124},7,"Healthcare",40,"healthcare",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":126,"slug":127},30,"research-report",{"id":129,"doc_module":4,"doc_module_name":25,"category_name":130,"show_sort_weight":91,"slug":131},9,"Religion & Spirituality","religion-spirituality",{"id":91,"doc_module":4,"doc_module_name":25,"category_name":133,"show_sort_weight":91,"slug":134},"World Cup","world-cup",{"id":136,"doc_module":4,"doc_module_name":25,"category_name":137,"show_sort_weight":136,"slug":138},10,"Lifestyle","lifestyle",{"id":140,"doc_module":4,"doc_module_name":25,"category_name":141,"show_sort_weight":112,"slug":142},19,"General","general"]