[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-273633-105":59,"doc-detail-273633-en":123},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":116,"head_meta":118,"extra_data":120,"updated_unix":122},105,"en","when-irrelevant-text-matters-affine-margin-shifts-in-multimodal-large-language-models","When Irrelevant Text Matters - Affine Margin Shifts in Multimodal Large Language Models","","Multimodal large language models (MLLMs) are exposed to auxiliary text, yet its effect on vision-language decisions is often unclear. This paper treats task-irrelevant context as a controlled intervention in a binary visual judgment setting, using paired comparisons with and without auxiliary text across multiple benchmarks. Irrelevant text consistently biases predictions toward negative responses, captured through a decision-margin analysis. Context-conditioned margins match an affine transform of context-free margins, enabling interpretable diagnostics of preference distortion. Code is released.",{"@graph":69,"@context":115},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/when-irrelevant-text-matters-affine-margin-shifts-in-multimodal-large-language-models/273633/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/when-irrelevant-text-matters-affine-margin-shifts-in-multimodal-large-language-models/273633.png","ImageObject",300,407,{"name":92,"@type":93},"Levi","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-28","2026-09-15",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",12,{"@type":107,"mainEntity":108},"FAQPage",[109],{"name":110,"@type":111,"acceptedAnswer":112},"What is the main finding about the relationship between margins with and without context?","Question",{"text":113,"@type":114},"Context-conditioned margins are often well approximated by an affine transformation of context-free margins, implying estimable distortion of model preference rather than random noise.","Answer","https://schema.org",{"og:url":83,"og:type":117,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":119,"canonical":83},"index,follow",{"doc_id":121,"site_id":62},273633,1789472327,{"code":4,"msg":5,"data":124},{"doc_id":121,"user_id":125,"nickname":92,"user_avatar":126,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":127,"file_id":128,"file_url":129,"file_type":130,"file_size":131,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":132,"language":133,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":134,"faqs":135,"seo_title":136,"seo_description":67,"update_tm":122,"read_time":137,"preview_page_count":4},7971461740909,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large  \nLanguage Models  \nYinfeng Wang 1 , Zhiyuan Yao 1 , Zheren Fu 1 , Lei Zhang 1 , Zhendong Mao 1 ,2 , *  \n1University of Science and Technology of China  \n2Institute of Artificial Intelligence, Hefei Comprehensive National Science Center  \n{wyf666, [yaozhiyuan}@mail.ustc.edu.cn](yaozhiyuan}@mail.ustc.edu.cn)  \n{fzr,leizh23,[zdmao}@ustc.edu.cn](zdmao}@ustc.edu.cn)  \narXiv : 2608 . 19208v2 [cs .CL] 1 Sep 2026  \nAbstract  \nMultimodal large language models (MLLMs) are frequently exposed to auxiliary textual context, the impact of which on vision-language tasks remains underexplored. In this paper, we investigate the influence of task-irrelevant context by formulating it as a controlled intervention within a binary visual judgment framework. Through controlled paired comparisons with and without auxiliary textual context, we observe that irrelevant text consistently biases model predictions across diverse benchmarks. To move beyond performance metrics, we characterize this sensitivity through \"decision margin\", which is defined by the log-probability difference between candidates. Our analysis reveals an empirical pattern: context-conditioned margins are often well approximated by an affine transformation of their context-free counterparts. We further interpret the fitted affine parameters as metrics for visual commitment preservation and directional answer bias. These findings provide a margin-level diagnostic view of irrelevant-context effects in MLLMs, indicating that irrelevant context does not manifest as stochastic noise, but as an estimable distortion of model preference. We release all the code in [https://github.com/Wangyf1998/](https://github.com/Wangyf1998/)[ ](https://github.com/Wangyf1998/)Irrelevant_Text_Matters.  \n1 Introduction  \nMultimodal large language models (MLLMs) have demonstrated strong cross-modal understanding capabilities, enabling complicated vision-language tasks such as visual question answering, multiturn reasoning, and multimodal agents. In these scenarios, the input image is often accompanied by auxiliary textual context, such as retrieved passages, user-provided side information, or history dialogue. Such context is not always aligned with the current task: retrieved passages may be offtopic (Mortaheb et al., 2025), while instructions  \n* Corresponding author.  \n(a) Controlled Intervention Experiment (b) Model Performance  \nFigure 1: Overview of the controlled intervention and its behavioral effects. (a) For each image-question pair, we compare a context-free input with a paired contextconditioned input, where an irrelevant context is inserted. (b) Across multiple benchmarks, irrelevant context not only reduces predictive accuracy, but also shifts model decisions toward negative responses.  \nand dialogue may also contain irrelevant information (Wang et al., 2024b ; Park et al., 2024) . This raises a question: Does irrelevant context affect MLLM predictions on vision-language tasks?  \nOur question is motivated by a broader observation in NLP community: large language models (LLMs) can be sensitive to textual context that isnot part of the task-relevant evidence. Prior studies have shown that irrelevant or distracting context can mislead LLMs in question answering and reasoning tasks (Jia and Liang, 2017 ; Shi et al., 2023 ; Amiraz et al., 2025) . Similar issues arise in retrieval-augmented generation, where retrieved passages containing irrelevant information will affect the model’s performance (Yoran et al., 2024 ; Yan et al., 2024 ; Cuconasu et al., 2024) . In MLLMs, recent work shows that they may over-rely on additional textual context and resulting in a strong bias (Deng et al., 2025 ; Chen et al., 2024b ; Zhang et al., 2024) . However, despite these observations,  \nContext-Free Context-Condition  \nlogP logP  \nm0  \n Negative Region  Positive Region   \n-6 -4 -2 0 shift 2 mc 4 6  \n Negative Region   Flip Region  Positive Region","cbCaiu9dCsWtC4YC","https://ap.wps.com/l/cbCaiu9dCsWtC4YC","pdf",1481805,23,"English","# Abstract\n# Introduction\n## Controlled intervention setup\n## Decision margin and affine relation\n## Implications for decision preference","[{\"question\":\"What is the main finding about the relationship between margins with and without context?\",\"answer\":\"Context-conditioned margins are often well approximated by an affine transformation of context-free margins, implying estimable distortion of model preference rather than random noise.\"}]","When Irrelevant Text Matters - Affine Margin Shifts in Multimodal Large Language Models | PDF",58]