[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-296958-105":59,"doc-detail-296958-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","tug-of-war-no-more-harmonizing-accuracy-and-robustness-in-vision-language-models-via-stability-aware-task-vector-merging","TUG-OF-WAR NO MORE - HARMONIZING ACCURACY AND ROBUSTNESS IN VISION-LANGUAGE MODELS VIA STABILITY-AWARE TASK VECTOR MERGING","","Foundation Vision-Language Models (VLMs) deliver strong benchmark performance yet remain highly vulnerable to adversarial attacks. Traditional adversarial fine-tuning can improve robustness but often needs extensive hyperparameter search and multiple retraining runs to balance clean and robust accuracy. Task-vector merging offers a cheaper alternative, but naïve merging yields a near-linear trade-off because it treats all parameters equally and overlooks cross-objective conflicts. This work introduces a prediction stability-aware framework that estimates per-parameter stability from gradients under both objectives and uses complementary masks—refined along adversarial trajectories and guided by prediction sensitivity—to suppress conflicting coordinates and promote flatter minima. Extensive experiments show superior clean–robust trade-offs and effective transfer to downstream tasks.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/tug-of-war-no-more-harmonizing-accuracy-and-robustness-in-vision-language-models-via-stability-aware-task-vector-merging/296958/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/tug-of-war-no-more-harmonizing-accuracy-and-robustness-in-vision-language-models-via-stability-aware-task-vector-merging/296958.png","ImageObject",300,407,{"name":92,"@type":93},"Adam","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-27","2026-09-18",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",16,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"Why does naïve task-vector merging fail to achieve a good clean–robust trade-off?","Question",{"text":113,"@type":114},"Naïve merging produces a near-linear trade-off because it equally weights all parameter coordinates and does not distinguish parameters that help both objectives from those that introduce conflicts.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What is the core idea of the proposed stability-aware merging framework?",{"text":118,"@type":114},"The method uses prediction stability as a proxy for cross-objective compatibility, estimating per-parameter stability from gradients under both natural and robust objectives to build complementary masks that retain jointly stable parameters while down-weighting unstable ones.",{"name":120,"@type":111,"acceptedAnswer":121},"How does the framework improve the merged result beyond using gradient-based masks alone?",{"text":122,"@type":114},"It refines the masks along adversarial parameter trajectories, using steps weighted by a prediction-sensitivity index, and its theoretical analysis links the mask contraction to reduced first-order cross-objective interference.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},296958,1789739179,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":145,"preview_page_count":4},1374404737137,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","TUG-OF-WAR NO MORE: HARMONIZING ACCURACY AND ROBUSTNESS IN VISION-LANGUAGE MODELS VIA STABILITY-AWARE TASK VECTOR MERGING  \nJunhao Dong1,2 , Xinghua Qu3 , Cong Zhang3 , Qi Rong Sua1 , Nguyen Duc Thai1 Wenbo Pan4 , Xinfeng Li15 , Piotr Koniusz6,7 , Yew-Soon Ong1,2∗  \n1Nanyang Technological University, 2 Centre for Frontier AI Research, IHPC, A*STAR,  \n3Bytedance,4 City University of Hong Kong, 5University of Sydney,  \n6University of New South Wales, 7Data61CSIRO [junhao003@ntu.edu.sg](junhao003@ntu.edu.sg)  \nABSTRACT  \nFoundation Vision-Language Models (VLMs) excel across benchmarks yet remain vulnerable to adversarial attacks. While adversarial fine-tuning improves robustness, attaining a desirable clean–robust performance trade-off typically requires costly hyperparameter searches with multiple retraining runs. A promising alternative is to merge task vectors (i.e., parameter displacements from pretrained models) to balance accuracy and robustness without retraining. However, we find that naive task-vector merging produces a near-linear trade-off, as it equally weights all coordinates and fails to distinguish weights that aid both objectives from those that create conflicts. To overcome this limitation, we propose a prediction stability-aware merging framework that composes task vectors from off-the-shelf naturally and robustly fine-tuned VLMs. Our key insight is that prediction stability serves as a proxy for cross-objective compatibility, enabling us to favor perturbation-invariant parameters while attenuating those with highcross-objective impact. Specifically, we estimate per-parameter stability from gradients under both objectives, building complementary masks that retain jointly stable coordinates while suppressing counterpart-sensitive ones. We further refine these masks along adversarial parameter trajectories, with steps weighted by a prediction-sensitivity index. Our theoretical analysis shows that the masks provably contract first-order cross-objective interference, and the prediction criticality index tracks curvature, biasing the merge toward flatter minima and better generalization. Extensive experiments across benchmarks and scenarios demonstrate our method consistently achieves superior clean–robust trade-offs over prior approaches, with the learned balance transferring effectively to downstream tasks.  \n1 INTRODUCTION  \nDespite redefining multimodal learning across diverse tasks, foundation Vision-Language Models (VLMs) like CLIP (Radford et al., 2021) remain alarmingly vulnerable under adversarial attacks (Zhang et al., 2022 ; Zhao et al., 2023) . Even subtle input perturbations can trigger huge performance drops, undermining their reliability in practice and posing severe security risks (Huang et al., 2025) .  \nBridging the gap between natural performance and robustness is thus essential for the safe and widespread deployment of VLMs. Previous efforts primarily focused on adversarial fine-tuning, where adversarial examples are adaptively integrated into training to enhance robustness (Mao et al., 2023 ; Schlarmann et al., 2024 ; Dong et al., 2025b ;c;e) . However, extensive empirical evidence indicates that even increasingly larger and advanced multimodal architectures continue to suffer from a persistent accuracy-robustness trade-off (Wang et al., 2024) . Rather than resolving this fundamental tension, most existing approaches rely on exhaustive hyperparameter searches and costly retraining to find acceptable compromises, limiting the scalability and efficiency of robust VLM solutions.  \n∗ Corresponding authors: Xinfeng Li ([lxfmakeit@gmail.com](lxfmakeit@gmail.com)), Yew-Soon Ong ([asysong@ntu.edu.sg](asysong@ntu.edu.sg)) .  \nGiven recent progress in parameter-space model merging, which combines fine-tuned models without joint training (Wortsman et al., 2022b ; Ilharco et al., 2023), a compelling question arises: Can model merging extend beyond similar tasks to reconcile the inherent conflict between natu","cbCaitDjryoAXszR","https://ap.wps.com/l/cbCaitDjryoAXszR","pdf",1938613,26,"English","# Abstract\n# Introduction\n## Vulnerability of foundation VLMs under adversarial attacks\n## Limits of fine-tuning and hyperparameter search\n## Stability-aware task vector merging (PISTOLE)","[{\"question\":\"Why does naïve task-vector merging fail to achieve a good clean–robust trade-off?\",\"answer\":\"Naïve merging produces a near-linear trade-off because it equally weights all parameter coordinates and does not distinguish parameters that help both objectives from those that introduce conflicts.\"},{\"question\":\"What is the core idea of the proposed stability-aware merging framework?\",\"answer\":\"The method uses prediction stability as a proxy for cross-objective compatibility, estimating per-parameter stability from gradients under both natural and robust objectives to build complementary masks that retain jointly stable parameters while down-weighting unstable ones.\"},{\"question\":\"How does the framework improve the merged result beyond using gradient-based masks alone?\",\"answer\":\"It refines the masks along adversarial parameter trajectories, using steps weighted by a prediction-sensitivity index, and its theoretical analysis links the mask contraction to reduced first-order cross-objective interference.\"}]","TUG-OF-WAR NO MORE - HARMONIZING ACCURACY AND ROBUSTNESS IN VISION-LANGUAGE MODELS VIA STABILITY-AWARE TASK VECTOR MERGING | PDF",66]