[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81539-en":3,"doc-seo-81539-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81539,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",6,"Technology","GRAINS Gradient based Attribution for Inference Time Steering of LLMs and VLMs","Inference-time steering improves LLMs and VLMs by modifying activations during generation without updating weights, avoiding the cost of fine-tuning. Existing steering approaches often use a single global intervention, ignore token-level influence, and underuse logit signals—especially when visual and textual inputs contribute unevenly. GRAINS introduces a contrastive, gradient-based method using Integrated Gradients to select top-k influential tokens and build per-layer directional steering vectors. It achieves stronger results on LLM and VLM tasks, boosting TruthfulQA accuracy, reducing MMHal-Bench hallucinations, and improving SPAVL alignment without harming fluency.","GRAINS: Gradient-based Attribution for  \nInference-Time Steering of LLMs and VLMs  \nDuy Nguyen 1 Archiki Prasad 1 Elias Stengel-Eskin2 Mohit Bansal 1  \n1UNC Chapel Hill 2The University of Texas at Austin  \narXiv :2507 . 18043v2 [ cs .CL] 10 Jul 2026  \nAbstract  \nInference-time steering provides a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying model activations without updating weights. However, existing methods often rely on a global intervention vector, overlook token-level influence, and underutilize model logits, especially in multimodal settings where visual and textual inputs contribute unevenly. We propose GRAINS, a contrastive, gradient-based approach that leverages Integrated Gradients to identify top-k influential tokens and construct directional steering vectors based on their contribution to preferred over dispreferred outputs. These vectors guide activation intervention at each layer, preserving the representational scale. GRAINS outperforms fine-tuning and prior steering methods on both LLM and VLM tasks: improving TruthfulQA accuracy by 13.22%(Llama-3.1-8B), reducing MMHal-Bench hallucinations from 0.624 to 0.514 (LLaVA-1.6-7B), and increasing SPAVL alignment by 8.11%, all without degrading fluency or general capabilities. 1  \n1 Introduction  \nDespite having strong performance across various tasks, LLMs and VLMs often generate undesirable outputs that lack grounding in the input query or context (Rame et al., 2024 ; Shi et al., 2024 ; Huang et al., 2024) . Fine-tuning addresses these issues by adapting models with task-specific datasets, but it requires significant computational resources and data, and risks catastrophic forgetting (Li and Hoiem, 2017 ; Lopez-Paz and Ranzato, 2017) . A promising alternative to fine-tuning is inferencetime steering (Zou et al., 2023 ; Liu et al., 2024c ; Li et al., 2024b ; Rimsky et al., 2024 ; Turner et al., 2024 ; Nguyen et al., 2025a), which adjusts hidden representations during inference without al-  \n1Code:[https://github.com/duykhuongnguyen/](https://github.com/duykhuongnguyen/)[ ](https://github.com/duykhuongnguyen/)GrAInS.  \ntering the model’s parameters. However, existing steering approaches generally rely on linear interventions to hidden states, often applying the same intervention across all tokens’ hidden states (Marks and Tegmark, 2023 ; Li et al., 2024b), ignoring the impact of specific tokens on model behavior. As illustrated in Fig. 1 (top), this can lead to overcorrection and loss of desired capabilities, such as fluency or factual accuracy (Nguyen et al., 2025b) . Moreover, most existing methods construct steering vectors solely from latent space representations of paired data by taking differences between hidden activations corresponding to desirable and undesirable outputs (Li et al., 2024b ; Rimsky et al., 2024 ; Turner et al., 2024 ; Nguyen et al., 2025b), ignoring rich signals from model logits that reveal which specific inputs (tokens) most drive undesirable outputs through their attribution-based contribution to model predictions. In VLMs, this limitation is especially problematic – textual and visual inputs do not contribute equally – some tokens play a key role in shaping the model’s output, while others have little to no influence (Cao et al., 2024 ; Sun et al., 2025 ; Lin et al., 2025) . Thus, constructing steering vectors purely in latent space without identifying which tokens are responsible for undesirable behavior can be ineffective and may cause unintended changes to the model’s behavior (Salinet al., 2022 ; Chen et al., 2024b) .  \nTo address these issues, we propose Gradientbased Attribution for Inference-Time Steering (GRAINS), a more selective and interpretable approach to inference-time steering compatible with both LLMs and VLMs, as outlined in Fig. 1 (bottom) and shown in more detail in Fig. 2. GRAINS identifies specific tokens—whether visual patches or language tokens—that","cbCaiqA6WuBxHq06","https://ap.wps.com/l/cbCaiqA6WuBxHq06","pdf",24625763,1,21,"English","en",105,"# Abstract\n# Introduction\n## Challenges in prior inference-time steering\n## GRAINS: gradient-based attribution and contrastive token selection\n## Attribution-guided intervention for LLMs and VLMs","[{\"question\":\"What problem does inference-time steering address in LLMs and VLMs?\",\"answer\":\"It targets undesirable outputs such as hallucinations or lack of grounding by adjusting hidden activations during generation, without changing model weights.\"},{\"question\":\"How does GRAINS differ from prior steering methods?\",\"answer\":\"GRAINS uses contrastive Integrated Gradients to identify token-level influential inputs (visual patches and language tokens) and constructs directional intervention vectors based on their positive and negative contributions.\"},{\"question\":\"What improvements does GRAINS report on LLM and VLM benchmarks?\",\"answer\":\"The method improves TruthfulQA accuracy (13.22% with Llama-3.1-8B), reduces MMHal-Bench hallucinations (0.624 to 0.514 with LLaVA-1.6-7B), and increases SPAVL alignment by 8.11%, without degrading fluency or general capabilities.\"}]",1784174150,53,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"grains-gradient-based-attribution-for-inference-time-steering-of-llms-and-vlms","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/grains-gradient-based-attribution-for-inference-time-steering-of-llms-and-vlms/81539/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does inference-time steering address in LLMs and VLMs?","Question",{"text":75,"@type":76},"It targets undesirable outputs such as hallucinations or lack of grounding by adjusting hidden activations during generation, without changing model weights.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does GRAINS differ from prior steering methods?",{"text":80,"@type":76},"GRAINS uses contrastive Integrated Gradients to identify token-level influential inputs (visual patches and language tokens) and constructs directional intervention vectors based on their positive and negative contributions.",{"name":82,"@type":73,"acceptedAnswer":83},"What improvements does GRAINS report on LLM and VLM benchmarks?",{"text":84,"@type":76},"The method improves TruthfulQA accuracy (13.22% with Llama-3.1-8B), reduces MMHal-Bench hallucinations (0.624 to 0.514 with LLaVA-1.6-7B), and increases SPAVL alignment by 8.11%, without degrading fluency or general capabilities.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":121,"slug":122},8,"Research & Report",30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]