[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84225-en":3,"doc-seo-84225-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84225,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs","Reliable confidence estimation is a core bottleneck in test-time adaptation for vision–language models (VLMs). Prompt tuning can improve zero-shot accuracy yet often harms calibration because entropy minimization encourages overconfident predictions. Existing methods use LLM-derived attributes and contrastive regularization but assume attributes independently, neglecting relational structure. ARGTCA models (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network with contrastive objectives to learn embeddings that reflect inter-attribute dependencies.","When Prompts Ignore Structure: Graph-Based Attribute Reasoning for  \nCalibrated VLMs  \nTanay Sodha* 1 and Aditya Sharma* 1 and Ramya Hebbalaguppe2 and Vinti Agarwal 1  \nand Pranav Murthy Yeluripaty 1  \n1Department of Computer Science and Information Systems,  \nBirla Institute of Technology and Science, Pilani, India  \n2TCS Research, New Delhi, India  \narXiv :2607 .07395v 1 [ cs .CV] 8 Jul 2026  \nAbstract  \nReliable confidence estimation remains a key limitation of test-time adaptation in vision– language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure.  \nWe propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) via contrastive objectives to produce structurally informed embeddings capturing inter-attribute dependencies.  \nWe introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across 9 benchmarks show that ARGTCA-DIV reduces average ECE(↓) by ∼ 37% over baselines, whereas ARGTCADISC consistently performs as the second-best variant, reducing average ECE by ∼ 17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.  \n1 Introduction  \nVision-language models (VLMs) such as CLIP (Radford et al., 2021) have shown strong zero-shot image recognition by aligning images and text in a shared embedding space through large-scale contrastive pretraining. CLIP performs zero-shot classification of an image by computing cosine similarities between its visual embedding and classconditioned text features generated from prompt templates such as “a photo of a {class}”. While effective, such templates are suboptimal and domainagnostic. Test-time prompt tuning (TPT) (Shu  \nE* qual contribution.  \nTPT (4 .59) C-TPT (4 .24) TCA (2 .56)  \nO-TPT (3 .62) A-TPT (2 .67) Ours (1 .62)  \nFigure 1: Reliability diagrams on Caltech101 dataset (ViT-B/16, ECE(↓) in parentheses) . TCA is overconfident (bars below diagonal); while TPT, C-TPT, OTPT, and A-TPT overcorrect into underconfidence (bars above diagonal) . ARGTCA-DISC (ours) achieves the closest alignment with the calibration diagonal across all confidence bins. Full results across all datasets are provided in Figure 8 in Appendix.  \net al., 2022) addresses this by optimizing prompt tokens for each test image using only its own augmented views, with no labeled data. However, TPTs entropy minimization objective inherently drives the model toward overconfident predictions, producing miscalibrated outputs, quantified by the Expected Calibration Error (ECE) (Guo et al., 2017) . It poses a fundamental barrier to deploying VLMs in safety-sensitive applications such as healthcare diagnostics and autonomous systems, where unreliable uncertainty estimates can have serious consequences.  \nA key insight motivating recent work is that calibration and accuracy are largely decoupled: different prompts can achieve nearly identical top-1 accuracy while exhibiting significantly divergent ECE (Yoon et al., 2024) . This implies that the geometry of the textual feature space, rather than predictive accuracy alone, governs how well-calibrated a  \nVLMs predictions are. C-TPT (Yoon et al., 2024) and O-TPT (Sharifdeen et al., 2025) exploit this by adding a feature-dispersion loss and angular dispersion respectively, that spreads class-conditioned text features across the hypersphere. TCA (Hebbalaguppe et al., 2025) complements these geometric objectives by adding LLM-extracted visual attributes for semantically grounded prompt initialization, combined with a contrastive intra-class and inter-class regulariza","cbCaisQPktZtRVYN","https://ap.wps.com/l/cbCaisQPktZtRVYN","pdf",7706525,1,16,"English","en",105,"# Abstract\n# Introduction\n## Problem: calibration degradation in test-time prompt tuning\n## Structural limitations of TCA\n## Proposed method: ARGTCA with Symbolic Attribute Graph\n## Attribute selection strategies (ARGTCA-DIV, ARGTCA-DISC)\n## Experimental validation across benchmarks","[{\"question\":\"Why does test-time prompt tuning often reduce calibration in VLMs?\",\"answer\":\"Entropy minimization during test-time prompt tuning drives the model toward overconfident predictions, which increases miscalibration measured by Expected Calibration Error (ECE).\"},{\"question\":\"What structural assumptions limit the calibration performance of TCA?\",\"answer\":\"TCA selects attributes as a flat set using similarity without relational context, so attributes can become overly clustered, and shared attributes across classes can lose discriminative power.\"},{\"question\":\"How does ARGTCA improve calibration compared with attribute-independent approaches?\",\"answer\":\"ARGTCA builds a Symbolic Attribute Graph by treating (class, attribute) pairs as nodes and uses a Graph Attention Network trained with contrastive objectives to capture inter-attribute dependencies.\"}]",1784194156,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"when-prompts-ignore-structure-graph-based-attribute-reasoning-for-calibrated-vlms","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/when-prompts-ignore-structure-graph-based-attribute-reasoning-for-calibrated-vlms/84225/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does test-time prompt tuning often reduce calibration in VLMs?","Question",{"text":75,"@type":76},"Entropy minimization during test-time prompt tuning drives the model toward overconfident predictions, which increases miscalibration measured by Expected Calibration Error (ECE).","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What structural assumptions limit the calibration performance of TCA?",{"text":80,"@type":76},"TCA selects attributes as a flat set using similarity without relational context, so attributes can become overly clustered, and shared attributes across classes can lose discriminative power.",{"name":82,"@type":73,"acceptedAnswer":83},"How does ARGTCA improve calibration compared with attribute-independent approaches?",{"text":84,"@type":76},"ARGTCA builds a Symbolic Attribute Graph by treating (class, attribute) pairs as nodes and uses a Graph Attention Network trained with contrastive objectives to capture inter-attribute dependencies.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":28,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]