[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82514-en":3,"doc-seo-82514-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82514,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",8,"Research & Report","Selective Test-Time Debiasing for CLIP via Reward Gating","Vision language models (VLMs) deliver strong zero-shot performance but can perpetuate social stereotypes in person-centric queries, producing skewed demographic distributions. Existing debiasing methods apply uniform corrections to all queries, forcing a fairness–utility trade-off: strong debiasing may harm semantically meaningful content in bias-insensitive inputs, while weak debiasing often leaves stereotypes in bias-sensitive ones. RewardGated Test-Time Adaptation (RG-TTA) selectively triggers fairness regularization by input bias sensitivity, improving cross-modal alignment for bias-insensitive cases. Experiments on fairness benchmarks show substantial bias reduction with improved zero-shot utility.","Selective Test-Time Debiasing for CLIP via Reward Gating  \nJaeho Han, Jisoo Yang, Hyeondong Woo, Mingyu Jeon, Sunjae Yoon, Junyeong Kim  \nDepartment of Artificial Intelligence, Chung-Ang University {wogh50, yjs229, hyeondong, smart2557, sunjaeyoon, [junyeongkim}@cau.ac.kr](junyeongkim}@cau.ac.kr)  \narXiv :2607 .00423v 1 [ cs .CL] 1 Jul 2026  \nAbstract  \nVision language models (VLMs) demonstrate strong zero-shot performance, but often perpetuate social stereotypes in person-centric queries, yielding skewed demographic distributions. Current debiasing methods apply uniform bias corrections across all input queries regardless of their bias sensitivity, creating a fundamental fairness–utility trade-off. Strong debiasing distorts semantically meaningful information in bias-insensitive queries, while weak debiasing fails to mitigate stereotypes in biassensitive ones. This one-size-fits-all approach hampers simultaneously achieving high utility on bias-insensitive queries and fairness on bias-sensitive queries. We introduce RewardGated Test-Time Adaptation (RG-TTA), a reinforcement learning-based test-time adaptation framework that selectively applies debiasing based on input sensitivity. RG-TTA adaptively triggers fairness regularization based on the bias sensitivity of each input during testtime policy adaptation, while focusing exclusively on optimizing cross-modal alignment for bias-insensitive inputs. Experiments on fairness benchmarks (e.g., FairFace, UTKFace) demonstrate substantial bias reduction while simultaneously improving zero-shot utility, resolving the trade-off of uniform debiasing.  \n1 Introduction  \nVision Language Models (VLMs) have demonstrated exceptional zero-shot capabilities across a wide range of multimodal tasks (Deng et al., 2009 ; Plummer et al., 2015), reaching the stage of real-world applications. By learning joint representations from web-scale image-text pairs, these models achieve strong cross-modal alignment without task-specific fine-tuning. However, this same training paradigm causes VLMs to internalize social stereotypes (Birhane et al., 2021) present in their training data, leading to biased outputs that reflect and potentially amplify social prejudices (Hall  \n(a) Bias-sensitive and Bias-insensitive Inputs  \nWeak debiasing ~~ ~~ Strong debiasing  \n➔ fairness ‒ utility tradeoff  \n(b) Existing methods: Uniform mitigation  \n50%  50%  \nMan 50% Woman 50%  \nBias Utility  \nWeak debiasing ~~ ~~ Strong debiasing  \n➔ fairness & utility preservation  \n(c) Ours: Selective mitigation via reward-gating for bias-sensitive inputs  \nFigure 1: (a) We categorize inputs into bias-sensitive and bias-insensitive, where only the former requires debiasing intervention. (b) Existing methods apply uniform mitigation, creating a structural trade-off: weak debiasing retains bias in sensitive queries (left), while strong debiasing distorts insensitive queries, degrading utility (right) . (c) Our approach employs selective mitigation via reward-gating, which applies strong debiasing only to bias-sensitive inputs while preserving insensitive ones, ensuring both fairness and utility.  \net al., 2023 ; Hamidieh et al., 2024 ; Janghorbani and De Melo, 2023 ; Zhao et al., 2021 ; Wolfe et al., 2023 ; Hausladen et al., 2025) . These biases manifest most critically in person-centric queries, where models produce skewed demographic distributions. For instance, querying “a photo of a doctor” yields disproportionately male images, or certain occupations become strongly associated with specific racial groups. Such behavior poses serious risk of reinforcing discriminatory decision-making.  \nExisting debiasing approaches for VLMs (Wanget al., 2021b ; Chuang et al., 2023 ; Zhang et al., 2025) share common design philosophy: they apply fixed bias correction uniformly across all lan-  \nFairness  \n0.8  \n0.5  \n0.2  \nRace  \n\n| Ours\u003Cbr>Joint V-L (CVPRʼ25)\u003Cbr> |  |\n| --- | --- |\n| Biased-prompts (CVPRʼ23)\u003Cbr>CLIP-clip |  CLIP (ICMLʼ21) |\n| (EMNLPʼ","cbCaigJ0u5WdWgeF","https://ap.wps.com/l/cbCaigJ0u5WdWgeF","pdf",2850903,1,15,"English","en",105,"# Abstract\n# Introduction\n## Bias-sensitive and Bias-insensitive Inputs\n## Existing methods: Uniform mitigation\n## Ours: Selective mitigation via reward-gating for bias-sensitive inputs\n# Fairness versus utility analysis","[{\"question\":\"What problem does the paper address for CLIP-style vision-language models?\",\"answer\":\"The paper addresses that VLMs can internalize social stereotypes and output biased results for person-centric queries, leading to skewed demographic distributions.\"},{\"question\":\"Why do existing debiasing methods create a fairness–utility trade-off?\",\"answer\":\"They apply uniform bias correction to every query, so strong debiasing can distort useful semantic information in bias-insensitive inputs, while weak debiasing may not sufficiently reduce bias in bias-sensitive inputs.\"},{\"question\":\"How does RG-TTA improve fairness without sacrificing zero-shot utility?\",\"answer\":\"RG-TTA performs reward-gated test-time adaptation by selectively applying fairness regularization based on each input’s bias sensitivity, while focusing optimization on cross-modal alignment for bias-insensitive inputs.\"}]",1784181051,38,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"selective-test-time-debiasing-for-clip-via-reward-gating","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/selective-test-time-debiasing-for-clip-via-reward-gating/82514/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address for CLIP-style vision-language models?","Question",{"text":75,"@type":76},"The paper addresses that VLMs can internalize social stereotypes and output biased results for person-centric queries, leading to skewed demographic distributions.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why do existing debiasing methods create a fairness–utility trade-off?",{"text":80,"@type":76},"They apply uniform bias correction to every query, so strong debiasing can distort useful semantic information in bias-insensitive inputs, while weak debiasing may not sufficiently reduce bias in bias-sensitive inputs.",{"name":82,"@type":73,"acceptedAnswer":83},"How does RG-TTA improve fairness without sacrificing zero-shot utility?",{"text":84,"@type":76},"RG-TTA performs reward-gated test-time adaptation by selectively applying fairness regularization based on each input’s bias sensitivity, while focusing optimization on cross-modal alignment for bias-insensitive inputs.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]