STAR - Test-Time Adaptation Can Enhance Universal Prompt Learning for Vision-Language Models

Sep 15, 2026

Uploaded by Ophelia

This paper studies universal prompt learning at test-time for vision-language models (VLMs), focusing on improving adaptation to noisy unlabeled target data containing both in-distribution (ID) and out-of-distribution (OOD) instances. It addresses limitations of existing methods that depend on unreliable pseudo-labels from weak uncertainty estimation and ignore class-specific diversity, which can introduce adaptation bias. The proposed STAR framework uses separability-aware gating with conjugate optimization and prototypical retrieval from a dynamic memory bank to calibrate OOD detection. Extensive experiments validate consistent gains over baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

1. Introduction

1.1 OOD detection and VLM background

1.2 Challenges in progressive test-time adaptation

1.3 Proposed STAR framework

This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. STAR: Test-Time Adaptation Can Enhance Universal Prompt Learning for Vision-Language Models Yiwei Fu ♣ Hui Wan ♡ Xiao Luo ♢ , * Minghua Deng ♣ , * ♣ School of Mathematical Sciences, Peking University ♡Department of Statistics and Data Science, Yale University ♢Department of Statistics, University of Wisconsin–Madison [fuyw@stu.pku.edu.cn](fuyw@stu.pku.edu.cn) , [hui.wan@yale.edu](hui.wan@yale.edu) , [xiao.luo@wisc.edu](xiao.luo@wisc.edu) , [dengmh@math.pku.edu.cn](dengmh@math.pku.edu.cn) Abstract This paper studies the problem of universal prompt learning at test-time for vision-language models (VLMs), aiming to enhance the adaptation of pre-trained VLMs to noisy unlabeled target data comprising both in-distribution (ID) and our-of-distribution (OOD) instances. However, existing test-time adaptation approaches often rely on unreliable pseudo-labels due to inadequate uncertainty estimation andoverlook class-specific diversity in the target domain, which may result in additional adaptation bias during test time. Motivated by this, we propose a novel framework named Separability-aware Conjugate Optimization with Prototypical Retrieval (STAR) for universal test-time prompt learning of VLMs. The core of our STAR is to incorporate aseparability-aware gating mechanism into conjugate optimization for reliable pseudo-learning with OOD samples. In particular, we first compute the Fisher score to quantify the separability between ID and OOD samples, which guides our soft gating mechanism for divided training. Then, we employ conjugate optimization to derive reliable pseudo-labels of unlabeled data for test-time adaptation. To further mitigate biases in OOD detection, we maintain a dynamic memory bank which stores high-confidence samples to build class-wise prototypes, which would serve as queries for prototypical retrieval to calibrate OOD detection. Extensive experiments on multiple benchmarks demonstrate that STAR consistently outperforms competing baselines. 1. Introduction Recent advances in deep neural networks (DNNs) have driven substantial progress in a wide range of downstream applications [14, 48], such as image classification and retrieval [17, 46] . However, an inherent assumption behind these models is that the training and test data are drawn from *Corresponding authors. Figure 1 . Trend of model performance with the progressive arrival of test data. Without adaptation, prompt learning methods tend to suffer from performance degradation as the number of noisy samples increases, whereas our proposed STAR effectively mitigates this issue by tuning with unlabeled data. an identical distribution, thereby limiting their deployment and applicability in real-world scenarios. Consequently, detecting OOD instances during inference has become a significant and promising research direction. VLMs, as exemplified by CLIP [45], ALIGN [22], and BLIP [28], have opened up new avenues for advancing OOD detection. For instance, CLIP is widely recognized as a pioneering VLM framework that employs joint contrastive learning in a selfsupervised manner to align image-text pairs in a unified embedding space, achieving remarkable performance on classification tasks compared to traditional vision-only models. A substantial body of research has investigated OOD detection in VLMs, falling into two main paradigms: scorebased methods [32, 34, 37, 39, 44] and tuning-based approaches [12, 38] . Score-based methods derive discriminative functions from output logits or latent representations to distinguish ID instances from OOD samples, but their robustness is often limited under domain shifts. In contrast, tuning-based frameworks adapt the model using available training data, frequently incorporating O

FAQs

What problem does STAR target in universal prompt learning for vision-language models?

STAR targets adapting pre-trained VLMs at test time using noisy unlabeled target data that includes both ID and OOD instances, where adaptation quality can degrade due to unreliable pseudo-labeling and distribution shifts.

How does STAR improve pseudo-label reliability during test-time adaptation?

STAR introduces a separability-aware soft gating mechanism into conjugate optimization, using Fisher score–based separability between ID and OOD samples to guide training and reduce bias from detrimental noise.

How does STAR mitigate biases in OOD detection?

STAR maintains a dynamic memory bank of high-confidence samples to build class-wise prototypes, which are used as queries for prototypical retrieval to calibrate OOD detection during adaptation.

Sep 15, 2026

Uploaded by Ophelia

STAR - Test-Time Adaptation Can Enhance Universal Prompt Learning for Vision-Language Models_Page 1

Open in WPS to view all 11 pages