Criterion-Conditional In-Context Learning - Evaluating Criterion-Shift Adaptation in Vision-Language Models

Sep 21, 2026

Uploaded by Violet

Vision-language models can learn new behaviors through in-context learning (ICL) without parameter updates by inducing tasks from a support set. Standard ICL assumes a fixed decision criterion once the task is induced, while real scenarios keep task intent stable but shift criteria by context. Criterion-Conditional In-Context Learning (CCICL) evaluates latent criterion inference and prediction adjustment under fixed semantics. Two metrics, Criterion Invariance and Criterion Sensitivity, and a multi-domain CC-Bench enable this evaluation, showing many models suffer rigid boundary bias and that multi-criterion training can reduce it and improve adaptability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

AccessFree to download

Abstract

1. Introduction

In-context learning and task induction

Criterion adaptation under fixed task semantics

Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models Kaiyun Yang * 1 Ruilin Yang * 2 Zhimin Yao 2 J. Wang 1 Wei Ge 2 arXiv :2607 .02575v1 [ cs .CV] 30 Jun 2026 Abstract Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the support set for task induction. In the standard ICL setting, once the task is induced, its decision criterion remains fixed. However, in real-world applications, many tasks exhibit a stable high-level intent, while their decision criteria shift according to specific requirements. Thus, we introduce a new setting, denoted as Criterion-Conditional In-Context Learning (CCICL), where models must infer the latent criterion from context and adjust predictions accordingly under fixed task semantics. To evaluate this capability, we propose two complementary metrics, Criterion Invariance and Criterion Sensitivity, capturing the model’s robustness and adaptability under criterion shifts. We further construct CCBench, a multi-domain benchmark that supports evaluation under the CC-ICL setting. By employing a dual-level data hierarchy, CC-Bench enables legitimate ground-truth variation conditioned on the active criterion even when the task remains fixed. Experiments on CC-Bench reveal that most models exhibit a rigid boundary bias, struggling to align their decisions with the latent criterion. We also find that even a simple multi-criterion training strategy can significantly reduce this bias, improving Criterion Sensitivity and enabling 7Bscale models to surpass proprietary models without degrading general multimodal performance. *Equal contribution 1University of Science and Technology of China, Hefei, China 2Megvii Technology Inc., Beijing, China. Correspondence to: Wei Ge <[gewei@megvii.com](gewei@megvii.com) >. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . 1. Introduction In-context learning (ICL) has become a central paradigm for large language models (LLMs) and vision-language models (VLMs), enabling models to perform new tasks at inference time by conditioning on a few-shot support set without parameter updates (Alayrac et al., 2022 ; Awadalla et al., 2023 ; Laurenc¸on et al., 2023 ; Yang et al., 2025 ; Bai et al., 2025b ; Wang et al., 2025) . This inference-time characterization is agnostic to how ICL capability is acquired, as prior work has shown it can be elicited through post-training (Chenet al., 2025b ; Jia et al., 2025 ; Dou et al., 2026) . Yet existing work (Jia et al., 2025 ; Li et al., 2026 ; Baldassini et al., 2024 ; Chen et al., 2025a) predominantly views ICL as a mechanism for task induction: the support set serves to resolve uncertainty about what the task is, assuming that once the task is induced (e.g., an addition task in Figure 1), its decision criterion remains fixed. However, beyond learning new tasks, ICL can also serve another underexplored yet important purpose: criterion adaptation under fixed task semantics. In real-world applications, many tasks exhibit a stable high-level intent, yet their decision criteria are highly situation-dependent, varying with cultural norms, industry standards, or individual preferences. As shown in Figure 1, in Industrial Inspection, the tolerance of surface scratches can vary under different criteria. In such scenarios, few-shot support sets do not define new tasks; instead, they serve to calibrate the model’s decision boundary, clarifying how a pre-existing concept should be judged under the current criterion. Despite the practical importance of criterion adaptation, this capability is largely overlooked by current evaluations of multimodal ICL. We attribute this absence to two main factors: (i) limited evaluation setting, where the common practice of evaluation follows a “single-trial” ICL setting. By

FAQs

What limitation of standard in-context learning motivated the CCICL setting?

Standard ICL treats the decision criterion as fixed after task induction, whereas many real applications keep intent stable but change the decision criterion by context and requirements.

How are criterion shifts evaluated in this work?

The paper proposes two complementary metrics: Criterion Invariance and Criterion Sensitivity to measure robustness and adaptability when the criterion changes.

What is CC-Bench and why is it needed?

CC-Bench is a multi-domain benchmark that supports evaluation under CC-ICL by using a dual-level data hierarchy to provide legitimate ground-truth variation conditioned on the active criterion while the task remains fixed.

Sep 21, 2026

Uploaded by Violet