Vocabulary Scaling Law - Tuning Open-vocabulary Predictors for Their Openness

Sep 28, 2026

Upload by Adam

Open-vocabulary learning on CLIP generalizes across diverse concepts, yet it underperforms in realistic streaming open-world evaluations, especially stability against distractor classes and extensibility to unseen novel classes. The paper introduces a “vocabulary scaling law” that bounds openness measures by performance on the full class-name universe, motivating robust fine-tuning. It proposes Submodular-Vocabulary Fine-tuning (SVFT), a bi-level method that greedily selects an informative subset of class-name embeddings and tunes them with an orthogonality constraint, improving both stability and extensibility through extensive experiments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Open-vocabulary CLIP and deployment gap

Stability and extensibility metrics

Vocabulary scaling law and key takeaways

Proposed approach: SVFT

Submodular subset selection and bi-level optimization

This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. Vocabulary Scaling Law : Tuning Open-vocabulary Predictors for Their Openness Ziliang Chen 1 , Yulu Li2 , Liangda Fang2 , Jusheng Zhang3 *, Yongsen Zheng4 , Quanlong Guan2 , Xipeng Chen 1 1Research Institute of Multiple Agents and Embodied Intelligence, Peng Cheng Laboratory, 2Jinan University, 3 Sun Yat-sen University, 4NTU Abstract Open-vocabulary learning on CLIP provides remarkable generalization on diverse concepts, however, falters under the realistic streaming open-world evaluations for Stability against distractor classes and Extensibility to novel classes. Current fine-tuning methods often fail these tests since they are mainly designed for closed-set conditions, leading to the performance gaps while the target vocabulary progressively scales. We formalize a “vocabulary scaling law” showing that these openness measures can be lower-bounded by performance on the full class-name universe, implying that robust fine-tuning should: (i) account for the entire vocabulary, (ii) tune class-name embeddings rather than context, and (iii) enforce orthogonality between prompt embeddings including training and open-set class names. Guided by our analysis, we propose Submodular-Vocabulary Fine-tuning (SVFT), a bi-level optimization framework that approximates the intractable objective of tuning all class name embedding by greedily selecting a small, informative subset of class names via constrained submodular maximization, thus, allows the employment of efficient greedy algorithm for the near-optimal class-name subset selection to finetune CLIP instead of using all open classes. Across extensive experiments, SVFT consistently improves both stability and extensibility, advancing the openness and practical robustness of CLIP-based vision–language models. 1. Introduction Vision-Language Models (VLMs) such as CLIP [9, 21] have marked a paradigm shift in visual recognition, leveraging natural language supervision from massive imagetext datasets to enable incredible few-shot and even zeroshot inference [34, 35] . Their hallmark capability, openvocabulary prediction, allows for the classification of images using arbitrary, user-defined category names, breaking free from the constraints of predefined, fixed-class datasets. This flexibility has positioned VLMs as a basic technology *indicate corresponding author; Figure 1 . The comparison between fine-tuning for openness and the other task setups for CLIP from the aspects of whether there are new classes join the training and evaluation, as well as whether the training and evaluation are continually executed. for a wide array of downstream applications that demand generalization to diverse and unforeseen visual concepts. Despite their successes, a critical gap remains between their typical evaluation and the demands of real-world deployment. In particular, most fine-tuning schemes and evaluation protocols for CLIP operate under the unrealistic assumption that the evaluated images are exactly consistent with the classes to construct the open vocabulary. In other words, existing open-vocabulary predictors are mostly evaluated with close-set classes in the stationary setups. In the pursuit of the openness of CLIP (Fig.1), the concepts of stability and extensibility have emerged as more rigorous metrics for re-evaluating CLIP. Stability measures a model’s robustness to maintain accuracy on known classes when the vocabulary is expanded with unseen ”distractor”concepts, while extensibility measures its zero-shot ability to correctly classify close-set and open-set categories along with the same vocabulary expansion (Fig.2) . As shown by [22], the CLIP family and their fine-tuning methods degrade significantly on these metrics while the vocabulary

FAQs

What problems does the paper address in open-vocabulary CLIP evaluation?

It targets failures in streaming open-world settings, focusing on stability against distractor classes and extensibility to novel classes not seen during fine-tuning.

What is the “vocabulary scaling law” introduced in the paper?

It states that openness measures are lower-bounded by performance on the scaled vocabulary containing the full open-set class-name universe, guiding how fine-tuning should be designed.

How does SVFT improve stability and extensibility?

SVFT tunes class-name embeddings via a bi-level optimization that selects a small informative subset using constrained submodular maximization, and enforces orthogonality between prompt embeddings to achieve near-optimal open-vocabulary behavior.

Sep 28, 2026

Upload by Adam