VocaDet Sample-Driven Open Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

Jul 16, 2026

Upload by Sage

Open-vocabulary object detection and segmentation targets arbitrary objects beyond predefined categories, yet prior vision-language and reference-based methods often depend on text prompts, limited examples, or costly feature matching, hindering scalability to expanding repositories. VocaDet introduces a sample-driven framework that learns object concepts from user-provided positive and negative collections without retraining. Visual representations are quantized into multi-granularity tokens via DINOv3 and clustering, stored in a vector database for efficient retrieval-based localization and segmentation. A background filtering mechanism reduces redundant matches for fixed cameras, and UA-DETRAC results show effective training-free open-vocabulary performance with continually expandable recognition.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Introduction

Method

arXiv :2607 .0854 1v 1 [ cs .CV] 9 Jul 2026 VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval ZhiXin Sun PowerChina Zhongnan Engineering Corporation Limited [sunzxjdi@gmail.com](sunzxjdi@gmail.com) Abstract Open-vocabulary object detection and segmentation aim to recognize arbitrary objects beyond predefined categories. Although recent vision-language and referencebased approaches have significantly advanced this field, they often rely on text prompts, limited visual examples, or expensive feature matching procedures, making them difficult to scale to large and continuously expanding object repositories. In this work, we propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework that learns object concepts directly from user-provided positive and negative sample collections without model retraining. The key idea is to transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database. Specifically, we employ DINOv3 as the visual feature extractor and apply agglomerative clustering with adaptive clustering sensitivity to generate multi-granularity visual tokens. These visual tokens, together with position-debiased representations and spatial topology information, are stored as expandable object memories in a vector database. During inference, query images are converted into visual tokens and efficiently matched against the stored object memories for object localization and segmentation. Furthermore, a background filtering mechanism is introduced to remove frequently occurring background patterns and reduce redundant retrieval operations in practical fixed-camera scenarios. Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training, while supporting continuously expandable recognition capability as additional positive and negative samples are accumulated. Code is available at [https://github.com/sunzx97/VocaDet](https://github.com/sunzx97/VocaDet). 1 Introduction Object detection and segmentation are among the most fundamental and widely applied tasks in computer vision. Owing to its strong real-time performance and mature engineering deployment, the YOLO series[1, 2] has been extensively adopted in industrial applications. However, such methods typically require large-scale data collection and task-specific training for different scenarios, and inevitably suffer from false positives and false negatives. Recently, methods such as SAM3[3], Grounding DINO[4], T-Rex2[5], DINO-X[6], and INSID3[7] have introduced context-aware detection and segmentation via text prompts or visual prompts, enabling open-vocabulary recognition. Nevertheless, these approaches usually require explicit visual prompt inputs or repeated similarity computation between visual prompts and target images, which limits their scalability to large reference contexts and prevents efficient inference. Preprint. Under review. In addition, recent approaches such as Rex-Thinker [8] and Rex-Omni [9] exploit multimodal large language models to further improve open-domain detection capabilities. Most relevant to our approach is Training-free [10], which introduces a training-free reference image-driven instance segmentation framework. By leveraging frozen DINOv2[11] semantic features and the categoryagnostic segmentation capability of SAM2[12], it constructs a feature memory bank with feature aggregation and semantic-aware matching. Given only a small number of reference images, this method achieves cross-domain generalization for automatic instance-level segmentation. However, existing approaches remain limited in scenarios where users provide large-scale positive and negative sample collections and expect the system to automatically learn from t

FAQs

What problem does VocaDet address in open-vocabulary detection and segmentation?

VocaDet targets open-vocabulary recognition where arbitrary objects must be detected beyond predefined categories. It addresses scalability issues caused by heavy feature matching, limited visual examples, and repeated similarity computations in existing approaches.

How does VocaDet learn object concepts without retraining?

It builds object memories directly from user-provided positive and negative sample collections. Continuous visual features are transformed into discrete multi-granularity visual tokens using DINOv3 and agglomerative clustering, then stored for retrieval.

How does VocaDet perform localization and segmentation during inference?

Query images are converted into visual tokens, then matched against the stored object memories in a vector database. Regions whose token nodes or local topological structures exceed a similarity threshold are identified as target objects, enabling segmentation.

Jul 16, 2026

Upload by Sage