C-GAP Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes

Jul 16, 2026

Safety-critical perception systems must reliably detect rare object classes within small label spaces, where long-tailed methods built for dense, large-category supervision provide limited help. Open-vocabulary detectors enable natural-language queries at inference time, making prompt quality a direct control knob. C-GAP tests whether iteratively refining prompts for frozen detectors can boost minority-class detection without retraining or extra annotations. It introduces a composite caption baseline and an LLM-driven refinement loop, achieving large AP@0.5 gains over baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Abstract

Introduction

C-GAP Framework Overview

Application: Smart Intersection Monitoring

Contributions

C-GAP: Class-Aware and Online Prompting Improves Vision–Language Models on Imbalanced Classes Francis Fernandez San Diego State University [fafernandez@sdsu.edu](fafernandez@sdsu.edu) Arash Jahangiri San Diego State University [ajahangiri@sdsu.edu](ajahangiri@sdsu.edu) Salimeh Sekeh San Diego State University [ssekeh@sdsu.edu](ssekeh@sdsu.edu) arXiv :2607 .09008v1 [ cs .CV] 10 Jul 2026 Abstract Safety-critical perception systems must reliably detect rare object classes within small label spaces, a setting that long-tailed detection methods, designed for hundreds of classes with dense annotation, fundamentally do not address. Open-vocabulary detectors offer a promising alternative, as they use natural language queries at inference time, making prompt quality a first-class lever for detection performance. We exploit this property to address class imbalance: rather than retraining models or collecting additional annotations, we ask whether iteratively refining the language prompts, fed to frozen detectors, can improve minorityclass detection. We introduce C-GAP (Caption-Guided Augmentation and Prompting), a detector-agnostic, annotation-free framework that operates in two phases. First, we establish a composite caption baseline combining per-image scene descriptions with class-quantity context, which we show outperforms scene-descriptiononly or class-quantity-only prompts across multiple open-vocabulary architectures and benchmarks. Second, an LLM iteratively refines each image’s caption individually, with trials triaged into accept, tentative, or regenerate buckets based on minority-class AP@0.5 against a dynamic threshold derived from the composite baseline. Refinement terminates early once sufficient AP@0.5 gain is achieved. No detector weights are updated at any stage. Our experiments shows that C-GAP improves minority-class average precision up to 53% over the baselines. On COCO, C-GAP improves minority-class AP@0.5 by ∼81% relative over the composite baseline (17 .69 → 32.09). Experiments confirm that composite captions provide the critical foundation for effective refinement: using scene-description-only or class-quantity-only prompts as the refinement starting point yields diminishing returns, supporting both stages of C-GAP as necessary contributions. Figure 1 . Chula Vista intersection: cars dominate (∼80%) while cyclists are severely underrepresented (∼8%), motivating minority-class prompt refinement. 1. Introduction Safety-critical perception systems must reliably detect rare object classes within small label space, a setting where long-tailed detection methods, designed for hundreds of categories with dense annotation, offer limited guidance [18, 33] . In urban traffic monitoring, minority classes such as cyclists and pedestrians account for a small fraction of instances yet are the most consequential for collision avoidance and regulatory compliance. Standard resampling and loss-reweighting remedies require labeled minority instances that may simply not exist in sufficient quantity, and collecting additional annotations is expensive. Because open-vocabulary detectors (OVDs) [51] accept free-form text queries at inference time rather thana fixed classification head, prompt quality becomes a direct lever for detection performance, requiring no annotation, no architectural change, and no retraining. We Figure 2 . Overview of C-GAP. Phase I (top): per-image Scene Description (tSDi) and Class Quantity (tCQi) captions are generated offline and concatenated into a Composite Caption ti,0 = concat(tSDi, tCQi), forming the initial caption set T0 . Phase II (bottom): a VLM refines T0 over K trials. Each candidate set Tk is evaluated by the frozen detector fθ , yielding minority-class AP50 (cm ;Tk) . Trials are triaged into B1 (regenerate, AP50 < τlow ), B2 (tentative, τlow ≤ AP50 ≤ τhigh ), or B3 (keep, AP50 > τhigh ) . The output is Tk ∗ where k ∗ = arg max0≤k≤K AP50 (cm ;Tk) . Detector weights θ

FAQs

What is the role of the composite caption baseline in C-GAP?

The composite caption baseline combines per-image scene descriptions with class-quantity context. Experiments show it is the critical foundation for effective refinement, while starting from scene-description-only or class-quantity-only prompts yields diminishing returns.

Jul 16, 2026

Upload by River Wang