COUNTGD - Multi-Modal Open-World Counting - NeurIPS 2024 Paper

Aug 23, 2026

Upload by Angel

Open-vocabulary object counting is extended to improve both generality and accuracy in open-world images. The paper repurposes GroundingDINO for counting and adds modules that let users specify target objects using text descriptions, visual exemplars, or both. Multi-modal prompting increases counting precision and flexibility, supports restricting counts with combined cues, and enables a single-stage open-world counting model named COUNTGD. Experiments report strong benchmark improvements across text-only and text+exemplar settings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

AccessFree to download

Abstract

1 Introduction

Open-world object counting motivation

Limitations of text-only vs exemplar-based methods

COUNTGD overview and contributions

Model architecture and multi-modal prompt mechanism

COUNTGD: Multi-Modal Open-World Counting Niki Amini-Naieni Tengda Han Andrew Zisserman Visual Geometry Group (VGG) University of Oxford {nikian,htd,[az}@robots.ox.ac.uk](az}@robots.ox.ac.uk) Abstract The goal of this paper is to improve the generality and accuracy of open-vocabulary object counting in images. To improve the generality, we repurpose an openvocabulary detection foundation model (GroundingDINO) for the counting task, and also extend its capabilities by introducing modules to enable specifying the target object to count by visual exemplars. In turn, these new capabilities – being able to specify the target object by multi-modalites (text and exemplars)– lead to an improvement in counting accuracy. We make three contributions: first, we introduce the first open-world counting model, COUNTGD, where the prompt can be specified by a text description or visual exemplars or both; second, we show that the performance of the model significantly improves the state of the art on multiple counting benchmarks – when using text only, COUNTGD is comparable to or outperforms all previous text-only works, and when using both text and visual exemplars, we outperform all previous models; third, we carry out a preliminary study into different interactions between the text and visual exemplar prompts, including the cases where they reinforce each other and where one restricts the other. The code and an app to test the model are available at [https://www.robots.ox.ac.uk/vgg/research/countgd/](https://www.robots.ox.ac.uk/vgg/research/countgd/) . Figure 1: COUNTGD is capable of taking both visual exemplars and text prompts to produce highly accurate object counts (a), but also seamlessly supports counting with only text queries or only visual exemplars (b) . The multi-modal visual exemplar and text queries bring extra flexibility to the open-world counting task, such as using a short phrase (c), or adding additional constraints (the words ‘left’ or ‘right’) to select a sub-set of the objects (d) . These examples are taken from the FSC-147 [42] and CountBench [39] test sets. The visual exemplars are shown as yellow boxes. (d) visualizes the predicted confidence map of the model, where a high color intensity indicates a high level of confidence. 1 Introduction Open-world object counting methods aim to enumerate all the instances of any category of object inan image. The ‘open-world’ refers to the model’s ability to count objects beyond the set of categories seen at training, thus enabling the user to specify categories of interest at inference without the need for model retraining. Recent techniques allow the user to specify the target object with only visual exemplars – bounding boxes around a few example objects in the image – [32, 35], or only text 38th Conference on Neural Information Processing Systems (NeurIPS 2024) . descriptions [1, 22] . By accepting either visual exemplars or text as prompts, open-world object counting methods can adapt to the specific object at inference time. This enables these techniques to count arbitrary classes of objects as specified by the user. Methods that use visual exemplars to specify the object currently significantly outperform text-based counting methods on multiple benchmarks. This is because visual exemplars provide more detailed information than text – it can take many words to precisely describe an object; and perhaps more importantly, they provide intrinsic information on the object’s appearance – because the exemplars are from the same image they already ‘factor in’ the viewpoint and lighting, variables that significantly affect the object’s appearance. However, while visual exemplar-based approaches are more accurate, they limit the capabilities and generality of the counting model. In this paper we introduce a counting model that is able to specify the target object using visual exemplars, a text description, or both together. The model, named COUNTGD, has superior

FAQs

What is COUNTGD designed to improve in open-world object counting?

COUNTGD targets higher generality and accuracy for open-vocabulary object counting in images by enabling flexible prompt-based object specification at inference time.

How does COUNTGD specify the target object for counting?

It accepts either text descriptions, visual exemplars, or both together, treating visual exemplars as text tokens and fusing them with text through attention mechanisms.

What are the reported performance benefits of using text and visual exemplars together?

Using both modalities significantly improves state-of-the-art results across counting benchmarks, while text-only performance remains comparable to or better than previous text-only approaches.

Aug 23, 2026

Upload by Angel