LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models - CVPR 2025 Supplemental Materials

Aug 26, 2026

Upload by Liam

Supplementary materials for LLMDet present ablation evidence on how different large language model backbones and vision-language components affect detector performance, including rare-class gains when using LLaVA-OneVision as a multimodal foundation. The document details a multi-step training pipeline that builds a stronger large vision-language model by combining vision experts (LLMDet and SigLIP), concatenating their features, and learning a projector into the LLM input space. It also reports benchmark comparisons across VQA, hallucination, and comprehensive understanding settings, and summarizes observed limitations in long-form image description generation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Ablations

More Ablation Studies

LLMDet Builds a Stronger Large VisionLanguage Model

Limitations

Implement Details of Zero-Shot Test on Referring Expression Comprehension

Supplementary Materials for LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models Shenghao Fu 1 ,3 ,4 , Qize Yang3 , Qijie Mo 1 ,4 , Junkai Yan 1 ,4 , Xihan Wei3 , Jingke Meng 1 ,4 , Xiaohua Xie 1 ,4 ,5 *, Wei-Shi Zheng 1 ,2 ,4 ,6∗ 1 School of Computer Science and Engineering, Sun Yat-sen University, China; 2Peng Cheng Laboratory, China; 3Tongyi Lab, Alibaba Group; 4 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China; 5 Guangdong Province Key Laboratory of Information Security Technology, China; 6Pazhou Laboratory (Huangpu), China [fushh7@mail2.sysu.edu.cn](fushh7@mail2.sysu.edu.cn), [xiexiaoh6@mail.sysu.edu.cn](xiexiaoh6@mail.sysu.edu.cn), [wszheng@ieee.org](wszheng@ieee.org) | LLM | AP | APr | APc | APf | | --- | --- | --- | --- | --- | | Qwen2-0.5b-instruct [12] | 44.4 | 36.4 | 39.2 | 50.5 | | LLaVA-OneVision-0.5b-ov [4] | 44.5 | 38.6 | 39.3 | 50.3 | | Qwen2-1.5b-instruct [12] | 44.6 | 35.3 | 39.5 | 50.8 | Table 1-1 . Ablations on large language models. 1. More Ablation Studies Effect of different large language models. By default, we use the LLM in LLaVA-OneVision-0.5b-ov [4], which is finetuned from Qwen2-0 .5b-instruct [12] . Since the LLMin LLaVA-OneVision-0.5b-ov is pretrained with abundant multi-modal data but with a different vision encoder, thepretraining can still improve the performance, especially for rare classes (+2 .2% APr ), as shown in Table 1-1. But we find that increasing the size of the LLM only slightly improves the performance, perhaps larger language models mainly improve in reasoning ability which does not benefit the detector’s visual representations. 2. LLMDet Builds a Stronger Large VisionLanguage Model In this subsection, we show that LLMDet can serve as a general vision foundation model and in turn gets a strong large multi-modal model. Recent large multi-modal models (LMM) are based on pretrained large language models and pretrained vision foundation models. Different vision foundation models will significantly affect the performance ofLMMs [16] . Since LLMDetis enhanced under the * : Corresponding authors are Xiaohua Xie and Wei-Shi Zheng. Part of the work was done when Shenghao Fu was an intern at Alibaba. | goal | Alignment | Training LMM | | --- | --- | --- | | task | Image Captioning | Instruction Following | | loss | llm loss | llm loss | | dataset | LSC 558k recap | LLaVA 1.5 instruction tuning dataset (665K) | Figure 2-1 . The multi-step training pipeline of using LLMDet to build a strong large multi-modal model. The large multi-modal model uses a mixture of vision encoders, including LLMDet and SigLIP. In each step, modules in orange color are tunable while modules in blue color are frozen. We first pretrain a new projector and then finetune the large multi-modal model with visual instruct tuning. supervision of long detailed image-level captions and prealigned with LLM, LLMDet inherits great potential to build a stronger LMM. Following recent advances [10, 11, 16], we build the LMM using a mixture of vision experts, i.e. a SigLIP [14] vision encoder and our LLMDet. As shown in Figure 2-1, the visual features from two vision encoders are concatenated along the channel dimension, and then a projector is utilized to map the features to the LLM’s input space. We start from LLaVA-OneVision-0.5b-ov [4] and | Method | GQA
[3] | POPE [6] | | | MME [2] | | | --- | --- | --- | --- | --- | --- | --- | | | | rand | pop | adv | perception | cognition | | OneVision-0.5b | 56.9 | 87.5 | 86.3 | 85.0 | 1238 | 240 | | OneVision-0.5b
+MM-GDINO | 61.2 | 88.9 | 88.1 | 86.6 | 1207 | 256 | | OneVision-0.5b +LLMDet | 61.2 | 88.8 | 88.0 | 86.0 | 1297 | 264 | Table 2-2 . Multi-modal performance using different vision encoders. OneVision-0 .5b is short for LLaVA-OneVision-0 .5bov [4] . insert our LLMDet to it as shown in Figure 2-1. We first pretrain a new projector a

FAQs

How do different large language models affect LLMDet’s detector performance?

The ablations compare LLM backbones and show that using the LLaVA-OneVision vision-language setup yields improvements, with noticeable gains on rare classes. Increasing LLM size only slightly improves performance, suggesting benefits concentrate on reasoning rather than visual representations.

What training pipeline does LLMDet use to build a stronger large multimodal model?

The pipeline uses multi-step training: first pretraining a projector, then finetuning the multimodal model with visual instruction tuning. It combines visual features from two encoders (LLMDet and SigLIP) and maps them into the LLM’s input space.

What limitations are observed in the co-trained LLM-detector output?

Even with prompts for detailed descriptions, the co-trained system tends to produce relatively short descriptions for whole images. Region-level outputs are also described as relatively simple grounding phrases, motivating collection of more informative region data.

Aug 26, 2026

Upload by Liam