UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery

Jul 16, 2026

Unmanned aerial vehicles increasingly require visual grounding that links diverse instructions to task-relevant targets in complex aerial scenes. Traditional referring expression comprehension benchmarks mostly support text-only queries and single-object outputs, limiting UAV applicability when reference images, multimodal instructions, absent targets, or multiple valid instances are involved. UniRef-UAV introduces Universal Referring and a multimodal benchmark supporting text, image, and text+image queries with modality-dependent target cardinality, plus in-domain and cross-domain evaluation protocols. It also presents UAV-URNet, a set-prediction baseline with stable, reproducible performance and analyses showing multimodal queries reduce ambiguity and unify query–target alignment.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Introduction

Motivation and problem setting

Limitations of existing REC in UAV scenarios

Universal Referring and multimodal inputs

Proposed benchmark and baseline (overview)

This work has been submitted to the IEEE Transactions on Multimedia for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. IEEE TRANSACTIONS ON MULTIMEDIA 1 UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery Haibin Tian, Huichao Xie, Xuelin Qian, Member, IEEE, Ruitao Lu, Junwei Han, Fellow, IEEE, and Dingwen Zhang, Member, IEEE arXiv :2607 .08267v 1 [ cs .CV] 9 Jul 2026 Abstract—Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets from diverse instructions in complex aerial scenes. Existing referring expression comprehension (REC) benchmarks and methods, however, are largely built around text-only queries and single-object outputs, which limits their applicability to practical UAV scenarios involving reference images, multimodal instructions, absent targets, and multiple valid target instances. To address this gap, we introduce Universal Referring, a generalized UAV referring task that jointly expands the query modality and the output cardinality. We construct UniRef-UAV, amultimodal benchmark that supports text-only, image-only, and text+image queries with modality-dependent target cardinality, where text-only and text+image queries admit no-target, singletarget, and multi-target grounding while image-only queries focus on existence-aware single-instance grounding. It also provides indomain and cross-domain evaluation protocols for visual-query generalization. We further present UAV-URNet, a detection-style baseline that maps heterogeneous queries into a shared query space and predicts variable-size target sets through set prediction. Extensive experiments show that UAV-URNet provides a stable and reproducible baseline with more consistent no-target discrimination and a more lightweight, reproducible implementation than large general-purpose MLLMs. Additional domain analysis, query-representation analysis, and ablation studies demonstrate that multimodal queries help reduce visual-query ambiguity and promote a more unified query–target alignment space. The annotations, visual query crops/images, train/validation/test splits, evaluation scripts, and baseline code will be made publicly available to facilitate reproducible research. Index Terms—UAV vision, multimodal referring, universal referring, visual grounding, variable-cardinality detection. I. INTRODUCTION UNMANNED aerial vehicles (UAVs) have become an im portant mobile sensing platform for security patrols [1], emergency search and rescue [2], [3], traffic monitoring [4], and industrial inspection [5] . Related visual systems have also expanded toward large-scale scene reconstruction and open-vocabulary scene querying [6]–[8], but practical UAV operations still require interfaces that ground task instructions to concrete visual targets. Beyond autonomous flight and navigation, these applications increasingly require UAVs Haibin Tian and Huichao Xie contributed equally to this work. Haibin Tian, Huichao Xie, Xuelin Qian, and Dingwen Zhang are with the School of Automation, Northwestern Polytechnical University, Xi’an 710072, China (e-mail: [haibintian@foxmail.com](haibintian@foxmail.com); [xiehuichao@mail.nwpu.edu.cn](xiehuichao@mail.nwpu.edu.cn); [xlqian@nwpu.edu.cn](xlqian@nwpu.edu.cn); [zdw2006yyy@nwpu.edu.cn](zdw2006yyy@nwpu.edu.cn)). Corresponding author: Dingwen Zhang. Ruitao Lu is with the College of Missile Engineering, Rocket Force University of Engineering, Xi’an 710038, China (e-mail: [lrt19880220@163.com](lrt19880220@163.com)). Junwei Han is with the School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing 400065, China (email: [jhan@nwpu.edu.cn](jhan@nwpu.edu.cn)). to interpret task-level instructions and associate them with specific visual targets in complex aerial scenes. For example, a rescue UAV must localize the indicated vic

FAQs

What problem does UniRef-UAV address in UAV referring tasks?

It addresses the mismatch between existing REC benchmarks (typically text-only, single-target outputs) and practical UAV needs involving multimodal instructions, reference images, absent targets, and multiple valid instances in aerial scenes.

What query modalities and output cardinalities does the UniRef-UAV benchmark support?

UniRef-UAV supports text-only, image-only, and text+image queries. Target cardinality depends on modality: text-only and text+image support no-target, single-target, and multi-target grounding, while image-only focuses on existence-aware single-instance grounding.

What is UAV-URNet and how does it perform query understanding?

UAV-URNet is a detection-style baseline that maps heterogeneous queries into a shared query space and uses set prediction to output variable-size target sets, providing a stable and reproducible baseline with improved no-target discrimination.

Jul 16, 2026

Upload by River Wang