Dual-Correlation Hypergraph Network for Unaligned RGBT Video Object Detection and a Large-Scale Benchmark

Jul 16, 2026

RGB-Thermal (RGBT) video object detection addresses the robustness limits of RGB-only methods under low-light, overexposure, and adverse weather such as rain and snow. Real RGBT pairs often suffer from spatial misalignment, harming cross-modal fusion. DHNet tackles this with a patch-based spatial alignment approach (PSAM) and a dual-correlation hypergraph fusion strategy (DHFM) that learns temporal and crossmodal complementary correlations. It also introduces DVT-VOD1000, a large-scale, scene-diverse benchmark with 1,000 sequences and 103,464 image pairs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Abstract

Index Terms

Introduction

Background: RGB-only limitations in VOD

Related Work and Dataset Gaps

Summary of Main Challenges and Proposed Approach

Dual-Correlation Hypergraph Network for Unaligned RGBT Video Object Detection and A Large-scale Benchmark Qishun Wang, Yapeng Li, Bin Luo, Zhengzheng Tu, and Chenglong Li, Senior Member, IEEE arXiv :2607 .08 19 1v 1 [ cs .CV] 9 Jul 2026 Abstract—RGB-Thermal (RGBT) Video Object Detection (VOD) has gained significant traction due to its ability to overcome the limitations of conventional RGB-based VOD under challenging conditions. However, spatial misalignment commonly exists between RGBT image pairs. To address this, we propose a Dual-Correlation Hypergraph Network (DHNet) that captures high-dimensional complementary information by explicitly modeling two types of correlations: temporal correlation across consecutive frames and spatial correlation from crossmodal features. Specifically, we first design a Patch-based Spatial Alignment Module (PSAM) to sequentially align the multimodal features at the local region level. Subsequently, we introduce a Dual Hypergraph Fusion Module (DHFM), which constructs separate temporal and multimodal hypergraphs to enhance object discriminability through dual-correlation learning. Furthermore, the field currently lacks a large-scale, scene-diverse benchmark dataset for comprehensive evaluation. To address this gap, we construct DVT-VOD1000, a large-scale RGBT VODdataset containing 1,000 video sequences with 103,464 RGBT image pairs. The dataset covers diverse scenarios, including campuses, parks, transportation, rural areas, night scenes, rain, and snow. Comprehensive experiments on VT-VOD50 and our DVT-VOD1000 demonstrate that DHNet achieves state-of-theart detection accuracy. The dataset and source code will be made publicly available on [https://github.com/tzz-ahu/ to support](https://github.com/tzz-ahu/ to support)[ ](https://github.com/tzz-ahu/ to support)academic research. Index Terms—RGB-Thermal, video object detection, multimodal fusion, hypergraph, benchmark dataset. I. INTRODUCTION AS a key visual perception task, Video Object Detection (VOD) serves as a core component in fields like security surveillance and autonomous driving [1]–[3] . However, VOD that relies solely on RGB information remains susceptible to robustness issues in extreme scenarios, such as low-light conditions at night, overexposure, and adverse weather including rain, snow, and fog. To mitigate these limitations, the work by Wang et al. [4] introduces a method that combines RGB and thermal (RGBT) for VOD and creates a corresponding benchmark dataset, VT-VOD50 . The authors apply manual processing to this dataset to achieve consistency in both resolution and spatial distribution. However, the VTVOD50 dataset contains only 100 RGBT video sequencesand fewer than 10,000 image pairs. Furthermore, its data Corresponding author: Zhengzheng Tu. Qishun Wang, Yapeng Li, Zhengzheng Tu, and Bin Luo are with Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei 230601, China (email: [qishunahu@163.com](qishunahu@163.com); [zhengzhengahu@163.com](zhengzhengahu@163.com); [luobin@ahu.edu.cn](luobin@ahu.edu.cn)). Chenglong Li is with Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Artificial Intelligence, Anhui University, Hefei, 230601, China (e-mail: [lcl1314@foxmail.com](lcl1314@foxmail.com)) . | <br>VT-VOD50 (a) | | <br><br> | | --- | --- | --- | | | | | Fig. 1. Existing VT-VOD50 and our proposed DVT-VOD1000, commonly exhibit weak spatial alignment of objects. As shown in (b), this issue manifestsas significant regional variations in the degree of misalignment. originates exclusively from traffic scenarios, representing a single data domain. These limitations pose constraints on the comprehensive evaluation of RGBT VOD methods and adversely affect model generalization. Wang et al. [4] introduce EINet for RGBT VOD. The model employs a negative activation function to identify noise region

FAQs

What problem does the document target in RGBT video object detection?

It targets spatial misalignment commonly existing between RGB and thermal frames, which weakens cross-modal fusion and reduces detection accuracy.

How does DHNet improve alignment and feature learning?

DHNet uses a Patch-based Spatial Alignment Module (PSAM) for local region alignment and a Dual Hypergraph Fusion Module (DHFM) to learn dual correlations from temporal information and crossmodal features.

What benchmark dataset is introduced, and what does it include?

The document introduces DVT-VOD1000, containing 1,000 video sequences and 103,464 RGBT image pairs, covering diverse scenes such as campuses, parks, transportation, rural areas, night conditions, rain, and snow.

Jul 16, 2026

Upload by Theodora