Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation - Research

Aug 28, 2026

Uploaded by Xiajie

Referring video object segmentation targets accurate foreground prediction for objects specified by natural-language expressions in video. Prior approaches either rely on 3DConvNet encoders or add 2D spatial encoders, yet they often produce spatial misalignment or false distractors because spatial-temporal interaction is delayed and implicitly realized during decoding. A Language-Bridged Duplex Transfer (LBDT) module is proposed to use language as an intermediary bridge, enabling explicit and adaptive interaction earlier in encoding via cross-modal attention and incorporating a Bilateral Channel Activation denoising strategy in decoding. Experiments on four benchmarks show state-of-the-art results with substantial absolute AP gains and lower computation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

Introduction

Background and motivation

Limitations of existing methods

Proposed approach: LBDT

Method overview and modules

Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation Zihan Ding 1,4,5 Tianrui Hui2,3 Junshi Huang4 Xiaoming Wei4 Jizhong Han2,3 Si Liu 1,5 * 1 Institute of Artificial Intelligence, Beihang University 2 Institute of Information Engineering, Chinese Academy of Sciences 3 School of Cyber Security, University of Chinese Academy of Sciences 4 Meituan 5 Hangzhou Innovation Institute, Beihang University Abstract Referring video object segmentation aims to predict foreground labels for objects referred by natural language expressions in videos. Previous methods either depend on 3DConvNets or incorporate additional 2D ConvNets as encoders to extract mixed spatial-temporal features. However, these methods suffer from spatial misalignment or false distractors due to delayed and implicit spatial-temporal interaction occurring in the decoding phase. To tackle these limitations, we propose a Language-Bridged Duplex Transfer (LBDT) module which utilizes language as an intermediary bridge to accomplish explicit and adaptive spatialtemporal interaction earlier in the encoding phase. Concretely, cross-modal attention is performed among the temporal encoder, referring words and the spatial encoder to aggregate and transfer language-relevant motion and appearance information. In addition, we also propose a Bilateral Channel Activation (BCA) module in the decoding phase for further denoising and highlighting the spatialtemporal consistent features via channel-wise activation. Extensive experiments show our method achieves new stateof-the-art performances on four popular benchmarks with 6.8% and 6. 9% absolute AP gains on A2D Sentences and J-HMDB Sentences respectively, while consuming around 7 × less computational overhead 1. 1. Introduction Referring video object segmentation (RVOS), which aims to segment the target object referred by a natural language expression in video frames, is an emerging task atthe intersection of computer vision and natural language processing. Different from semi-automatic video object segmentation (SVOS) [3, 7, 31, 38], where the target ob- *Corresponding author 1 [https://github.com/dzh19990407/LBDT](https://github.com/dzh19990407/LBDT) Spatial language Temporal Figure 1 . Illustration of our main idea. (Top) For temporal→language→spatial transfer, the referring words (e.g.,“jumping”) can aggregate language-relevant motion information from the temporal features, which can help the spatial encoder recognize correct actions. (Bottom) For spatial→language→temporal transfer, the referring words (e.g., “brown”) can aggregate the language-relevant appearance information from the spatial features, which helps the temporal encoder remove the disturbance of background motion (e.g., the rotating wheel) . ject is referred by the manually annotated mask in the first frame, RVOS is more challenging for identifying the targets due to the variance of free-form expressions. Providing amore natural way for human-computer interaction, RVOS opens up a wide range of applications including languagebased video editing [5], language-guided video summarization [29], and video question answering [14, 39], etc. The keys to solving RVOS are spatial-temporal interaction and cross-modal alignment [12, 41] . Existing methods mainly focus on the latter and design several mechanisms (e.g., cross-modal attention [30, 37], capsule routing [25], and dynamic convolution [6, 36]) to mine the semantic correspondence between vision and language modalities. However, all these methods have limitations on spatial- temporal interaction due to the reliance on 3D ConvNets (e.g., I3D [1]) . Concretely, since the poses and locations of moving objects vary in adjacent frames, aggregating spatially misaligned multi-frame features via 3D operators (e.g., 3D convolution and 3D pooling) may confuse the original appearance information in the target frame, leading to inaccurate segmentation results. To al

FAQs

What problem does referring video object segmentation address?

It segments the target object described by a natural-language expression within video frames.

Why do existing approaches often fail?

They may introduce spatial misalignment and false distractors because spatial-temporal interaction happens too late and remains implicit during decoding.

How does the proposed LBDT module improve spatial-temporal interaction?

It uses language as a bridge so temporal features transfer to referring words and then guide spatial encoding (and vice versa) through cross-modal attention, enabling earlier explicit interaction.

Aug 28, 2026

Uploaded by Xiajie