Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing

Jul 16, 2026

Upload by Liam

Video–text temporal localization aligns natural language queries with the precise temporal boundaries of video segments, a core problem in multimodal understanding. The framework addresses two key gaps in prior work: weak hierarchical modeling of temporal structure and limited capability for complex many-to-many modality correspondences. It combines a multi-scale temporal convolutional encoder for motion patterns at multiple granularities with capsule-based dynamic routing that iteratively refines segment–query associations via structured agreement updates. Joint multi-task learning optimizes boundary regression, cross-modal alignment, and capsule diversity. Experiments on ActivityNet Captions reach 42.9% Recall@0.5 and 41.1% mean IoU, outperforming strong transformer baselines while remaining computationally efficient.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Problem: Video–Text Temporal Localization

Limitations of Existing Methods

Proposed Framework and Key Ideas

Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing Gengtian Shi 1 , Jinze Yu2,1 , Chenhao Wu 1 , Shaofei Wang 1 , Eiji Fukuzawa 1 , Junjie Tang3 , Hiroshi Onoda 1 , Jiang Liu 1 1 Graduate School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan 2 Generative AI Innovation Center, Amazon Web Services, Japan 3Amazon, Germany [shigengtian@akane.waseda.jp](shigengtian@akane.waseda.jp), [jinzeyu@amazon.co.jp](jinzeyu@amazon.co.jp) arXiv :2607 .05093v2 [ cs .CV] 7 Jul 2026 Abstract Video–text temporal localization requires precise alignment between natural language queries and corresponding video segments, a fundamental challenge in multimodal understanding. We present a novel framework that addresses two critical limitations of existing methods: inadequate modeling of hierarchical temporal structure and inability to handle complex many-to-many correspondences between modalities. Our approach introduces a multi-scale temporal convolutional encoder that captures motion patterns across different temporal granularities—from instantaneous frame transitions to extended action sequences. We further propose a capsule-based dynamic routing mechanism that iteratively refines segment–query associations through structured agreement updates, enabling flexible modeling of non-monotonic alignments. These components are unified through a multi-task learning objective that jointly optimizes temporal boundary regression, cross-modal semantic alignment, and capsule diversity. Extensive experiments on ActivityNet Captions demonstrate significant improvements, achieving 42.9% Recall@0.5 and 41.1% mean IoU, surpassing strong transformer-based baselines while maintaining computational efficiency. Our results validate that combining hierarchical temporal modeling with structured semantic routing provides an effective solution for fine-grained video–language understanding. Introduction The proliferation of video content has created an urgent need for systems that can understand and navigate temporal relationships between visual content and natural language. Video–text temporal localization, also known as temporal sentence grounding (TSG) or video moment retrieval (VMR), addresses this challenge by identifying precise temporal boundaries in videos that correspond to textual descriptions (Gao et al. 2017; Hendricks et al. 2017; Lan et al. 2023; Zhang et al. 2023) . This capability is essential for applications ranging from instructional video understanding and human–robot interaction to content-based video retrieval and interactive video editing (Chen et al. 2018; Liet al. 2020) . Despite recent advances in vision–language modeling, achieving fine-grained temporal alignment remains fundamentally challenging. Current pretrained models such as CLIP (Radford et al. 2021), BLIP (Li et al. 2022), and their video-adapted variants (e.g., CLIP4Clip (Luo et al. 2021), VideoBERT (Sun et al. 2019), X-CLIP (Ma et al. 2022)) excel at coarse-grained video–text matching but struggle with precise moment-level localization. These models typically operate on entire video clips or large temporal segments, lacking the temporal resolution necessary to identify exact action boundaries, subtle transitions, and complex visuallinguistic correspondences (Li et al. 2023) . The challenge is particularly acute in instructional and procedural videos, where multiple related actions occur inclose temporal proximity with ambiguous boundaries. Consider a cooking video where ”adding ingredients” and ”mixing the batter” may overlap temporally, or an assembly tutorial where distinct textual instructions correspond to partially concurrent visual actions. As illustrated in Figure 1, real-world scenarios frequently exhibit three types of complexity: (1) single queries spanning multiple disjoint video segments,(2) multiple queries mapping to overlapping temporal regions, and (3) subtle transitions between semantically re

FAQs

What problem does the paper address?

The paper addresses video–text temporal localization (temporal sentence grounding), which finds precise video time boundaries that correspond to natural language descriptions.

What are the two main limitations of existing approaches highlighted in the paper?

Most methods do not explicitly model hierarchical temporal structure across different time scales, and their alignment mechanisms assume overly simple (often monotonic or one-to-one) correspondences, failing on many-to-many relationships.

How does the proposed method improve temporal localization?

It uses a multi-scale temporal convolutional encoder to capture motion at multiple granularities and capsule-based dynamic routing to iteratively refine segment–query associations through structured agreement updates, optimized jointly with multi-task learning objectives.

Jul 16, 2026

Upload by Liam