STMTrack - Template-free Visual Tracking with Space-time Memory Networks - CVPR 2021 paper

Sep 12, 2026

Upload by Kyle

Boosting offline-trained Siamese trackers becomes increasingly difficult as the fixed template from the first frame is thoroughly exploited, leaving limited resistance to target appearance changes. Existing template-updating approaches often require costly numerical optimization and intricate handcrafted strategies, restricting real-time use. This paper introduces STMTrack, a space-time memory network framework that leverages historical target information via a novel memory mechanism to emphasize informative regions, and uses pixel-level similarity for more accurate bounding boxes, achieving strong results at 37 FPS.

LanguageEnglish

CategoryGeneral

FormatPDF

Pages10

Reading Time4 min

AccessFree to download

Introduction

Problem motivation and challenges

Template-based vs template-updating vs memory-based tracking

Contributions and evaluation overview

STMTrack: Template-free Visual Tracking with Space-time Memory Networks Zhihong Fu, Qingjie Liu􀀃, Zehua Fu, Yunhong Wang State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China Hangzhou Innovation Institute, Beihang University {fuzhihong, qingjie.liu, [yhwang](yhwang}@buaa.edu.cn)[}](yhwang}@buaa.edu.cn)[@buaa.edu.cn](yhwang}@buaa.edu.cn), zehua [fu@163.com](fu@163.com) Abstract Boosting performance of the of􀀃ine trained siamese trackers is getting harder nowadays since the 􀀂xed information of the template cropped from the 􀀂rst frame has been almost thoroughly mined, but they are poorly capable of resisting target appearance changes. Existing trackers with template updating mechanisms rely on time-consuming numerical optimization and complex hand-designed strategies to achieve competitive performance, hindering them from real-time tracking and practical applications. In this paper, we propose a novel tracking framework built on top of a space-time memory network that is competent to make full use of historical information related to the target for better adapting to appearance variations during tracking. Specifically, a novel memory mechanism is introduced, which stores the historical information of the target to guide the tracker to focus on the most informative regions in the current frame. Furthermore, the pixel-level similarity computation of the memory network enables our tracker to generate much more accurate bounding boxes of the target. Extensive experiments and comparisons with many competitive trackers on challenging large-scale benchmarks, OTB-2015, TrackingNet, GOT-10k, LaSOT, UAV123, and VOT2018, show that, without bells and whistles, our tracker outperforms all previous state-of-the-art real-time methods while running at 37 FPS. The code is available at [https:](https:)// [github.com/fzh091](github.com/fzh091) 7/STMTrack. 1. Introduction Visual object tracking is an essential task in computer vision with applications in various 􀀂elds such as humancomputer interactions [29], video surveillance [54], and autonomous driving [22] . Signi􀀂cant efforts have been devoted to address this problem, yet there is still a great gap to the practical applications due to the challenging factors such as occlusions, fast motions, and non-rigid deforma- 􀀃 Corresponding author. Figure 1: Visualized comparisons of our method with representative trackers SiamFC++ [56] and DiMP-50 [2] . Our method can estimate more accurate target state when targets suffer from partial occlusions and non-rigid deformations. tions [13, 53, 20], which urge us to develop trackers with strong adaptiveness and robustness. The goal of visual tracking is to locate an object in the subsequent frames of a video given its initial annotation in the 􀀂rst frame. In recent years, with the advancements of deep learning techniques, deep trackers have dominated the tracking 􀀂eld, among which two methodologies are widely studied, and one popular methodology addresses object tracking as a similarity matching problem between the target template and the search frames in an embedding space of􀀃ine trained. The representative template-matching methods are siamese trackers [1, 65, 61, 14, 50, 6, 24, 23, 56, 15] . These methods usually do not update the template and thus are hard to adapt to appearance changes caused by occlusions, non-rigid deformations, etc. To solve this problem, some trackers [2, 11] are equipped with sophisticated template updating mechanisms and thus show stronger robustness than siamese trackers. However, online template updating requires much more computational resources, which impends trackers from real-time tracking. Furthermore, these customized updating strategies [16, 58, 65, 25, 59, 7] introduce hyper-parameters that require tricky tuning. Note that when tracking moving objects humans remember their identities in visual working memory to maintain temporal continuity in a cons

FAQs

Why is offline-trained Siamese tracking becoming harder to improve?

The template cropped from the first frame provides limited additional adaptability, so trackers become weak when appearance changes occur during tracking.

What makes existing template-updating methods unsuitable for real-time applications?

They typically rely on time-consuming optimization and complex hand-designed strategies with difficult-to-tune hyperparameters, increasing computational cost.

How does STMTrack use historical information differently from template updating?

STMTrack stores historical target information in a space-time memory network and predicts the target state from this memory, avoiding explicit template use and online updating.

Sep 12, 2026

Upload by Kyle