Learning Data Association for Multi-Object Tracking using Only Coordinates - Abstract

Aug 23, 2026

Uploaded by Mali

A Transformer-based module named TWiX addresses data association in multi-object tracking by estimating an affinity score between pairs of tracklets extracted from two temporal windows. Built from detections produced by a pretrained detector, it uses only coordinates from bounding boxes and is trained with supervised contrastive learning to discriminate pairs from the same object versus different objects. TWiX avoids intersection-over-union, motion priors, and camera motion compensation, and when inserted into an online cascade matching pipeline, yields state-of-the-art results on DanceTrack and KITTIMOT and competitive performance on MOT17.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages29

Reading Time1 h 13 min

AccessFree to download

Introduction

Multi-object tracking as data association

Online vs offline association

Contributions overview

Related works

time STEP 3 : C-TWiX, ONLINE TRACKING WITH A CASCADE MATCHING PIPELINE Update tracks and age Delete Highlights Learning Data Association for Multi-Object Tracking using Only Coordinates Mehdi Miah, Guillaume-Alexandre Bilodeau, Nicolas Saunier • Our Transformer-based model, TWiX, can learn to associate objects using only coordinates; • We show that motion priors or intersection-over-union measure are not required for tracking. Using pairs of tracks is sufficient ; • Tracking with TWiX give competitive or state-of-the-results on several dataset. Learning Data Association for Multi-Object Tracking using Only Coordinates Mehdi Miaha , Guillaume-Alexandre Bilodeaua , Nicolas Sauniera aPolytechnique Montr´eal, 2500 Chemin de Polytechnique, Montr´eal, H3T 1J4, Qu´ebec, Canada Abstract We propose a novel Transformer-based module to address the data association problem for multi-object tracking. From detections obtained by apretrained detector, this module uses only coordinates from bounding boxes to estimate an affinity score between pairs of tracks extracted from two distinct temporal windows. This module, named TWiX, is trained on sets of tracks with the objective of discriminating pairs of tracks coming from the same object from those which are not. Our module does not use the intersection over union measure, nor does it requires any motion priors or any camera motion compensation technique. By inserting TWiX within an online cascade matching pipeline, our tracker C-TWiX achieves state-of-the-art performance on the DanceTrack and KITTIMOT datasets, and gets competitive results on the MOT17 dataset. The code will be made available upon publication. Keywords: tracking, transformer, data association, motion, multi-object tracking 1. Introduction Multi-object tracking (MOT) consists in detecting all objects of interest, such as cars or pedestrians, and assigning them a unique identity throughout a video. Common applications are road safety analysis (Zangenehpour et al., 2016), video-surveillance and environment awareness in self-driving cars (Sarcinelli et al., 2019) . With the improvement of object detectors (Duanet al., 2019; Carion et al., 2020; Ge et al., 2021), a popular paradigm to solve MOT is tracking-by-detection, which consists of two steps: detecting objects Preprint submitted to Pattern Recognition March 14, 2024 in each frame of the video and associating detections that correspond to the same object. Under this paradigm, MOT is mainly solved as a data association problem: given two sets of detections, the objective is to find those that refer to the same object. This data association can either be done inan online setting or in an offline setting. In the former one, no information coming from the future can be exploited to track objects. This is the case for real-time applications, like self-driving cars. The offline setting is more suitable for applications such as road traffic and safety analysis. In this setting, it is common to first associate detections between adjacent frames to create tracklets (continuous fragments of trajectories), which are later associated to form complete trajectories of objects. Usually, given some detections, associations can be made using cues such as appearance (color and texture) and spatio-temporal information (position and motion) . Several offline tracking algorithms (Wojke et al., 2017; Bergmann et al., 2019; Pang et al., 2021) rely on appearance cues, abandoning the motion information. In contrast, many online tracking algorithms (Bewley et al., 2016; Wang et al., 2021a) are only based on motion to make them efficient for real-time applications. In the case of occlusions or missed detections, online trackers generate new boxes using probabilistic methods or by learning multi-modal distributions of trajectories (Saleh et al., 2021; Tokmakov et al., 2021) . This extrapolation may provoke some drifts due to the autoregressive nature of the pre

FAQs

What problem does TWiX solve in multi-object tracking?

TWiX targets the data association task, estimating which track pairs correspond to the same object across temporal windows.

What inputs does TWiX use to compute affinity between tracklets?

It uses only coordinate information from bounding boxes extracted from tracklets, without relying on appearance features.

How does the proposed online cascade matching pipeline perform with C-TWiX?

C-TWiX achieves state-of-the-art results on DanceTrack and KITTIMOT and competitive results on MOT17.

Aug 23, 2026

Uploaded by Mali