OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization

Jul 16, 2026

Upload by Logic

Temporal Action Localization (TAL) is often constrained by segment annotations or offline access to full videos, which limits scalability and online deployment. The work introduces Point-Supervised Online TAL (POTAL), localizing actions in streaming videos using only one temporal point per instance. It presents OnPoint, an offline-to-online multi-level distillation framework transferring knowledge from a point-supervised offline teacher through pseudo-segment distillation, class-activation sequence distillation, and anticipatory window-level distillation, with robustness via point-label integration and actionness-guided attention calibration. Experiments on five datasets show consistent gains over strong baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages34

Reading Time1 h 26 min

Introduction

Temporal action localization and online constraints

Supervision cost and point supervision

POTAL task and OnPoint approach

arXiv :2607 .00289v 1 [ cs .CV] 1 Jul 2026 OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization Sakib Reza 1 ,3⋆, Gauri Jagatap2, Mohsen Moghaddam3, Octavia Camps 1, and Andrea Fanelli2 1 Northeastern University, Boston, MA 02115, USA {reza.s, [o.camps}@northeastern.edu](o.camps}@northeastern.edu) 2 Dolby Laboratories, Inc. , San Francisco, CA 94103, USA {gauri.jagatap, [andrea.fanelli}@dolby.com](andrea.fanelli}@dolby.com) 3 Georgia Institute of Technology, Atlanta, GA 30332, USA {sreza32, [mohsen.moghaddam}@gatech.edu](mohsen.moghaddam}@gatech.edu) Abstract. Temporal Action Localization (TAL) typically relies on segment annotations or offline access to full videos, limiting scalability and online use. We introduce Point-Supervised Online TAL (POTAL), which localizes actions in streaming videos using only one temporal point per instance. To solve POTAL, we propose OnPoint, an offline-to-online multi-level distillation framework that transfers knowledge from a pointsupervised offline teacher to an online student via (i) pseudo-segment instance distillation, (ii) class-activation sequence distillation, and (iii) anticipatory window-level distillation. We further improve robustness by incorporating the original point labels into student training and by refining anchor decoding with actionness-guided attention calibration. Experiments on five datasets show OnPoint consistently outperforms strong baselines, establishing a solid foundation for POTAL. † Keywords: Video Understanding · Temporal Action Localization 1 Introduction With the rapid growth of video platforms, Temporal Action Localization (TAL) [42, 45], which identifies action boundaries and class labels in untrimmed videos, has become a central task in computer vision. However, many real-world systems must localize actions directly from streaming video while minimizing annotation cost. Examples include augmented reality (AR) task assistance for surgical training or industrial maintenance [2,51], as well as live sports analytics [37], surveillance [41], and robotics [10] . In such settings, models must make frame-by-frame predictions without future context while operating under limited supervision, as dense action boundary annotation is often costly and infeasible at scale. ⋆ Work primarily done during an internship at Dolby Laboratories.† Project Page: [https://sakibreza.github.io/OnPoint/](https://sakibreza.github.io/OnPoint/) 2 S. Reza et al. A key challenge arises from the inference regime. Most TAL methods [8, 20, 21,44,46–48,50,52] assume offline access to complete videos, allowing models to leverage future context during training and inference. In contrast, real deployments require online inference, where actions must be localized in streaming video as frames arrive. Recent Online Temporal Action Localization (OnTAL) approaches [12, 34] address this setting but rely on full segment-level supervision during training, limiting scalability in continuously recorded environments where annotation is scarce. A second challenge concerns the supervision cost. Training state-ofthe-art TAL or OnTAL models typically requires dense temporal annotations specifying action start and end boundaries, which are expensive to obtain. Recent large multimodal foundation models [4, 28, 32] show promising zero-shot capabilities but remain unreliable for precise temporal action instance detection and boundary localization (Fig. 1) . Consequently, fully automatic label generation remains impractical for many real-world deployments. A practical alternative is point supervision, where each action instance is annotated with a single timestamp. Prior work [27] and our supplementary study (Supp. Sec. C) show that point labeling can reduce annotation effort by up to 6 × while maintaining reliability. This paradigm has been explored in Point-Supervised TAL (PSTAL) [27, 53], which learns action boundaries from spars

FAQs

What limitation motivates Point-Supervised Online Temporal Action Localization (POTAL)?

Most TAL methods assume offline access to complete videos and dense segment supervision, which hinders scalable, streaming deployment. POTAL targets streaming inference while reducing annotation cost by using only one temporal point per action instance.

How does OnPoint transfer knowledge from the offline teacher to the online student?

OnPoint uses offline-to-online multi-level distillation, including pseudo-segment instance distillation, class-activation sequence distillation, and anticipatory window-level distillation, so the online model can learn structured supervision under streaming constraints.

What techniques improve robustness and decoding quality in the online student?

The framework incorporates the original point labels into student training and refines anchor decoding using actionness-guided attention calibration, improving boundary localization reliability in the online setting.

Jul 16, 2026

Upload by Logic