GOT-JEPA - Joint-Embedding Predictive Architecture - Model Adaptation and Occlusion Handling

Sep 21, 2026

Uploaded by Jiven

GOT-JEPA presents a model-predictive pre-training framework for generic object tracking that improves robustness under unseen targets, distractors, and dynamic scene changes while reasoning about occlusion at fine granularity. It extends JEPA by having a teacher predictor generate pseudo-tracking models from a clean current frame and a student predictor learn to predict the same pseudo-tracking models from corrupted inputs. This yields stable pseudo supervision and reliable tracking models under adverse observations. Additionally, OccuSolver enhances occlusion perception via object-aware visibility estimation and iterative refinement guided by tracker-generated object priors, improving reference labels and benchmark performance across seven benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Generic object tracking challenges

Tracking-by-detection limitations

Model prediction and generalization

Occlusion and out-of-view handling

GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture Shih-Fang Chen, Jun-Cheng Chen Member, IEEE, I-Hong Jhuo Member, IEEE, and Yen-Yu Lin Senior Member, IEEE arXiv :2602 . 14771v5 [ cs .CV] 16 Jun 2026 Abstract—The human visual system tracks objects by integrating current observations with previously observed information, adapting to target and scene changes, and reasoning about occlusion at fine granularity. In contrast, recent generic object trackers are often optimized for training targets, which limits robustness and generalization in unseen scenarios, and their occlusion reasoning remains coarse, lacking detailed modeling of occlusion patterns. To address these limitations in generalization and occlusion perception, we propose GOT-JEPA, a modelpredictive pre-training framework that extends JEPA from predicting image features to predicting tracking models. Given identical historical information, a teacher predictor generates pseudotracking models from a clean current frame, and a student predictor learns to predict the same pseudo-tracking models from a corrupted version of the current frame. This design provides stable pseudo supervision and explicitly trains the predictor to produce reliable tracking models under occlusions, distractors, and other adverse observations, improving generalization to dynamic environments. Building on GOT-JEPA, we further propose OccuSolver to enhance occlusion perception for object tracking. OccuSolver adapts a point-centric point tracker for object-aware visibility estimation and detailed occlusion-pattern capture. Conditioned on object priors iteratively generated by the tracker, OccuSolver incrementally refines visibility states, strengthens occlusion handling, and produces higher-quality reference labels that progressively improve subsequent model predictions. Extensive evaluations on seven benchmarks show that our method effectively enhances tracker generalization and robustness. The code will be available at [https://github.com/chenshihfang/GOT](https://github.com/chenshihfang/GOT). Index Terms—Generic Object Tracking, Model Prediction, JEPA, Online Learning, Point Tracking, Few-Shot Learning I. INTRODUCTION GENERIC Object Tracking (GOT) [1]–[3] allows machine visual tracking of any arbitrary target object specified by only an initial bounding box in the first frame and predicting its location in subsequent frames. Given this limited information, learning a robust tracker to accurately predict the target location in each frame of a dynamic environment is very challenging, especially in adverse conditions, where unseen target objects, complex distractors, and object deformations are present. Additionally, handling occlusions and out-of-view scenarios is crucial for long-term tracking. Shih-Fang Chen and Yen-Yu Lin are with the Department of Computer Science, National Yang Ming Chiao Tung University (e-mail: [csf.cs09@nycu.edu.tw](csf.cs09@nycu.edu.tw); [lin@cs.nycu.edu.tw](lin@cs.nycu.edu.tw)) (Corresponding author: Y-Y Lin). Jun-Cheng Chen is with the Research Center for Information Technology Innovation, Academia Sinica (e-mail: [pullpull@citi.sinica.edu.tw](pullpull@citi.sinica.edu.tw)). I-Hong Jhuo is with Microsoft AI (e-mail: [ihjhuo@gmail.com](ihjhuo@gmail.com)). Copyright © 2026 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to [pubs-permissions@ieee.org](pubs-permissions@ieee.org). (a) Learn to Build a Robust Model Predictor Visible points (solid) Invisible points (hollow) (b) OccuSolver Allows Predicted Points to Become Object-Aware Fig. 1. (a) GOT-JEPA extends the JEPA architecture to build a robust model predictor for model adaptation. The t-Predictor generates diverse tracking models for pseudo-labeling, while the s-Predictor predicts them using corrupted c

FAQs

What limitation in existing generic object trackers does GOT-JEPA address?

Recent generic trackers are often optimized for training targets, which reduces robustness and generalization on unseen scenarios. Their occlusion reasoning is also relatively coarse and lacks detailed modeling of occlusion patterns.

How does GOT-JEPA train its model predictors?

A teacher predictor produces pseudo-tracking models from a clean current frame, while a student predictor learns to predict the same pseudo-tracking models from a corrupted version of that frame.

What is OccuSolver and how does it improve occlusion handling?

OccuSolver adapts a point-centric point tracker to estimate object-aware visibility and capture occlusion patterns. It iteratively refines visibility states using object priors generated by the tracker, producing higher-quality reference labels for subsequent predictions.

Sep 21, 2026

Uploaded by Jiven