Segmenting Fast and Slow Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing

Jul 16, 2026

Upload by Anda

Object-centric open-vocabulary video instance segmentation inspired by DETR faces a key barrier: real-time on-device inference at high frame rates remains difficult due to the computational cost of dense mask prediction and multimodal fusion. This paper presents SegFS, a dual-stream fast–slow framework that improves efficiency by using a sparse keyframe open-vocabulary object model, then projecting instance representations into backbone feature space to condition a lightweight fast network. Experiments show up to 14× lower latency than MOBIUS while maintaining competitive OV-VIS accuracy.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Abstract

Introduction

Problem: real-time OV-VIS on edge devices

Existing OV-VIS architectures and bottlenecks

Keyframe amortization idea

SegFS Framework Overview

Slow path on sparse keyframes

Fast path on every frame via feature-space conditioning

arXiv :2607 .00124v1 [ cs .CV] 30 Jun 2026 Segmenting, Fast and Slow: Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing Luca Barsellotti3⋆ Martin Sundermeyer 1 Mattia Segu 1 Nikita Araslanov2 Muhammad Ferjad Naeem 1 Marcella Cornia3 Yongqin Xian 1 Maxim Berman 1 1 Google 2 TU Munich; Munich Center for Machine Learning 3 University of Modena and Reggio Emilia Abstract. Object-centric models inspired by DETR have become the dominant paradigm for open-vocabulary video instance segmentation (OV-VIS) . While recent efforts have reduced the computational cost of pixel decoding, textual modality fusion, and object decoding to make these architectures more suitable for mobile devices, real-time on-device inference at high frame rates remains an open challenge. In this paper, we introduce SegFS, a dual-stream fast-slow framework that significantly improves efficiency without sacrificing accuracy. On sparse keyframes, an open-vocabulary object-based model predicts instance-level representations. These representations are then projected back into the backbone feature space to condition a lightweight fast network, which efficiently relocalizes and segments the instances in subsequent frames. By shifting instance propagation from object decoding to feature-space conditioning, our approach decouples multimodal semantic understanding from dense mask prediction and enables efficient temporal propagation. The proposed fast branch achieves up to 14 × lower latency than the mobileoriented MOBIUS model, while maintaining competitive segmentation performance on standard OV-VIS benchmarks. Keywords: Real-Time · Open-Vocabulary · Mobile Device · Video Instance Segmentation 1 Introduction While modern deep learning models exhibit remarkable accuracy and scalability, their substantial computational overhead heavily restricts real-time deployment on edge devices. This bottleneck becomes particularly severe for dense spatiotemporal tasks. In this work, we focus on Open-Vocabulary Video Instance Segmentation (OV-VIS), which aims at identifying and tracking object masks of open-set categories specified by text prompts. Established architectures, such as GLEE [39], comprise three main vision components: (i) a visual backbone that ⋆ Work done during an internship at Google. 2 L. Barsellotti et al. Latency (ms) FLOPS (G) 15 10 5 0 60 40 20 0 MOBIUS-Mini-M 40 20 0 400 200 0 GLEE-Lite 30 20 10 0 60 40 20 0 TROY-VIS Backbone Feature Enhancer Object Decoder SegFS (Ours) Fig. 1: Efficiency analysis of OV-VIS architectures. We compare the FLOPs (top) and on-device latency on a Samsung Galaxy S25 Ultra (bottom) for the core components of three baseline models, MOBIUS-Mini-M [31], GLEE-Lite [39], and TROY-VIS [36], when assuming an input image with resolution 480 ×480 and 40 textual categories, and the Fast Feature Aggregation from our proposed SegFS. produces multi-scale features; (ii) a feature enhancer that fuses the multi-scale features in a single high-resolution feature map and injects the text embeddingsto guide the segmentation process toward identifying corresponding instances; and (iii) an object decoder that extracts representative feature vectors describing the identified instances. Among these, the feature enhancer represents the most severe computational bottleneck. Recent works, such as MOBIUS [31] and TROY-VIS [36], have introduced lighter and more mobile-friendly feature enhancer modifications. However, such components still overwhelmingly dominate the inference cost: Fig. 1 visualizes the FLOPS and the on-device latency on a Samsung Galaxy S25 Ultra of the components of MOBIUS-Mini-M, GLEE-Lite, and TROY-VIS. This dominance is even more pronounced in on-device latency than in theoretical FLOPs, and scales prohibitively with higher image resolutions and vocabulary size. To overcome this bottleneck, we draw inspiration from keyframe-

FAQs

What challenge does OV-VIS face when deploying on edge or mobile devices?

The heavy computational overhead of dense spatiotemporal processing limits real-time deployment at high frame rates. The feature enhancer is identified as a major inference bottleneck, especially as resolution and vocabulary size increase.

How does SegFS improve efficiency without losing accuracy?

SegFS uses a dual-path design: the slow path runs on sparse keyframes to compute instance representations, then projects them back into backbone feature space to condition a lightweight fast network. The fast path performs efficient instance relocalization and segmentation on subsequent frames.

What performance gains does SegFS report compared with mobile-oriented baselines?

SegFS achieves up to 14× lower latency than the MOBIUS-Mini-M model and delivers about 3× amortized FPS, crossing the 30 FPS real-time threshold while preserving competitive segmentation performance (≤1 AP relative to the reference upper bound on standard benchmarks).

Jul 16, 2026

Upload by Anda