Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Jul 16, 2026

Continuous egocentric video enables assistance that is proactive rather than merely reactive, but prior systems either wait for user queries or respond to detected events without judging whether interruption is appropriate. The work frames proactive help as a context-dependent decision problem requiring temporal reasoning over accumulated history. It introduces Vinci2, a proactive on-device assistant advancing Vinci, and EgoServe, the first large-scale benchmark covering multiple service categories and memory horizons. It also proposes EgoMemo, a training-free memory-augmented agent for retrieval-augmented, context-grounded responses.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

Abstract

Keywords

Introduction

Motivation and problem framing

Limitations of existing paradigms

Proposed approach: Vinci2

Evaluation gap and EgoServe

Modeling approach: EgoMemo

arXiv :2607 . 11523v1 [ cs .CV] 13 Jul 2026 Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos Sitong Gong 1 ,2 ,3, Tianyu Yan⋆1 ,2 ,3, Caixin Kang⋆2 ,3, Bo Zheng2 , Xiang Ruan 1, Huchuan Lu 1, Kaipeng Zhang2, Yoichi Sato3, and Yifei Huang†2 ,3 1 Dalian University of Technology 2 Alaya Lab 3 The University of Tokyo [stgong@mail.dlut.edu.cn](stgong@mail.dlut.edu.cn) , {tianyu.yan, caixin.kang, bo.zheng, [kaipeng.zhang}@shanda.com](kaipeng.zhang}@shanda.com) , {ruanxiang, [lhchuan}@dlut.edu.cn](lhchuan}@dlut.edu.cn) , [ysato@iis.u-tokyo.ac.jp](ysato@iis.u-tokyo.ac.jp) , [hyf015@gmail.com](hyf015@gmail.com) Abstract. When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user’s history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at Vinci2 . Keywords: Egocentric Vision · Proactive VLM · LLM Agent ⋆ Equal contribution.† Corresponding author. 2 Sitong Gong et al. (a) Reactive Paradigm (b) Semi-proactive Paradigm Reaching that high may shake the phone. Please use the side volume button for a steadier shot. You left a file transfer running on your laptop downstairs a few minutes ago. Since you're free now, do you want to check its status? You’re looking for salt. You put it on the kitchen table after buying it the day before yesterday. Memory Construction Multimodal Context Knowledge Graph Episodic Messages (c) Proactive Paradigm (Ours) Egocentric Video Stream Fig. 1: Three paradigms of egocentric assistants. (a) Reactive: responds only to explicit user queries. (b) Semi-proactive: monitors the video stream for predefined taskrelevant events. (c) Proactive (ours): autonomously decides when and how to intervene without any user prompt. 1 Introduction The promise of proactive egocentric intelligence is an assistant that sees what you see, understands your context as it evolves, and offers help at the right moment without being asked. Recent progress in Video-LLMs [30, 33 , 51 , 53 , 54], streaming visual perception [27, 46 , 70], and egocentric foundation models [22, 64 , 69] has brought this vision within reach, enabling continuous comprehension and reasoning over first-person video. Yet while the perception capabilities are maturing rapidly, the question of how and when an assistant should proactively intervene remains largely unaddressed. Existing egocentric assistants operate under two limitin

FAQs

What problem does Vinci2 address in continuous egocentric videos?

It tackles when an intelligent assistant should speak up without being asked, deciding whether and how to intervene based on evolving context and user history.

How does EgoServe support evaluation of proactive assistance?

EgoServe is a large-scale benchmark with service instances organized across multiple temporal memory horizons and service categories, including needs ranging from safety alerts to habit coaching.

What is EgoMemo and how does it determine whether assistance is warranted?

EgoMemo is a training-free, memory-augmented agent that maintains multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives, then performs retrieval-augmented reasoning at each timestep to decide and generate grounded responses.

Jul 16, 2026

Upload by Valentina