S-EMBER - A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval

Sep 21, 2026

Uploaded by Maeve

S-EMBER introduces a large-scale diagnostic benchmark for streaming egocentric episodic retrieval, addressing the mismatch between offline video-based evaluation and the continuous reality of wearable intelligence. Built from 3,141 Ray-Ban Meta smart-glasses videos totaling 388 hours, it models causal, active recall triggered by online visual events. The dataset includes 9,448 QA pairs with manual visual proof via precise temporal localization and supports flexible response lengths. Benchmarking shows a localization paradox: semantic reasoning benefits from model scale while temporal grounding precision remains a stagnant architectural bottleneck despite higher resolution and denser frames.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

Streaming egocentric episodic retrieval

Dataset construction and queries

Cognitive taxonomy and response granularity

Evaluation findings and localization paradox

arXiv :2607 .02689v 1 [ cs .CV] 2 Jul 2026 S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval Xiaodong Wang 1 , Xuanyi Zhao 1 , Pedro Rodriguez 1 , Devendra Singh Sachan 1 , Barlas Oguz 1 , Seungwhan Moon2 , Shang-Wen Li 1 , Gargi Ghosh 1 , Xin Dong2 , Wen-Tau Yih 1 1 FAIR, Meta, 2 Reality Labs, Meta As wearable devices enable continuous first-person recording, AI assistants must reason across longtime horizons to recall past experiences—a capability known as episodic memory. Current benchmarksoften rely on offline evaluation with access to entire video files, failing to simulate the streaming reality of wearable intelligence. We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses. S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream. We provide 9,448 QA pairs requiring manual visual proof through precise temporal localization and supporting flexible response lengths to simulate natural human-AI interaction. Our extensive benchmarking of frontier models uncovers a localization paradox: while semantic reasoning improves with parameter scale, temporal grounding precision remains a stagnant architectural bottleneck that does not benefit from brute-force increases in model size, resolution, or frame density. S-EMBER establishes a hardware-authentic foundation for developing grounded, reliable episodic memory in the next generation of wearable AI agents. Correspondence: Xiaodong Wang at [xiaodongwang@meta.com](xiaodongwang@meta.com), Scott Yih at [scottyih@meta.com](scottyih@meta.com) Code: [https://github.com/facebookresearch/S-EMBER](https://github.com/facebookresearch/S-EMBER) Dataset: [https://huggingface.co/datasets/facebook/S-EMBER](https://huggingface.co/datasets/facebook/S-EMBER) 1 Introduction Accurately recalling information about events and reasoning about their temporal relationships—i.e., episodic memory (Tulving, 1972)—is a crucial capability for personal assistants that aim to process always-on video of everyday events. For example, suppose we want an agent to answer “what ingredients do I need to pick upon the way home to bake white chocolate ube cookies again and where can I buy them?” With the rapid growth of consumer smart glasses 1 , assistants could have access to videos of (1) the previous recipe and recording of baking cookies,(2) the contents of ingredients at home to compare to the recipe, and (3) what stores stocked these ingredients last time. In this example, the assistant must navigate temporal dependencies spread across disparate video segments while filtering for contextually relevant details. To bridge the gap between current offline models and this streaming necessity, we introduce a new egocentric benchmark designed to measure how effectively models locate and utilize information within long-form visual histories (§3) . To address these requirements, we introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale diagnostic foundation comprising 3,141 unique videos totaling 388 hours of organic human activity. Captured via consumer-grade Ray-Ban Meta smart glasses by 613 unique users, S-EMBER provides a hardware-authentic, head-centered perspective that mirrors the actual direction of human attention. By moving beyond the fixed, body-mounted rigs characteristic of traditional egocentric datasets, S-EMBER ensures the visual stream aligns with the dynamic, everyday environments that a personal AI assistant must realistically navigate. Beyond its authentic perspective, S-EMBER fundamentally shifts the evaluation paradigm from the static search found in current VideoQA benchmarks to a causal, streaming recall. We formalize this as grounded stream

FAQs

What problem does S-EMBER address in existing egocentric memory benchmarks?

It addresses the gap between offline evaluation that assumes access to entire video files and the streaming reality of wearable devices, requiring causal recall triggered by current visual context.

How is S-EMBER constructed and what data does it include?

S-EMBER comprises 3,141 unique videos totaling 388 hours of organic activity captured by Ray-Ban Meta smart glasses. It provides 9,448 QA pairs with manual visual proof through temporal localization.

What is the localization paradox reported by the benchmark results?

As model scale increases, semantic reasoning accuracy improves, but temporal grounding precision stays nearly stagnant, indicating temporal localization does not improve simply by increasing model size, resolution, or frame density.

Sep 21, 2026

Uploaded by Maeve