ReflectWorld-MM: An Entity-Oriented Multi-Media Memory System for Open-Ended Video Streams

Jul 16, 2026

Upload by Hazel

Long-term video assistants that continuously watch, remember observations, and reason over past experiences remain difficult due to memory designs that store frame- or token-based features inside limited contexts. ReflectWorld-MM presents an entity-oriented multi-media memory system for open-ended video streams. It includes an entity-resolving perception front-end, a hierarchical long-term memory unifying multi-scale episodic, evolving entity-centric semantic, and procedural memory, and a persistent indexed realization for real-world assistant integration.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Introduction

Background and limitations of existing memory designs

Prior multimodal long-term memory approaches

ReflectWorld-MM overview and core components

ReflectWorld-MM: An Entity-Oriented Multi-Media Memory System for Open-Ended Video Streams Xiaokang Ma1 , Yifan Sun2,* , Zhihong Jin3,* , Jie Gu1,†, Yudong Luo1 , Shenyi Shao1 Chu Tang1 , Jingmin Chen1 , Li Pu1 1Rightly Robotics 2Hangzhou Institute for Advanced Study, UCAS 3Zhejiang University arXiv :2607 .09759v 1 [ cs .CV] 6 Jul 2026 Abstract Building assistants that can continually watch the world, remember what they see, and reason over their accumulated experience is a long-standing goal, and recently multimodal agents equipped with long-term memory over video streams have attracted increasing interest. Unfortunately, existing systems either keep their memory inside the model context or ina flat feature store, and organize it around frames rather than around the persistent entities a stream is really about, which confines them to bounded videos and weakens their ability to track who and what reappears over time. In this paper, we propose ReflectWorld-MM, an entity-oriented multi-media memory system for open-ended video streams. It consists of three parts. The first is a perception front-end that turns a streaming video into entity-resolved observations under abounded shortterm memory. The second is a hierarchical long-term memory, grounded in human memory theory, that couples a multi-scale episodic memory, an evolving entity-centric semantic memory, and a procedural memory. The third is a complete realization, built for real-world operation, that ingests arbitrary streams and plugs into off-the-shelf assistants. Across six longvideo and lifelong-memory benchmarks, ReflectWorld-MM achieves the best accuracy on all six, outperforming strong memory agents and a frontier model. 1 Introduction An intelligent assistant that keeps watching the world should also be able to remember it. Cameras are now everywhere, from wearable glasses to household robots and smartphones. There is thus a growing need for systems that perceive a continuous multi-media stream, accumulate what they observe into a long-term memory, and answer questions or act upon that memory at any later time. This is the difference between a video-understanding model, which reasons over one clip, and an assistant that must recall who appeared yesterday and how a situation has changed since. A large proportion of current approaches store this memory implicitly, inside the model. Streaming and long-video models compress frames into a memory bank, a sparse token cache, or a key–value cache that is read back during decoding (He et al. 2024; Song et al. 2024; Zhang et al. 2025; Diet al. 2025; Chen et al. 2024) . These methods are effective on videos of bounded length, but their memory is single-scale *Interns at Rightly Robotics. †Corresponding author. Figure 1: From egocentric video to entity-oriented multimedia memory. A person wearing Meta glasses wandersan office, producing a live video stream. ReflectWorld-MM perceives the stream continuously and writes memory. The OpenClaw assistant can then answer the question about the water cup by retrieving the matching memory. and content-agnostic, organized by frame or token rather than by entity and kept in the model context or in main memory. Accordingly, it degrades once the stream grows far beyond the videos the model was tuned on. In contrast, a separate line of work studies explicit memory for language agents, and decomposes it into episodic, semantic, and procedural components in the spirit of cognitive theory (Packer et al. 2023; Park et al. 2023; Xu et al. 2025; Chhikara et al. 2025; Zhong et al. 2024) . Unfortunately, these systems are text-only and conversation-driven: they neither perceive a video stream nor resolve the visual entities that a multi-media memory must be built around. The closest works to ours equip a multimodal agent with an explicit long-term memory over video. M3-Agent organizes memory as an entity-centric multimodal graph and reasons over it with a multi-turn retr

FAQs

What problem does ReflectWorld-MM address in long-video multimodal agents?

It addresses the inability of existing systems to organize memory around persistent entities, leading to poor tracking and degraded performance when streams exceed the bounded training setup.

How does ReflectWorld-MM turn a video stream into memory-relevant representations?

A perception front-end converts streaming video into entity-resolved observations using working context, scene understanding, and entity history so past information participates in interpreting the present.

What are the three main components of ReflectWorld-MM’s memory system?

It combines (1) entity-resolved perception, (2) a hierarchical long-term memory with multi-scale episodic, evolving entity-centric semantic, and procedural memory, and (3) a complete persistent realization that plugs into off-the-shelf assistants for arbitrary streams.

Jul 16, 2026

Upload by Hazel