Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator

Jul 16, 2026

Upload by Levi

Embodied navigation seeks agents that understand multimodal goals, reason over 3D space, and reliably reach targets in real environments. Progress is limited by the scarcity of scalable, high-fidelity, physically grounded interactive worlds. Real scans offer realism but lack scale, while synthetic simulators scale easily yet suffer from significant sim-to-real gaps. Image2Sim builds high-quality interactive environments from posed RGB-D sequences using neural 3D feature-Gaussians and geometry-aware panoramic rendering, then scales data generation to large instruction- and action-aligned navigation datasets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

Abstract

Introduction

Related Data Challenges

Image2Sim Framework

Neural Scene Construction

Geometry-Aware One-Step Rendering

Embodied Data Engine and Instruction Generation

arXiv :2607 .05765v 1 [ cs .CV] 7 Jul 2026 Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator Zihan Wang1 , Seungjun Lee1 , Yinghao Xu2 , Gim Hee Lee1 1National University of Singapore, 2HKUST [zihan.wang@u.nus.edu](zihan.wang@u.nus.edu) , [gimhee.lee@nus.edu.sg](gimhee.lee@nus.edu.sg) Abstract Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into near 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale. Project page: [github.com/MrZihan/Image2Sim](github.com/MrZihan/Image2Sim) 1 Introduction Large language models [1–3], vision foundation models [4–6], and visual generative models [7–9] suggest that scaling data can lead to substantial gains in capability. However, similar scaling in embodied navigation [10–14] has not delivered comparable progress in robust real-world generalization, largely due to the lack of suitable data. A key limitation lies in the availability of scalable sources of high-fidelity, physically grounded, and interactive 3D environments. Existing data sources expose a fundamental tradeoff between visual fidelity and scalability. Training data is typically drawn from two primary sources: real-world 3D scans [15–18] and synthetic procedural environments [19–21] . Real-world scans provide strong visual fidelity, but are expensive to acquire and difficult to scale. Procedural environments improve scalability, but often introduce substantial sim-to-real gaps due to unrealistic assets, layouts, and rendering statistics. Recent generative video models [22–24] offer another potential source of realistic visual data. However, they struggle to maintain the rigid-body consistency and explicit collision structure required for reliable closed-loop interaction. These limitations suggest that scalable embodied data requires both real-world visual fidelity and explicit 3D physical grounding. Preprint. (a) Traditional Navigation Data Pipeline Scanned or Synthetic 3D Meshes (b) Image2Sim Neural Simulator (Ours) Labor-intensive Human Annotation Figure 1: Comparison of (a) traditional navigation data pipeline and (b) our Image2Sim framework. These failure modes share a common cause: existing pipelines often couple 3D geometry and visual synthesis within a single mechanism, forcing a tradeoff among scale, physical grounding, and visual fidelity. To overcome this tradeoff, we introduce Image2Sim, a real-time neural environment engine that c

FAQs

What data does Image2Sim generate at scale for training?

It generates near 20K interactive scenes and synthesizes more than 10 million navigation training samples, including executable actions and navigation instructions aligned with generated trajectories, enabling effective training and real-world zero-shot transfer.

Jul 16, 2026

Upload by Levi