ROBOSNAP One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation

Jul 16, 2026

Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. ROBOSNAP presents a real-to-sim framework that converts a single RGB image into a simulation-ready scene using a layered design: collision-aware interaction assets for stable robot contact and a 3D Gaussian splatting visual layer to preserve faithful background appearance. Experiments on DROID scenes and real-robot tasks validate reliable trajectory replay, task-specific synthetic data generation, and meaningful sim-real correlation via the new DROID-Sim dataset.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

Introduction

Related Work

3D Generation and Scene Synthesis

􀂃ROBOSNAP: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation Shujie Zhang 1 4 * Jingkun Yi 1 3 * Weipeng Zhong 1 2 Zirui Zhou4 Yangkun Zhu 1 Hanqing Wang 1 Xudong Xu 1 † Weinan Zhang 1 2 Chunhua Shen 1 3 1 Shanghai AI Laboratory 2 Shanghai Jiao Tong University 3Zhejiang University 4Tsinghua University Homepage: [https://robosnap.github.io](https://robosnap.github.io) arXiv :2607 .06699v 1 [ cs .RO] 7 Jul 2026 Real Sim Real Sim Figure 1: From a single RGB image, ROBOSNAP reconstructs a reusable simulation-ready scene with interactive physical assets and visual context. The recovered scenes support trajectory replay (top-left), task-specific data generation and augmentation (top-right, bottom-left), and policy evaluation with meaningful sim-real correlation (bottom-right) . Abstract: Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. In this work, we present ROBOSNAP, a real-to-sim framework that turns a single RGB image into a simulation-ready scene. The key idea is a layered design that separates the physics-critical interaction area from the surrounding visual context: collision-aware foreground assets are refined for stable robot interaction, while a 3D Gaussian splatting visual layer preserves faithful background appearance under novel views. Experiments on DROID scenes and real-robot tasks show that ROBOSNAP achieves reliable trajectory replay in therecovered scenes, supports task-specific synthetic data generation for policy training, and yields meaningful sim-real correlation for policy evaluation. To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID. Extensive experiments suggest that the value of real-to-sim methods lies not only in high-fidelity visual *Equal contribution. †Corresponding author. reconstruction, but in turning real environments into reusable infrastructure for robot learning and evaluation. Keywords: Real-to-Sim-to-Real, Robot Data Generation, Vision-LanguageAction Models, Robot Manipulation 1 Introduction Recent robot foundation models formulate manipulation as a conditional action generation task from visual, linguistic, and proprioceptive inputs [1, 2, 3, 4, 5] . As these models scale, large-scale training data and reproducible evaluation have become critical bottlenecks. Although real-world datasets and benchmarks provide substantial physical demonstrations and standardized protocols [6, 7, 8, 9], scalable data acquisition and flexible policy evaluation remain costly, labor-intensive, and hardwarebound. Simulation offers a complementary path for scalable data synthesis, scene augmentation, and repeatable policy assessment, thereby motivating the construction of interactive simulation scenes that are both physically plausible and visually faithful to real-world deployment environments. Existing approaches only partially satisfy these requirements. Procedural and generative scene synthesis methods have scaled simulatable environments for robot learning [10, 11, 12, 13, 14], but they primarily focus on creating diverse simulation scenes rather than recovering reusable interactive replicas of specific in-the-wild real-world scenes. Reconstruction-based real-to-sim methods improve scene alignment but often require multi-view capture or manual refinement [15, 16, 17, 18, 19] . Recent single-image systems reduce the capture burden [20, 21, 22], but their outputs typically target narrower endpoints: retrieval-based digital cousins, task or demonstration synthesis, or partially recovered scenes with static background. As a result, they do not generally recover persistent simulation worlds that can be re-rendered, edited, and reused from new viewpoints,

FAQs

What is ROBOSNAP and what problem does it address?

ROBOSNAP is a one-shot real-to-sim framework that turns a single RGB image into a simulation-ready scene. It targets the need for physically stable and visually faithful environments that can support generalizable robot learning and reproducible evaluation.

How does ROBOSNAP build a simulation-ready scene from one RGB image?

It reconstructs the interaction area as collision-aware objects and support surfaces, aligns it to a gravity-consistent frame, and refines object poses to fix floating artifacts and unstable contacts. The surrounding context is modeled as a separate visual layer using background completion, 3D Gaussian splatting, and scene lighting.

What does DROID-Sim contribute to real-to-sim research?

DROID-Sim is a real-to-sim companion dataset built from 564 real-world scenes in DROID. It provides reusable simulation environments, enabling experiments that evaluate trajectory replay, task-specific synthetic data generation, and sim-real correlation for policy evaluation.

Jul 16, 2026

Upload by River Wang