RE3 SIM Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation

Jul 16, 2026

Upload by Logic

Real-world robotics data collection is costly, slow, and dependent on skilled operators, while simulations often underperform in sim-to-real transfer due to geometric and visual mismatches. RE3 SIM proposes a 3D-photorealistic real-to-sim system that reconstructs scenes and renders cross-view cameras in a physics-based simulator in real time. Using privileged information to collect expert demonstrations efficiently in simulation and imitation learning to train policies, it achieves zero-shot sim-to-real transfer with average success above 58%. The method also produces a large-scale simulation dataset to improve object-generalized robustness.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Background

3D Reconstruction

arXiv :2502 .08645v4 [ cs .RO] 10 Jul 2026 RE3 SIM: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation Xiaoshen Han 1 ,2 ,∗ Junqiu Yu2 ,∗ Minghuan Liu 1 Yilun Chen2 ,† Xiaoyang Lyu3 Yang Tian2 Bolun Wang2 Weinan Zhang 1 ,† and Jiangmiao Pang2 ,† Website: [https://re3sim.github.io/](https://re3sim.github.io/) Fig. 1: Illustration of RE3 SIM. a) RE3 SIM allows zero-shot policy transfer on various tasks. b) The system pipeline to generate high-quality data. c) High-fidelity rendering results. d) Consistency in success rates between real and simulated environments. Abstract—Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware. Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps. To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE3 SIM, addressing geometric and visual sim-to-real gaps. RE3 SIM employs advanced 3D reconstruction and rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator. By utilizing privileged information to collect expert demonstrations efficiently in simulation, and train robot policies with imitation learning, we validate the effectiveness of the real-to-sim-to-real system across various manipulation task scenarios. Notably, with only simulated data, we can achieve zero-shot sim-to-real transfer with an average success rate exceeding 58% . To push the limit of real-to-sim, we further generate a large-scale simulation dataset, demonstrating how a robust policy can be built from simulation data that generalizes across various objects. I. INTRODUCTION We have witnessed impressive generalization capabilities of robotic models [1]–[3] in certain tabletop or kitchen scenarios, achieved through high-quality teleoperation data. However, collecting real-world expert data [4], [5] remains a 1 Shanghai Jiao Tong University, 2 Shanghai AI Laboratory, 3The University of Hong Kong. * denotes equal contribution.† denotes corresponding authors. time-consuming and costly process. Despite advancementsin teleoperation systems [6]–[8], the labor involved is still intensive, making this a key challenge in robotics research. Recent efforts [9], [10] have explored inter-institutional collaboration to address this issue. In contrast, simulation data offers the advantage of exponential scalability with computational resources, making it an appealing and renewable alternative for training robotic policies. Research on sim-toreal transfer [11],[12] has increasingly focused on generating high-quality simulated or synthetic data to train real-world robotic policies. Unfortunately, simulation data often exhibits significant sim-to-real gaps, making it necessary to collect target domain data for effective policy fine-tuning. In real-to-sim-real scenarios, consistency with the real world in appearance and geometry is essential. High-quality RGB rendering ensures visual consistency between real and simulated domains, while accurate mesh reconstruction captures scene geometry for realistic interactions. To achieve this, we propose a 3D-photorealistic real-to-sim-to-real system, RE3 SIM, short for reconstruction-rendering-based real-to-sim. RE3 SIM faithfully replicates real-world scenarios by integrating photorealistic RGB rendering with precise 3D geometry reconstruction. The visual rendering is realized via Gaussian rasterization [13], while the 3D geometry is reconstructed using multi-view stereo (MVS) techniques [14], [15] . This dual-stage pipeline enables efficient, high-fidelity simulation, where physical dynamics are handled by physics engine backends [16]–[18], and real-time rendering is achieved through a dedicated hybrid rendering engine. Specifically, RE3 SIM adopts a sequ

FAQs

Why is real-world data collection challenging for robotics, and how does RE3 SIM address it?

Real-world expert data collection is time-consuming and costly, requiring skilled operators and expensive hardware. RE3 SIM uses a scalable real-to-sim pipeline to generate high-fidelity simulation data, reducing the need for extensive real data.

What core components enable RE3 SIM to reduce sim-to-real gaps?

RE3 SIM combines photorealistic RGB rendering with accurate 3D geometry reconstruction. It uses Gaussian rasterization for visual rendering and multi-view stereo (MVS) for geometry recovery, then aligns real and simulated coordinates for consistent interaction.

How effective is RE3 SIM for zero-shot transfer, and what evidence is provided?

Zero-shot sim-to-real policies trained using only simulated data achieve an average success rate exceeding 58%. Experiments on multiple tabletop manipulation tasks validate generality and applicability of the approach.

Jul 16, 2026

Upload by Logic