RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Jul 16, 2026

Upload by Violet

Generalist robot manipulation policies have advanced, yet existing benchmarks cannot systematically and comprehensively test key policy capabilities. Current benchmarks often use short-horizon or skill-narrow tasks with similar manipulation patterns and limited capability dimensions, and they typically evaluate either in simulation or in the real world alone—simulation is efficient but misses physical deployment challenges, while real-world testing is direct but costly, slow, and hard to reproduce. RoboDojo unifies sim-and-real evaluation with 42 simulation tasks and 18 real-world tasks, covering complementary capabilities and deployable conditions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages46

Reading Time1 h 56 min

Introduction

Related Work

RoboDojo Benchmark

Simulation Benchmark

Real-World Benchmark

Evaluation Integrity and Anti-Gaming Protocols

Technical Implementation

RoboDojo Leaderboard

Experiments

Future Extensions

Conclusion

2026-7-5 RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies Tianxing Chen1∗§ Yue Chen4∗§† Zixuan Li3∗ Junyuan Tang3∗ Kailun Su3∗ Haoran Lu4∗ Weijie Wan3∗ Baijun Chen1∗ Songling Liu4∗ Haowen Yan3 Honghao Su3 Zhiyang Dou6 Kaixuan Wang1 Dandan Zhang13 Yunze Liu3 Yan Qin16 Qiwei Liang16 Qiwei Wu16 Zijian Lin3 Wenwei Lin3 Yuran Wang10 Minghua He4 Tianshu Wu4 Ruihai Wu4 Jingquan Zhou18 Kai-Chong Lei3 Haibao Yu1 Yuanfeng Ji5 Weiyang Jin1 Guanyu Lin9 Xiaofan Li17 Qi Xiong3 Renjing Xu16 Zhongyu Li12 Wenhao Chai8 Enze Xie1 Ziwei Wang11 Yao Mu14 Hao Dong4 Wojciech Matusik6 Mingyu Ding7† Wenbo Ding3† Ping Luo1† Masayoshi Tomizuka2† 1MMLab@HKU 2UC Berkeley 3THU 4PKU 5 Stanford 6MIT 7UNC 8Princeton 9 CMU 10NUS 11NTU 12 CUHK 13IC 14 SJTU 15NU 16HKUST (GZ) 17ZJU 18Yale ∗ Co-first Authors § Co-project Leaders †Corresponding Authors arXiv :2607 .04434v 3 [ cs .RO] 8 Jul 2026 [Website:](Website: RoboDojo-Benchmark.com)[ RoboDojo-Benchmark.com](Website: RoboDojo-Benchmark.com) Code: Benchmark, XPolicyLab Document Leaderboard Unified Benchmark Generalization Memory Long-Horizon Precision Open Simulation Real-World 18 Tasks · 3 Embodiments Reproducible : Layout, Light, Robot Setup Hardware Facing Real-World Challenges 42 Tasks · 5 Dimensions Figure 1: Overview of RoboDojo. RoboDojo unifies efficient simulation evaluation and reproducible real-world testing for generalist robot manipulation, covering 42 simulation tasks, 18 real-world tasks, heterogeneous parallel simulation, RoboDojo-RealEval, XPolicyLab, and a continuously updated leaderboard. Contents 1 Introduction 3 2 Related Work 4 2.1 Robot Learning in Manipulation ........................................ 4 2.2 Evaluation for Robot Manipulation ....................................... 4 3 RoboDojo Benchmark 5 3.1 Simulation Benchmark ............................................. 5 3.1.1 Task Design ............................................... 5 3.1.2 Training Data Setting ........................................... 7 3.1.3 Evaluation Setting ............................................ 7 3.2 Real-World Benchmark .............................................. 7 3.2.1 Task Design ............................................... 8 3.2.2 Training Data Setting .......................................... 8 3.2.3 Evaluation Setting ........................................... 9 3.3 Evaluation Integrity and Anti-Gaming Protocols ................................ 9 4 Technical Implementation 10 4.1 Simulation Platform ............................................... 10 4.1.1 Simulation Platform Setup ....................................... 10 4.1.2 Physically Grounded Digital Asset Library .............................. 10 4.1.3 Heterogeneous Parallelism ....................................... 10 4.1.4 Simulation Data Collection ........................................ 11 4.2 Real-World Platform: RoboDojo-RealEval .................................... 11 4.3 XPolicyLab ................................................... 12 5 RoboDojo Leaderboard 12 5.1 Simulation Benchmark Leaderboard ...................................... 12 5.2 Real-World Benchmark Leaderboard ...................................... 13 6 Experiments 13 6.1 Analysis of RoboDojo Simulation Performance ................................ 14 6.2 Analysis of RoboDojo Real-World Performance ................................. 17 6.3 Evaluation Efficiency .............................................. 18 6.3.1 Simulation Evaluation Efficiency ................................... 18 6.3.2 Real-World Evaluation Efficiency ................................... 19 6.4 Evaluation Stability ............................................... 19 6.4.1 Simulation Evaluation Stability .................................... 19 6.4.2 Real-World Evaluation Stability .......................

FAQs

What limitations do existing robot manipulation benchmarks have that RoboDojo addresses?

They often focus on short-horizon or skill-narrow tasks with limited capability dimensions, and they evaluate either simulation or real-world settings separately. RoboDojo targets comprehensive, systematic assessment across complementary conditions.

How many tasks does RoboDojo include for simulation and for real-world evaluation?

RoboDojo provides 42 simulation tasks and 18 real-world tasks. These tasks are designed to cover diverse, challenging, and complementary manipulation capabilities.

What capabilities does RoboDojo measure in the simulation benchmark?

The simulation benchmark evaluates five capability dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following.

Jul 16, 2026

Upload by Violet