GraspIT A Dataset Bridging the Sim to Real gap and back for Validated Grasping SE(3) Pose Generation

Jul 16, 2026

Upload by Miles

Robust robotic grasping of novel objects requires datasets that pair photorealistic RGB-D observations with physically validated grasp-quality annotations and a principled simulation-to-real bridge. GraspIT closes this gap by generating tabletop scenes in NVIDIA Isaac Sim, validated through a four-stage physical slip test on parallel Franka Panda instances, yielding trajectory reachability checks and continuous quality scores beyond force-closure. From ~2.3M candidates, 83% pass (s≥0.50), while force-closure-only failures form graded hard negatives. A real↔sim projection back-projects labels onto 100 real scenes, releasing ~316k annotated RGB-D frame sets with masks, 6-DoF poses, physical properties, scored 6-DoF grasps, and open-source Docker tools supporting high-resolution demonstrations for learning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Abstract

Introduction

Related Work

6-DoF Object Pose Datasets

Grasping Datasets

GraspIT: A Dataset Bridging the Sim-to-Real gap and back for Validated Grasping SE(3) Pose Generation Paul Koch Fraunhofer IPK Pascalstraße 8-9, 10587 Berlin, Germany [paul.koch@ipk.fraunhofer.de](paul.koch@ipk.fraunhofer.de) Adem Karakurt Fraunhofer IPK Andr Sers Fraunhofer IPK arXiv :2607 .05869v 1 [ cs .RO] 7 Jul 2026 Abstract Robust robotic grasping of novel objects requires datasets that simultaneously provide photorealistic RGB-D observations, physically validated grasp quality annotations, and a principled bridge between simulation and the real world, which existing datasets lack to provide jointly. GraspIT addresses this gap: tabletop scenes in NVIDIA Isaac Sim are annotated via a four-stage physical slip-test on parallel Franka Panda instances, producing trajectoryreachability checks and continuous quality scores beyond force-closure.Of ∼2.3M candidates, 83% pass as good (s≥0 .50); the 17% that passed force-closure but failed the slip-test provide graded hard negatives. A Real↔Sim loop back-projects these labels onto 100 real-world scenes. The release provides ∼ 316k annotated RGBD frame sets across 1035 sim and 100 real scenes, with instance masks, 6-DoF poses, physical object properties, and scored 6-DoF grasps. All tools are open-source and Docker-containerized. The trajectory planning within Isaac Sim further allows streaming of high resolution demonstrations for tabletop manipulation policy learning and behavior cloning. 1. Introduction Robotic manipulation of everyday objects—picking, placing, stacking, and handing over — remains an unsolved challenge despite decades of research [11, 32] . Recent large-scale imitation learning systems such as RT-2 [3],π0 [1], and Octo [29] demonstrate multi-task instructionfollowing in laboratory settings; yet, their generalization collapses when confronted with objects, scenes, or lighting conditions outside the training distribution. A central root cause is the absence of physically grounded visual representations: the encoders driving these systems are trained on semantic similarity objectives (CLIP [31], SigLIP [37]) that do not encode metric geometry, 3D object understanding, or the physical grounding that determines whether a Figure 1 . Grasp ITerative robot tabletop object manipulation dataset: illustration of the four-stage iterative concept. proposed grasp or object manipulation will succeed [8, 41] . Collecting the training data needed to close this gap is expensive. Accurate 6-DoF grasp annotation requires either real-robot execution at scale or a physics simulator; real RGBD observations necessitate a physical setup; linking the two so that simulation annotations are valid for real sensor data requires a careful calibration pipeline. No existing dataset provides all of these simultaneously (see Tab. 1) . We introduce GraspIT (Fig. 1), a dataset and generation system that addresses this gap through four principled design decisions (Fig. 2): 1) Quality over quantity. Every grasp candidate is validated by a four-stage physical slip test executed by a simulated Franka Panda arm (Fig. 6), producing a continuous quality score s ∈ {0, 0.25 , 0.50 , 0.75 , 1.0} that captures failure modes (slip under gravity, oscillation, pendulum swing) that are absent from force-closure metrics. 2) Robot-in-the-loop reachability filtering. Every candidate is validated for collision-free trajectory reachability by the robot arm before the slip test, ensuring that all positive labels are actionable. 3) Real↔Sim bidirectional link. Real-world scene geometry is captured via a robotmounted camera and registered into Isaac Sim; simulationvalidated labels are back-projected onto all real camera frames. This means a single real-world capture session yields both real RGB-D images and unlimited additional photorealistic synthetic views with full annotation (Fig. 5) . 4) Open, containerized, and extendable. All generation tools are packaged as Docker containers, enabling

FAQs

What problem does GraspIT aim to solve in robotic grasping datasets?

It addresses the lack of datasets that jointly provide photorealistic RGB-D observations, physically validated grasp quality annotations, and a calibrated bridge between simulation and real sensors.

How does GraspIT validate grasp quality beyond force-closure?

Each grasp candidate undergoes a four-stage physical slip test on a simulated Franka Panda arm, producing continuous quality scores that capture failure modes absent from force-closure metrics.

How are simulation labels made valid for real-world scenes in GraspIT?

Real scene geometry is captured with a robot-mounted camera, registered into Isaac Sim, and simulation-validated labels are back-projected onto all frames from the real capture session.

Jul 16, 2026

Upload by Miles