3D Point World Models: Point Completion Enables More Accurate Dynamics Learning

Jul 16, 2026

Upload by Jiven

3D Point World Models (3DPWM) presents a task-agnostic world model for robotic control that operates entirely in 3D space. The method completes partial point clouds to recover consistent geometry, then learns action-conditioned dynamics within the completed 3D scene. Operating on completed geometry improves long-horizon rollout reliability and cost evaluation for model-based planning. Experiments across multiple robotic embodiments and tabletop manipulation benchmarks show significantly more stable 100–300+ step rollouts, support for open-loop and closed-loop planning, and sim-to-real transfer success.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Introduction

Motivation and limitations of existing world models

Related approaches: video, 2.5D, and explicit 3D models

Proposed method: 3DPWM pipeline

arXiv :2607 .00148v1 [ cs .RO] 30 Jun 2026 3D Point World Models: Point Completion Enables More Accurate Dynamics Learning Skand Peri, Hung Nguyen, Chanho Kim, Li Fuxin, Stefan Lee Oregon State University Project Page: [https://3dpwm.github.io/](https://3dpwm.github.io/) Abstract: Learning predictive models of the world enables robotic control through planning, potentially allowing robots to improvise solutions on new tasks. However, large video-based dynamics models lack explicit 3D spatial structure and suffer from geometrically inconsistent long-term rollouts with compounding errors. Emerging 3D dynamics models based on partial point clouds improve geometric consistency but remain sensitive to occlusions and accumulated prediction drift. To address these challenges, we present 3D Point World Models (3DPWM)– a task-agnostic world model that operates entirely in 3D space by first completing partial point clouds and then learning action-conditioned dynamics in this completed 3D scene. By operating on completed geometry, 3DPWM enables reliable long-horizon rollouts and more accurate cost evaluation for modelbased planning while supporting adaptation to new tasks. Experiments across different robotic embodiments and tabletop manipulation benchmarks demonstrate that 3DPWM achieves significantly more reliable long-horizon rollouts (100-300+ steps), supports both open-loop and closed-loop planning, and enables successful sim-to-real transfer. Keywords: 3D dynamics learning, point completion 1 Introduction Embodied agents acting in physical environments benefit from having a 3D understanding of the world. To endow such agents with capabilities to reason about their surroundings and plan their actions accordingly, they should be able to not just interpret the current state of the world, but also predict the outcomes of an action that they would take. In other words, agents should learn a reliably accurate world model [1] that facilitates planning in the real world. Prior work in world model-based robot control has predominantly learned task-specific world models from 2D images [2–6], using them for reinforcement learning or model based planning [7–9] . These models typically train both transition and reward functions, which must be fine-tuned or retrained when adapting to new tasks. More recently, large video models [10–12] have emerged by training on vast internet-scale video datasets to learn transition dynamics. However, these models face two critical challenges for real-world deployment. First, they are prone to hallucinating unrealistic or implausible future frames. Second, to enable action execution in physical environments, they require a learned inverse kinematics (IK) model to infer actions from generated videos [13–15] . Learning such robust and generalizable IK models remains challenging due to the inherent ambiguity in action-state correspondences – for example, the same visual motion of a robot arm lifting an object could correspond to very different torque commands depending on the object’s (unobserved) mass or friction properties. Another class of models that has been explored is that of 2.5D video models [16–18] that jointly predict geometric properties such as depth and surface normals alongside RGB frames. They are more grounded in the physical 3D world, however, since such models process these additional features in a Figure 1: Overview. We propose 3DPWM, a task-agnostic world model trained from demonstrations and deployed for planning via model-predictive control. Given a single-view RGB-D observation, the system constructs a partial point cloud, performs point completion, and then rolls out actionconditioned trajectories in 3D space. 2D fashion, they rely on implicit geometric reasoning and still remain prone to multi-view inconsistencies. For example, normals and depth inferred from one camera viewpoint may not match those inferred from another. Furthermore, owing to the dynamics model

FAQs

What problem does 3DPWM address in existing video-based dynamics models?

Video-based models often lack explicit 3D structure and can produce geometrically inconsistent long-term rollouts, with compounding errors. They may also hallucinate implausible futures and require additional components like inverse kinematics to execute actions.

How does 3DPWM improve geometric consistency and planning accuracy?

3DPWM first completes partial point clouds to obtain a completed 3D scene, then learns action-conditioned dynamics using this completed geometry. This makes long-horizon rollouts more reliable and improves cost evaluation for model-based planning.

How is 3DPWM validated, and what capabilities does it support?

Experiments across different robotic embodiments and tabletop manipulation benchmarks show more reliable 100–300+ step rollouts. The approach supports both open-loop and closed-loop planning and enables successful sim-to-real transfer.

Jul 16, 2026

Upload by Jiven