Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

Jul 16, 2026

Evaluating end-to-end autonomous driving (E2E-AD) remains difficult because existing simulators often force a trade-off between closed-loop interactivity and real-world visual fidelity. Point as Skeleton introduces a generative sensor simulation framework that produces driving video observations via a state-updated autoregressive generator using ego states, actor states, scene maps, and point-cloud skeleton conditions. Resetand-Roll adapts rolling diffusion for simulation rollouts, while point-cloud skeletons stabilize stepwise error accumulation. A nuPlan-based renderer-level closed-loop interface enables evaluation under ego deviations from logs, validated on nuScenes and nuPlan.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Introduction

Spatiotemporal Extrapolation

Instantaneous Interactivity

Point as Skeleton Generative Simulator

arXiv :2607 .065 16v 1 [ cs .CV] 7 Jul 2026 Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation Songbur Wong 1 , Xiaosong Jia3✉ , Junqi You 1 , Bo Zhang4 , Pei Xu2 Renqiu Xia 1 , Yuping Qiu2 , Shaofeng Zhang5 , Zelin Zhao6 , Xuechao Yan2 Yuchen Zhou3 , Yurui Chen2 , Wen Guo2 , Hang Xu2 , Junchi Yan 1✉ 1 Shanghai Jiao Tong University 2Yinwang Intelligent Technology Co., Ltd. 3Fudan University 4 Shanghai Artificial Intelligence Laboratory 5University of Science and Technology of China 6 Georgia Institute of Technology ✉ Corresponding authors Abstract Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes) . We present Point as Skeleton, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and pointcloud skeleton conditions. To support closed-loop rollout, we introduce Resetand-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that Point as Skeleton improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at [https://github.com/krauwu/point-as-skeleton](https://github.com/krauwu/point-as-skeleton). 1 Introduction End-to-end autonomous driving (E2E-AD) [1, 2, 3, 4, 5] has attracted increasing attention in recent years. A central challenge in evaluating E2E-AD systems is to provide sensor observations that respond to the actions selected by the driving policy. Open-loop benchmarks such as nuScenes [6] provide realistic sensor data, but the observations are fixed by the recorded trajectory and cannot react to policy-induced actions. In contrast, closed-loop simulators such as CARLA [7, 8, 9] provide interactive feedback, but their rendered sensor appearance can still differ from real-world camera data. This motivates generative sensor simulation, where a learned generative model synthesizes camera observations from simulator-provided ego states, actor states, and scene conditions. However, closed-loop simulation goes beyond standard driving-scene reconstruction [10] and generation [11, 12] . The ego vehicle may deviate from the logged trajectory, while the planner and traffic simulator update online. The generator therefore needs to render multi-view observations under off-log motions and repeatedly refreshed conditions. This introduces two additional requirements:(I) Spatiotemporal Extrapolation. Closed-loop policies may deviate from the logged motion in speed or heading, exposing vehicles and pedestrians that are not observed in the recorded images. Preprint. | | Multi-view Image<br>Point Cloud & 3D Bbox<br>Agent Track ID | Paste<br>Simulator: Transportation Flows Projection Matrices | | --- | --- | --- | | Log Trajectory Simulation Trajectory | | | Generator Point Cloud Skeleton Controlling Transportation Simulator Camera Control Generation E2E-AD planner Figure 1: Point as Skeleton Generative Simulator. We utilize offline logs to build a foregroundbackground decoupled Point Cloud Skeleton. At each time step, the ev

FAQs

Why is closed-loop autonomous driving evaluation harder than open-loop benchmarks?

Open-loop benchmarks use fixed recorded sensor observations tied to the logged trajectory, so they cannot react to policy actions. Closed-loop simulation is interactive, but its rendered sensor appearance may still differ from real-world camera data, motivating generative sensor simulation.

What is the core idea of Point as Skeleton?

Point as Skeleton uses a generative sensor simulation framework where an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. Offline accumulated point-cloud assets provide stabilizing geometric cues for rollout.

How does Resetand-Roll support closed-loop rollout with rolling diffusion?

Resetand-Roll adapts rolling diffusion inference to the simulation interface by preventing future-conditioned latent states from being carried across simulation steps. This helps align generation with step-wise condition updates during closed-loop execution.

Jul 16, 2026

Upload by River Wang