CARLA-GS Decoupling Representation Reasoning and Physics Simulation for Autonomous Driving Corner-Case Synthesis

Jul 16, 2026

Safety evaluation for autonomous driving depends on rare, safety-critical interactions, driving the need for simulators that can deliberately synthesize realistic corner cases with photorealistic sensing. Corner-case generation spans multiple sources: visual representation, semantic scene reasoning, and vehicle trajectory generation plus control. Prior methods isolate components or struggle with spatiotemporal consistency and physical realism. CARLA-GS unifies them by reconstructing an editable Gaussian scene from real data, using a multiagent LLM for risk reasoning and intent-level waypoints, and delegating low-level motion feasibility to CARLA with PID control, then reprojecting simulated states for ego-centric rendering. Experiments on Waymo show controllable synthesis and physically feasible, video-consistent results.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Safety-critical rare-event simulation and corner cases

Scene reconstruction: NeRF, diffusion world models, and 3D/4D Gaussian Splatting

Limits of vanilla 3DGS and the need for semantic reasoning and physical execution

CARLA-GS modular decoupling framework overview

CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesis Kaicong Huang 1 , Meng Ma 1 , Ruimin Ke 1∗ arXiv :2607 .0760 1v 1 [ cs .RO] 8 Jul 2026 Abstract—Safety evaluation for autonomous driving is dominated by rare, safety-critical interactions, motivating simulators that can deliberately synthesize corner cases with photorealistic observations. Corner-case generation is inherently a multisource problem spanning visual representation, scene reasoning, and vehicle trajectory generation and control. Prior knowledgeand model-based approaches typically focus on scene or trajectory components in isolation, while diffusion-based methods attempt end-to-end generation but still struggle to ensurespatiotemporal consistency and physical realism. To unify these aspects within a single framework, we propose CARLA-GS, a modular corner-case synthesis pipeline that decouples visual representation, semantic reasoning, and physics-based execution while maintaining tight cross-module coupling. Starting from real driving data, we reconstruct an editable gaussian scene with additional geometry-consistent constraints. A multiagent LLM then performs scene-level reasoning to identify risky interactions and generate intent-level waypoint trajectories, while the low-level motion control is delegated to CARLA, where a PID controller ensures kinematic and dynamic feasibility. The simulated vehicle states are finally re-projected into the gaussian scene for ego-centric rendering. This design enables high-level semantic reasoning, low-level physically executable motion, and photorealistic corner-case generation within a unified pipeline. Experiments on the Waymo Open Datasetshow, both quantitatively and qualitatively, that our framework enables controllable corner-case generation and produces photorealistic, spatiotemporally consistent videos aligned with semantic intent and physically feasible motion. I. INTRODUCTION Autonomous driving (AD) operates in open-world traffic where safety evaluation is inherently a rare-event problem, known as the Curse of Rarity [1] . This motivates simulation pipelines that intentionally expose corner cases. Meanwhile, camera-centric AD systems require simulators to provide not only trajectory rollouts but also photorealistic observations for reliable closed-loop evaluation [2] . This requirement has renewed interest in scalable drivingscene reconstruction for safety-oriented simulation. NeRFbased methods achieve high-fidelity view synthesis but remain expensive for large-scale optimization and rendering [3], while diffusion-based world models provide strong generative diversity yet still struggle with physical plausibility and multi-view consistency under real-time constraints [4], [5] . In contrast, 3D/4D Gaussian Splatting (GS) [6], [7] enables real-time differentiable rendering with explicit scene primitives, making it attractive for urban driving simulation [8], [9], [10] . However, vanilla 3DGS is 1 Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute, 110 Eighth Street, Troy, NY USA 12180 . ∗ Corresponding author. Email: [ker@rpi.edu](ker@rpi.edu) designed for photometric reconstruction, lacking instancelevel control over traffic participants and remaining sensitive to reconstruction artifacts that may affect occlusion or collision outcomes [11], [12] . Moreover, corner-case generation is not purely a visual problem. It requires semantic reasoning over agent intent, interaction context, and future motion evolution. Prior work synthesizes corner cases through data-driven, knowledgebased, or learning-based pipelines [13], [14], [15] . However, these approaches often lack semantic scene understanding and high-level decision reasoning for identifying plausible risky interactions, limiting their flexibility in generating targeted behaviors. Inspired by [16], we adopt LLM-based reasoning as a semantic planning modu

FAQs

What problem does CARLA-GS address in autonomous driving safety evaluation?

It targets safety evaluation dominated by rare, safety-critical interactions by enabling deliberate synthesis of photorealistic corner cases for closed-loop testing.

How does CARLA-GS decouple representation, reasoning, and physics in its pipeline?

It reconstructs an editable Gaussian scene for visual representation, uses a multiagent LLM for semantic reasoning and intent-level waypoint trajectories, and delegates low-level feasible control to CARLA with PID ensuring kinematic and dynamic feasibility.

How does CARLA-GS improve physical realism and spatiotemporal consistency compared with prior approaches?

It avoids purely end-to-end diffusion generation by separating semantic intent generation from physics-consistent execution, then reprojects simulated vehicle states into the Gaussian scene for ego-centric photorealistic rendering.

Jul 16, 2026

Upload by Evangeline