SplatCtrl: Perception-Action Coupling via Gaussian Scene Representations and Reactive Robot Control

Jul 16, 2026

Upload by Rowan

Robotic manipulators excel in structured environments but struggle in unstructured, dynamic settings. This paper introduces SplatCtrl, a real-time framework that unifies scene reconstruction and reactive motion generation for collision-free control in unseen and continuously changing environments. Using 3D Gaussian Splatting, it adds voxel-based filtering and dynamic Gaussian relocation for efficient RGB-D updates, then derives continuous signed distance functions from isotropic Gaussians. The resulting stable collision-probability metrics are integrated into control barrier functions, enabling smooth, reliable motion adaptation. Experiments validate performance in simulation, on a physical robot, and in shared human-robot workspace.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Motivation: structured vs. unstructured environments

Scene representation requirements

Limitations of classical and implicit representations

Perception–action coupling for reactive control

Paper contributions

SplatCtrl: Perception-Action Coupling via Gaussian Scene Representations and Reactive Robot Control Siddarth Jain 1∗ and Ho Jin Choi 1,2∗ arXiv :2607 .08948v 1 [ cs .RO] 9 Jul 2026 Abstract—Robotic manipulators excel in structured environments but face substantial challenges in unstructured and dynamic settings. This paper presents SplatCtrl, a unified framework for real-time scene reconstruction and reactive robot motion generation to enable collision-free robotic arm control in previously unseen and continuously changing environments. Building on 3D Gaussian Splatting (3D-GS), we introduce a hybrid voxel-based filtering and dynamic Gaussian relocation strategy that supports efficient scene reconstruction from RGB-D streams while accommodating environmental changes. For safe and reactive control, we further propose a method for deriving continuous signed distance functions from isotropic Gaussians, providing stable and differentiable collision probability estimates that bridge classical distance fields with the modern implicit representation. These continuous distance metrics are incorporated into control barrier functions, resulting in a unified perception-action coupling framework that supports smooth and reliable real-time motion generation in response to scene changes. Experimental validation in simulation, on physical robot, and within shared humanrobot workspace demonstrates the framework’s effectiveness, achieving integrated scene reconstruction and reactive control in uncertain, and dynamic environments. I. INTRODUCTION Robotic manipulators demonstrate high levels of performance in structured environments, such as factory floors, where tasks are repetitive and object locations are predetermined. These environments are engineered, often at substantial cost, to enhance repeatability, ensure operational safety, and minimize uncertainty. In contrast, unstructured real-world settings present significantly greater challenges: they are dynamic and unpredictable, with objects and obstacles that may shift or emerge unexpectedly. Under such conditions, robots must rely on sensory inputs to perceive their surroundings, and adapt their motions in real time. A key requirement for robotic manipulation in unstructured settings is the ability to construct reliable representations of the environment. Such representations serve as the foundation for perception, planning, and control, yet their design is constrained by the competing demands of fidelity, efficiency, and adaptability. Classical representations [1], [2], such as point clouds have been widely adopted due to their simplicity and compatibility. However, they are inherently sparse and often face challenges with occlusions and maintaining temporal consistency. Voxel grids provide a structured 1Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, [USA.](USA. sjain@merl.com)[ sjain@merl.com](USA. sjain@merl.com) 2University of Pennsylvania, Philadelphia, Pennsylvania, USA. This research was completed during H. Choi’s internship at MERL. [cr139139@seas.upenn.com](cr139139@seas.upenn.com) ∗ indicates equal contribution. Fig. 1: SplatCtrl leverages real-time RGB-D data to reconstruct a scene representation and continuously updates robot’s motion accordingly. Left: With moderate obstacle height, the robot traverses over the obstacles. Right: When obstacle height increased, the updated scene reconstruction triggers an alternative robot motion (shown in green) . representation that facilitates occupancy reasoning, but they face challenges in balancing resolution with computational efficiency. More recent efforts have explored implicit neural representations, such as Neural Radiance Fields (NeRFs) [3], which model geometry and appearance as continuous functions. While these approaches achieve impressive scene representation fidelity, their practical deployment is constrained by substantial computational demands and limited adaptability to dynamic enviro

FAQs

What problem does SplatCtrl address for robotic manipulation?

SplatCtrl targets collision-free robotic arm control in unstructured and dynamic environments where objects and obstacles change unexpectedly. It combines online scene reconstruction with reactive motion generation so the robot can adapt in real time.

How does SplatCtrl reconstruct scenes from RGB-D streams?

It extends 3D Gaussian Splatting by using voxel-based filtering together with a dynamic Gaussian relocation strategy. This supports efficient reconstruction and continuous updates as the environment changes.

How does SplatCtrl estimate collision risk for safe control?

SplatCtrl derives continuous signed distance functions from isotropic Gaussians to obtain stable, differentiable collision probability estimates. These metrics are incorporated into control barrier functions to enable smooth and reliable reactive motion.

Jul 16, 2026

Upload by Rowan