SyncSpace: Layout-Conditioned 3D Gaussian Splatting for Space Reskinning in Mixed Reality

Jul 16, 2026

Upload by Noah

SyncSpace is a system that delivers spatial alignment and visual consistency between a generated 3D Gaussian Splatting (3DGS) world and a real physical room. Depth sensing extracts 3D bounding boxes, which are rendered into a layout-only panorama as a geometric prior for a generative world model. The method produces a Gaussian-splat scene where objects are re-semantized to a target style without per-object control. A coarse-to-fine registration aligns the result to the scan, with manual pinch refinement when needed, enabling hand-tracked engulfment interaction and multi-style reskinning while preserving room layout.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages3

Reading Time8 min

Introduction

SyncSpace

World Generation

Spatial Alignment

SyncSpace: Layout-Conditioned 3D Gaussian Splatting for Space Reskinning in Mixed Reality Qinchuan Zhang [qinchuan@usc.edu](qinchuan@usc.edu)[ ](qinchuan@usc.edu)University of Southern California Los Angeles, California, USA Weibo Xu [weiboxu@usc.edu](weiboxu@usc.edu)[ ](weiboxu@usc.edu)University of Southern California Los Angeles, California, USA Yunge Wen [yungew@mit.edu](yungew@mit.edu) Massachusetts Institute of Technology MIT Media Lab Cambridge, Massachusetts, USA arXiv :2607 . 10050v1 [ cs .HC] 11 Jul 2026 Figure 1: SyncSpace reskins a real room into a stylistically unified, spatially congruent MR world in minutes. Top: the generation pipeline—scan the room, abstract it to a layout-only white panorama, and generate a layout-conditioned Gaussian-splat world. Bottom: the in-VR experience—touching a virtual orb makes the virtual world rise and engulf the real room until it fully replaces it, preserving layout while re-imagining appearance. Abstract We present SyncSpace, a system that achieves both spatial alignment and visual consistency between a generated 3DGS world and physical space. We first scan the space via depth sensing to extract 3D bounding boxes, which we render into a layout-only panorama and feed as a geometric prior to a generative world model, producing a Gaussian splat scene in which objects are re-semantized to fit a target style without per-object control. We then align the generated scene to physical space with a coarse-to-fine registration algorithm, refined manually via pinch gestures when automatic registration does not converge. We demonstrate a hand-tracked engulfment interaction in which the virtual world rises to replace the physical space, and show a single space reskinned into multiple stylistically distinct worlds with its layout preserved. 1 Introduction Mixed reality headsets can now render immersive, room-scale virtual worlds in real time, enabling gaming, creative exploration, and themed entertainment at home. Depth-sensor-based 3D scanning can reliably extract a space’s walkable range and prevent collisions via geometric shape matching, but generating a virtual world that is both stylistically unified and registered to this scan remains an open problem. Substitutional-reality and proxy-based systems [4, 7] match individual real objects to shape-similar virtual props, while layoutdriven approaches [1, 8, 12] instead retrieve entire pre-authored room templates to fit real-world geometry, constructing walkable VR at the scale of the whole space. Both are constrained to whatever their asset library contains, producing stylistically inconsistent results. The emergence of 3D generative models has enabled recent style-transformation systems [9, 11] to replace retrieval with generation, such as diffusion-based texture propagation onto scanned meshes or per-object 2D-to-3D conversion, yet global style coherence is still imposed post hoc rather than emerging intrinsically from the generative process. 3D Gaussian Splatting [6] has further enabled real-time, photorealistic rendering, allowing generative world models [10] to synthesize an entire room in a single generative pass, bypassing per-object assembly altogether; however, the resulting content is unregistered to the user’s physical space, precluding safe walking. No existing approach simultaneously achieves Zhang et al. whole-room stylistic coherence, spatial registration to the physical environment, and safe embodied navigation. We present SyncSpace, a system that achieves both spatial alignment and visual consistency between a generated 3DGS world and physical space. We first scan the space via depth sensing to extract 3D bounding boxes, which we render into a layout-only panorama and feed as a geometric prior to a generative world model, producing a Gaussian splat scene in which objects are re-semantized to fit a target style without per-object control. We then align the generated scene to physical spac

FAQs

What problem does SyncSpace address in mixed reality reskinning?

SyncSpace targets the challenge of generating a stylized 3DGS world that is both visually coherent and spatially registered to a scanned physical room, enabling safe navigation rather than unregistered content.

How does SyncSpace convert a real room into a layout-only representation for generation?

It scans the room using depth sensing to obtain axis-aligned 3D bounding boxes, renders them with a uniform white material, and encodes the resulting layout as a white equirectangular panorama used as a geometric prior.

How does SyncSpace align the generated 3DGS scene with the physical space?

It performs scale alignment by matching bounding boxes, then uses a coarse-to-fine registration algorithm for point-cloud alignment. When automatic registration does not converge, it supports manual refinement via pinch gestures.

Jul 16, 2026

Upload by Noah