RELATE - Physically Plau sible Multi-Object Scene Synthesis Using Structured Latent Spaces - NeurIPS 2020 Paper

Aug 20, 2026

Upload by Aditya

RELATE presents an end-to-end model for generating physically plausible scenes and videos containing multiple interacting objects. Trained on raw, unlabeled data, it combines an object-centric GAN with an explicit mechanism to capture correlations between individual objects, yielding a physically interpretable parameterization. The correlation modeling is shown to be necessary for learning disentangled object positions and identity, enabling realistic scene editing. RELATE further extends naturally to dynamic settings, producing high-fidelity videos and outperforming prior object-centric generation on both synthetic and real-world datasets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Introduction

Problem setting and motivation

Background: structured latent spaces and BlockGAN

Key limitation: object independence assumption

RELATE approach and contributions

Modeling object correlations in latent space

Benefits: disentanglement and physical plausibility

Applications and evaluations

RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces Sébastien Ehrhardt 1 􀀃 Oliver Groth1 􀀃 Áron Monszpart2,3 Martin Engelcke1 Ingmar Posner1 Niloy J. Mitra2,4 Andrea Vedaldi1 1Department of Engineering Science, University of Oxford 2Department of Computer Science, University College London 3Niantic, 4 Adobe Research {hyenal,[ogroth}@robots.ox.ac.uk](ogroth}@robots.ox.ac.uk) Abstract We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained end-to-end on raw, unlabeled data. RELATE combines an object-centric GAN formulation with a model that explicitly accounts for correlations between individual objects. This allows the model to generate realistic scenes and videos from a physically-interpretable parameterization. Furthermore, we show that modeling the object correlation is necessary to learn to disentangle object positions and identity. We ﬁnd that RELATE is also amenable to physically realistic scene editing and that it signiﬁcantly outperforms prior art in object-centric scene generation in both synthetic (CLEVR, ShapeStacks) and real-world data (cars) . In addition, in contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual ﬁdelity. Source code, datasets and more results are available at [http://geometry.cs.ucl.ac.uk/projects/2020/relate/](http://geometry.cs.ucl.ac.uk/projects/2020/relate/) . 1 Introduction We consider the problem of learning to generate plausible images of scenes starting from parameters that are physically interpretable. Furthermore, we wish to learn such a capability from raw images alone, without any manual or external supervision. Image generation is often approached via Generative Adversarial Networks (GAN) [10] . These models learn to map noise vectors, used as a source of randomness, to image samples. While the resulting images are realistic, the random vectors that parameterize them are not interpretable. To address this issue, authors have recently proposed to structure the latent space of deep generative models, giving it a partial physical interpretability [28, 29, 36] . For example, HoloGAN [28] samples volumes and cameras to generate 2D images of 3D objects, and BlockGAN [29] creates scenes by composing multiple objects. The resulting GANshave been shown to learn concepts such as viewpoint and object disentangling from raw images. BlockGAN is of particular interest because, via its relatively strong architectural biases, it provides interpretable parameters for the scene, incorporating concepts such as position and orientation. However, BlockGAN comes with a signiﬁcant limitation in that it assumes that objects are mutually independent. This approximation is acceptable only when objects interact weakly, but it is badly violated for medium to densely packed scenes, or for scenes such as stacking wooden blocks or cars following a path, where the (object) correlation is strong. 􀀃 indicates equal contribution 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. Figure 1: Image generation using RELATE. Individual scene components, such as background and foreground objects are represented by appearance z0 and pairs of appearance and pose vectors (zi ; 􀀒i); i 2 f1; : : : ; Kg, respectively. The key spatial relationship module 􀀀 adjusts the initial independent pose samples ^􀀒i to be physically plausible (e.g., non-intersecting) to produce 􀀒i. The structured scene tensor W is ﬁnally transformed by the the generator network G to produce an image ^I. RELATE is trained end-to-end in a GAN setup (D denotes the discriminator) on real unlabelled images. Recent work in object-centric generative modeling has attempted to speciﬁcally address this by capturing correlations in latent s

FAQs

What does RELATE generate and how is it trained?

RELATE generates physically plausible scenes and videos with multiple interacting objects. It is trained end-to-end in a GAN setup using raw, unlabeled images without manual supervision.

Why is object correlation modeling important in RELATE?

The paper shows that explicitly modeling correlations is necessary to learn disentangled object positions and identity. Without this component, results may look realistic but fail to align parameters with physically plausible object associations.

How does RELATE handle dynamic scenes and what evidence is reported?

RELATE extends naturally from static object-centric generation to dynamic settings. The paper reports generation of high-visual-fidelity videos and demonstrates that it outperforms prior work on object-centric scene generation benchmarks.

Aug 20, 2026

Upload by Aditya