Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

Jul 16, 2026

Upload by Eden

Compositionality enables scalable, data-efficient robot learning by combining simpler elements into complex behaviors. The framework presents generative learning from demonstration that links perception and motion via shared object-level representations. Scenes are rendered using object-centric neural fields combining canonical neural fields with latent-conditioned deformations, yielding smooth, consistent, interpretable spatial and geometric variations. Motion generation uses a temporal mixture-of-experts gating mechanism to combine object-conditioned movement primitives into full trajectories. Simulation completes long-horizon manipulation with less training data, while real-world tests show robustness to noise, category-level generalization with language segmentation, and operation directly on 3D scene representations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Learning from Demonstration (LfD)

Movement Primitives and Limitations

Proposed Generative Object-Centric MoE Framework

Compositional Motion Generation from Demonstration with Object-Centric Neural Fields Ahmet Tekden 1 Yasemin Bekiroglu 1 ,2 arXiv :2607 .07 129v 1 [ cs .RO] 8 Jul 2026 Abstract—Compositionality, by organizing complex behavior as combinations of simpler elements, enables robot learning that is scalable and data efficient. Leveraging this principle, we propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations. We render scenes from object-centric neural representations that integrate canonical neural fields with latentconditioned deformations, capturing positional and geometric variations in a smooth, consistent, and interpretable way. For motion generation, a temporal mixture-of-experts (MoE) employs a gating mechanism to combine object-conditioned movement primitives over time, producing complete trajectories. This spatial–temporal compositionality maintains the data efficiency of movement primitives while grounding motion in visual structure, enabling systematic generalization across diverse scene configurations. In simulation, long-horizon manipulation tasks are successfully completed using the proposed model, which requires significantly less training data than other image-based baselines. Real-world experiments further demonstrate the method’s robustness to noise, its ability to generalize at the category level through language-based segmentation models, and its capacity to operate directly on 3D scene representations. Index Terms—Learning from Demonstration, Deep Learning in Grasping and Manipulation I. INTRODUCTION Learning from Demonstration (LfD) [1] is a widely used approach for teaching robots task-specific skills from human demonstrations. A central challenge is representing diverse, long-horizon behaviors from a limited number of demonstrations. Treating motion as a single, monolithic sequence overlooks its compositional structure, which complicates the learning process. Everyday tasks unfold through sequential, object-level interactions—reaching, grasping, moving, and placing—that together accomplish the overall goal. This motion-level compositionality [2] can be complemented by scene-level compositionality, grounding motion in objectcentric scene representations and providing a natural foundation for compositional motion modeling. Figure 1 illustratesan example of such object-centric structure, showing how the scene is decomposed into object-specific representationsand how those representations relate to different phases of a Manuscript received: December 18, 2025; Revised May 25, 2026; Accepted June 24, 2026 . This paper was recommended for publication by Editor Wei Pan upon evaluation of the Associate Editor and Reviewers’ comments. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, the Chalmers AI Research Center (CHAIR), and the Chalmers Gender Initiative for Excellence (Genie) . 1Department of Electrical Engineering, Chalmers University of Technology, SE-412 96 Gothenburg, Sweden. 2Department of Computer Science, University College London, WC1E 6BT London, U.K. Email: [tekden@chalmers.se](tekden@chalmers.se)[ ](tekden@chalmers.se)Digital Object Identifier (DOI): see top of this page. Fig. 1: System overview. Top: Task sequence with object interactions, followed by the ground-truth and reconstructed images. Middle: Object-specific masks inferred from latent codes, illustrating object-centric scene decomposition. Bottom: Temporal gating weights showing each object’s influence over the trajectory; snapshots above the plot correspond to key events (T0–T3), marked by vertical dotted lines. manipulation task. By supporting data efficiency, modularity, and generalization, compositional formulations present a compelling alternative to monolithic approache

FAQs

What problem does the paper address in learning from demonstrations?

It tackles the challenge of representing diverse, long-horizon robotic behaviors from limited demonstrations, where treating motion as a single monolithic sequence makes learning difficult.

How does the proposed method connect perception with motion?

It uses object-centric shared representations: scenes are rendered from object-centric neural fields, and motion generation is grounded by object-conditioned primitives combined over time.

How does the temporal mixture-of-experts model produce full motion trajectories?

A gating mechanism selects and blends multiple object-conditioned movement experts over time, composing primitives into complete trajectories while maintaining data efficiency.

Jul 16, 2026

Upload by Eden