Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning

Jul 16, 2026

Upload by Aurora

The document presents a model for robotic manipulation planning that abstracts continuous, high-dimensional sensorimotor interactions into discrete object and action symbols. It jointly discovers high-level manipulation primitives and object categories via a binary bottleneck, trained with random interactions to predict multi-modal outcomes such as object motion, contact, and force feedback. Using the learned binary representations, a discrete planner exploits intermediate effect-trajectory steps for partial action execution and precise low-level control, and enables few-shot generalization on novel objects based on behavior rather than visual similarity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Problem of long-horizon sensorimotor reasoning

Limitations of existing object/skill abstractions

Motivation for multimodal interaction and effect-based learning

Temporal structure and planning precision

Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning Burcu Kilic 1 , Berke Kartal 1 , Fatih Dogangun 1 , Erhan Oztop2,3 , Emre Ugur 1 arXiv :2607 .00031v1 [ cs .RO] 22 Jun 2026 Abstract—To perform complex manipulation planning, autonomous robots are required to abstract continuous, highdimensional sensorimotor interactions into discrete object and action representations. Earlier work either categorized objects based on visual appearances, which fails to distinguish objects that appear similar but behave differently, or based on effects under interaction, but was limited to predefined actions. To address these limitations, we propose a model that jointly discovers highlevel manipulation primitives and object categories through a binary bottleneck layer, trained to predict multi-modal outcomes, including object motion, contact, and force feedback, from random interaction data. Building on these discovered binary representations, we leverage a discrete planning method that uses intermediate steps in the predicted effect trajectory to enable partial action executions for precise low-level control. Additionally, we evaluate our framework’s generalization capabilities on novel objects by assigning object categories through comparing a small number of interaction effects with the predicted effects of learned object symbols, enabling few-shot generalization based on behavior rather than visual similarity. We conduct experiments on tabletop repositioning and stacking tasks, and confirm that our effect-driven planning approach outperforms both a state-of-theart method and a visual-based alternative in planning precision across seen and novel objects. Index Terms—Robot learning, Representation learning, Autonomous mental development, Few shot learning I. INTRODUCTION Autonomous robot development aims to build intelligent agents capable of learning through own experience, reasoning about their surroundings, and adapting to novel scenarios [1] . However, a fundamental challenge is that reasoning over highdimensional continuous sensorimotor spaces becomes unmanageable in long-horizon manipulation tasks. Operating in such complex sensorimotor spaces requires a layer of abstraction on both percepts and actions [2], [3] . For example, an infant that stacks multiple blocks on top of each other does not plan low-level joint movements directly from environmental pixels; rather, it thinks in terms of high-level intentions like picking a block and placing it on top of another one. Prior studies either manually constructed such abstractions [4] or used reconstruction-based clustering [5], ignoring the role of interaction. However, Sun [6] argued that the concepts of objects and skills emerge directly from the agent’s interaction with the world and are linked to their goals and needs. Following this principle, an autonomous robot must discover its own representations by continuously exploring and manipulating objects, while learning to make sense of This work was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) ARDEB 1001 Program (124E227) and by the INVERSE Project under Grant 101136067, funded by the European Union. 1Bogazici University, 2 Ozyegin University, 3 Osaka University. the physical dynamics. For this physical exploration to be effective, the robot’s interactions cannot be limited to purely visual observations. Multi-modal feedback such as force and contact can convey complementary information to visual feedback about action outcomes, an observation also supported by developmental studies showing that young infants learn action properties through both visual and tactual interactions with objects [7] . Motivated by this, our approach allows a robot to autonomously categorize objects and learn highlevel manipulation skills directly from multi-modal physical interactions. Through continuous physical interactions, children learn to categ

FAQs

What core challenge does the work address in robotic manipulation planning?

It addresses how to make high-dimensional, continuous sensorimotor reasoning tractable for long-horizon manipulation by abstracting interactions into discrete object and action representations.

How does the proposed model learn object categories and manipulation primitives?

It uses a joint discovery model with a binary bottleneck layer trained to predict multi-modal interaction outcomes (e.g., motion, contact, and force feedback) from random interaction data.

How does the approach enable planning and generalization on novel objects?

The discrete planning method leverages intermediate steps in the predicted effect trajectory to execute partial actions with precise low-level control, and it assigns object categories by matching a few observed interaction effects to predicted effects for few-shot generalization based on behavior.

Jul 16, 2026

Upload by Aurora