ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning

Jul 16, 2026

Upload by Hazel

Open-ended tabletop manipulation requires agents to understand natural language while adapting to dynamic environments and execution failures. ACE (Agentic Control for Embodied manipulation) introduces a zero-shot workflow reasoning framework for tabletop pick-and-place driven by natural language. Instead of direct low-level action mapping, ACE combines agentic workflow decomposition with reusable robot-executable skills: a mask-mediated visual grounding interface and a pick-and-place primitive. Active sub-goals are grounded, tracked, verified, and passed to a task-agnostic policy. Closed-loop post-execution verification enables advance, retry, repair, or replan based on outcomes. Evaluation on complex long-horizon tasks shows task-level zero-shot generalization without task-specific retraining, including 50% success for equation formation and 70% for constraint retrieval.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Figure 1: Comparison of robotic manipulation paradigms

ACE framework overview

arXiv :2607 .04 162v 1 [ cs .RO] 5 Jul 2026 ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning Iok Tong Lei 1,†, QianZhi Li2,†, Ying Jie Yap 1 , Yujie Zhang4 , Rui Zhong2 , Haichao Gui2,* , Xiaolong Liu3,* , Zhidong Deng 1,* 1Department of Computer Science, Tsinghua University, China 2National College for Excellent Engineers, Beihang University, China 3Wuxi Dexteroushands Robotic Technology Co. 4Independent Researcher †These authors contributed equally to this work. * Corresponding authors. Abstract: Open-ended tabletop manipulation requires agents to not only understand natural language but also adapt to dynamic environments and execution failures. We present ACE (Agentic Control for Embodied manipulation), a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language. Rather than relying on direct low-level action mapping, ACE combines agentic workflow reasoning with two robot-facing executable skills: a visual grounding interface and a reusable pick-and-place primitive. To bridge semantic reasoning and physical control, the active sub-goal is grounded into a mask-mediated vision-action interface. This unified mask specifies the target object and destination, is tracked over time, exposed for human verification, and ultimately passed to a task-agnostic downstream policy for execution. Crucially, ACE operates ina closed loop supported by a multi-timescale memory. After an action is executed, the system automatically verifies whether the intended sub-goal succeeded, using the outcome to advance, retry, repair, or replan. This enables online adaptation to user corrections, scene changes, and physical failures. We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval. ACE demonstrates task-level zero-shot generalization on novel semantic constraints and randomized tabletop scenes without task-specific retraining. Specifically, while standard end-to-end baselines struggle to complete these logically demanding tasks, ACE achieves a 50% success rate in equation formation and a 70% success rate in constraint retrieval. This contrast demonstrates that explicit workflow reasoning and mask-mediated control offer a robust, practical route toward adaptable robotic manipulation. Keywords: Embodied reasoning, Robot manipulation, Zero-shot planning 1 Introduction Natural language provides an intuitive interface for open-ended robotic control, allowing users to specify high-level goals and refine intent dynamically. However, this flexibility makes execution inherently difficult. Instructions are often underspecified, and task constraints may emerge only after execution begins [1, 2, 3] . This challenge is particularly visible in tabletop pick-and-place. While a basic robotic primitive, executing open-ended tasks—such as forming equations from number cubes based on abstract constraints—requires more than recognizing objects and executing grasps. Traditional manipulation systems often rely on task-specific policies or libraries of fixed rules [4, 5], which struggle to generalize when goals vary significantly across episodes. To succeed, a system (A) Vision-Action Model (VA) VA Model Low Level Actions Lacks Instruction Following Capability (B) Vision-Language-Action Model (VLA) Lacks Robust Zero-Shot Reasoning for Complex, Long-Horizon Task (C) Proposed ACE Method Enables Logically Complex, Long-Horizon Reasoning and Planning Figure 1: Comparison of robotic manipulation paradigms. (A) Vision-Action (VA) Models execute low-level actions based on visual observations. (B) Vision-Language-Action (VLA) Models can condition actions on language, but direct language-to-action mapping may struggle with strict compositional reasoning, long-horizon sequencing, and recovery under limited task data. (C) Our Proposed ACE Framework decouples reasoning

FAQs

What problem does ACE address in tabletop pick-and-place?

ACE targets open-ended manipulation where natural-language instructions are under-specified and where execution may fail due to semantic or physical issues. It aims to maintain reliable performance without task-specific retraining.

How does ACE connect workflow reasoning to robot control?

ACE decouples high-level reasoning from low-level physical execution using a mask-mediated vision-action interface. The agent produces semantic pick-and-place masks via grounding skills, which condition a reusable vision-action policy for execution.

How does ACE handle failures or user corrections during execution?

After each action, ACE verifies whether the intended sub-goal succeeded. The verification outcome drives online adaptation, including advancing, retrying, repairing, or replanning based on changes in the scene, user feedback, or physical failures.

Jul 16, 2026

Upload by Hazel