Manual2Skill - Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models

Oct 26, 2026

Humans can interpret abstract instruction manuals and transform them into executable manipulation, while robots struggle to decode such instructions into actionable steps. Manual2Skill presents a robot learning framework that uses a vision-language model to extract structured information from manual images, build hierarchical assembly graphs, estimate relative 6D poses per step, and generate motion plans for autonomous execution. Experiments assemble real IKEA furniture with efficiency and precision, enabling long-horizon tasks from manuals.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

AccessFree to download

Introduction

Key idea: learning from visual manuals

Proposed pipeline: VLM extraction, pose estimation, motion planning

Contributions

arXiv :2502 . 10090v3 [ cs .RO] 18 Oct 2025 Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models Chenrui Tie∗1 Shengxiang Sun∗2 Jinxuan Zhu 1 Yiwei Liu4 Jingxiang Guo 1 Yue Hu5 Haonan Chen 1 Junting Chen 1 Ruihai Wu3 Lin Shao 1 1National University of Singapore 2University of Toronto 3Peking University 4 Sichuan University 5Zhejiang University Manual2Skill Fig. 1: Overview of Manual2Skill Framework. We propose Manual2Skill, which learns manipulation skills from manuals, enabling robots to understand and execute complex manipulation tasks in a manner akin to humans. The green region showcases the input of our pipeline: the pictures of the assembly manual and real parts. The blue region depicts our pipeline: 1) a VisionLanguage Model (VLM) generates a Hierarchical Assembly Graph, 2) a per-step pose estimation module predicts the 6D-poses of components, and 3) a motion planning and execution module controls the robot arms to assemble the furniture autonomously. Abstract—Humans possess an extraordinary ability to understand and execute complex manipulation tasks by interpreting abstract instruction manuals. For robots, however, this capability remains a substantial challenge, as they cannot interpret abstract instructions and translate them into executable actions. In this paper, we present Manual2Skill, a novel framework that enables robots to perform complex assembly tasks guided by highlevel manual instructions. Our approach leverages a VisionLanguage Model (VLM) to extract structured information from instructional images and then uses this information to construct hierarchical assembly graphs. These graphs represent parts, subassemblies, and the relationships between them. To facilitate task execution, a pose estimation model predicts the relative 6D poses of components at each assembly step. At the same time, a motion planning module generates actionable sequences for realworld robotic implementation. We demonstrate the effectiveness of Manual2Skill by successfully assembling several real-world IKEA furniture items. This application highlights its ability to manage long-horizon manipulation tasks with both efficiency and precision, significantly enhancing the practicality of robot learning from instruction manuals. This work marks a step forward in advancing robotic systems capable of understanding and executing complex manipulation tasks in a manner akin to human capabilities. Project Page: [https://owensun2004.github.io/Furniture](https://owensun2004.github.io/Furniture)Assembly-Web/ *Equal contribution. I. INTRODUCTION Humans can learn manipulation skills from instructions in images or texts; for example, people can assemble IKEA furniture or LEGO models by following a manual’s instructions. This ability enables humans to efficiently acquire long-horizon manipulation skills from sketched instructions. In contrast, robots typically learn such skills through imitation learning [59] or reinforcement learning [43], both of which require significantly more data and computation. Replicating the human ability to transfer abstract manuals to real-world actions remains a significant challenge for robots. Manuals are typically designed for human understanding, using simple schematic diagrams and symbols to convey manipulation processes. This abstraction makes it difficult for robots to comprehend such instructions and derive actionable manipulation strategies [32, 49, 48] . Developing a method for robots to effectively utilize human-designed manuals would greatly expand their capacity to tackle complex, long-horizon tasks while reducing the demand of collecting extensive demonstration data. Manuals inherently encode the structural information of complex tasks. They decompose high-level goals into midlevel subgoals and capture task flow and dependencies, such as sequential steps or parallelizable subtasks. For example, furniture assembly manuals

FAQs

What problem does Manual2Skill address for robots?

Robots cannot readily interpret abstract instruction manuals and convert them into executable manipulation actions. Manual2Skill targets this gap for complex, long-horizon assembly tasks.

How does Manual2Skill use manuals to generate robot actions?

A vision-language model extracts structured assembly information from manual images to form a hierarchical assembly graph, then a pose estimation model predicts relative 6D poses at each step, and a motion planner generates actionable sequences for robotic execution.

What evidence is provided to validate the framework?

The framework is demonstrated by assembling several real-world IKEA furniture items, showing efficient and precise handling of long-horizon manipulation tasks guided by manual instructions.

Oct 26, 2026

Upload by Danger Angel