Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction

Jul 16, 2026

Upload by Finn

Human–robot collaborative assembly demands that robots interpret ambiguous human corrective instructions and turn them into physically executable motions. Vision–language models (VLMs) offer semantic understanding but can choose logically inconsistent targets and misjudge whether actions will succeed. This work presents a replanning framework that converts human instructions into action-target candidates, then applies an Internal Correction Model for pre-execution logical verification and an External Correction Model for post-execution visual verification. Integrated VLM reasoning with 6-DoF grasp generation and collision-free planning, achieving 66.7% real-world success for object fixation and strong tool-selection accuracy.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Introduction

Industry 5.0 and HRC Motivation

Core Challenge: Interpreting Ambiguous Human Intent

Limitations of Direct VLM Application

Proposed Dual-Correction Replanning Framework

arXiv :2602 . 14551v2 [ cs .RO] 13 Jul 2026 FULL PAPER Replanning Human–Robot Collaborative Tasks with Vision–Language Models via Semantic and Physical Dual–Correction Taichi Katoa , Takuya Kiyokawab* , Namiko Saitob,c , and Kensuke Haradab,da Department of Systems Science, School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan; b Department of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan; c Microsoft Research Asia, Shinagawa, Tokyo, Japan; d Industrial Cyber-physical Systems Research Center, The National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo, Japan. ARTICLE HISTORY Compiled July 14, 2026 ABSTRACT Human–robot collaborative assembly requires robots to interpret ambiguous cor rective instructions while producing physically executable motions. Vision–language models (VLMs) provide semantic reasoning but may select logically inconsistent targets or misjudge execution outcomes. We propose a replanning framework that maps human instructions to Action Target candidates, including grasp poses and tool selections, and combines an Internal Correction Model for pre-execution logical verification with an External Correction Model for post-execution visual verification. The framework integrates VLM reasoning with 6-DoF grasp generation and collision free trajectory planning. Simulation ablations show configuration-dependent effects: internal correction improves candidate validity, whereas external correction enables recovery for a low-latency VLM but can reduce success when visual verification pro duces false negatives. Experiments with an upper-body humanoid robot achieved 66.7% success in real-world object fixation, 100% in initial tool selection, and 75.0% in corrective tool selection. These results demonstrate interactive replanning across spatial and semantic collaborative tasks while identifying visual-state verification as a key limitation. KEYWORDS Replanning, Vision–Language Models, Human–Robot Collaboration 1. Introduction While recent advances in manufacturing industry have advanced rapidly, associated social challenges such as workforce shortages and the demand for flexible production systems have been increasing. To address these issues, a new concept known as Industry 5.0 [1] has been proposed. Unlike conventional manufacturing paradigms that focus on machine-centered automation, Industry 5.0 emphasizes human-centered production activities [2] . Within this paradigm, Human–Robot Collaboration (HRC) is * Corresponding Author: Takuya Kiyokawa. Email: [kiyokawa@sys.es.osaka-u.ac.jp](kiyokawa@sys.es.osaka-u.ac.jp) regarded as a key technology for enabling adaptive and interactive assembly tasks. Previous studies on HRC have proposed various collaborative approaches from perspectives such as motion planning [3], role allocation [4], and safety assurance [5] . They assume predefined workflows or structured commands. In real-world assembly tasks, however, there are many situations in which humans are responsible for processes that robots cannot perform independently, while robots act in a supportive role. In such scenarios, a cooperative relationship is required in which humans take the initiative and robots provide appropriate assistance to complete the task. One of the main challenges in human–robot collaborative assembly is enabling robots to understand human intentions and actions. When humans work together, they often use ambiguous verbal instructions, such as “hold it a little more to the left” or “pass me a larger tool,” to adjust each other’s actions and smoothly coordinate the task. Humans naturally resolve such ambiguity through shared context and implicit intention, but enabling robots to do the same remains a fundamental challenge. Many existing systems rely on predefined fixed mo

FAQs

What problem does the paper address in human–robot collaborative assembly?

It addresses how robots can interpret ambiguous human corrective verbal instructions and generate actions that are both semantically consistent and physically executable.

How does the proposed framework use vision–language models (VLMs)?

It maps human instructions to a set of action-target candidates (e.g., grasp poses and tool selections) selected through VLM reasoning.

What roles do the Internal and External Correction Models play?

The Internal Correction Model verifies logical consistency before execution, while the External Correction Model performs post-execution visual verification to support recovery when outcomes differ from expectations.

Jul 16, 2026

Upload by Finn