Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition

Jul 16, 2026

Upload by Rowan

Long-horizon household robotics requires composing many language-conditioned vision-language-action skills, but skill boundaries are rarely explicit and local success may not guarantee global executability. A skill can satisfy its own postcondition while leaving the robot, objects, or camera views in a state where the next skill cannot start. The work studies semantic handoff failures in BEHAVIOR-1K using an agent-orchestrated execution harness with π0.5-based skill checkpoints, typed arguments, step budgets, and multi-view VLM verification that decides advance, retry, or replan. By contrasting clean skill-boundary snapshots with chained terminal states from previous skills, the paper finds that multiple skills reach high success from snapshots yet composed rollouts stall, enabling trace-based diagnostics of readiness, grounding, and control execution gaps for future VLA libraries.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Semantic handoff problem

Agent-orchestrated VLA execution harness

Diagnostic framing and contributions

arXiv :2607 .06256v 1 [ cs .RO] 7 Jul 2026 Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition Ke Rui* , Yushen Zuo* , Jiawei Wang*†, Haoran Jia, Jinming Ma, Weitao Zhou, Minglei Li† SimpleAI * Equal contribution †Corresponding authors: {wangjiawei, [liminglei](liminglei}@simpleai.tech)[}](liminglei}@simpleai.tech)[@simpleai.tech](liminglei}@simpleai.tech) Fig. 1: Turning on the radio through skill composition. The agent issues a sequence of typed skill calls such as move_to, pick_up_from, and press, and a multi-view VLM verifier checks each handoff before the next skill runs. When a check fails the agent recovers within the same loop—re-planning to a new sub-goal after a navigation that does not reach the radio, and retrying after a grasp that does not close—advancing only once the post-condition holds, until the radio turns on. This plan-act-verify-replan loop is our agent harness; Fig. 2 gives the full architecture. Abstract—Long-horizon household tasks require robots to compose many language-conditioned skills, but the boundary between two skills is rarely explicit. A skill may satisfy its own postcondition while leaving the robot, objects, or camera views in a state from which the next skill cannot start. We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness. The harness calls π0.5-based skill checkpoints trained from cleaned BEHAVIOR-1K demonstrations, assigns each skill typed arguments and a step budget, and uses multi-view VLM verification to decide whether execution should advance, retry, or replan. To separate isolated skill competence from long-horizon composition, we compare the same checkpoints from clean skillboundary snapshots and from chained terminal states produced by previous skills. Selected navigation, grasping, placement, and door-opening skills reach 77–100% success from snapshots under human-reviewed verification, yet composed rollouts still stall from chained states. The resulting traces attribute failures to next-skill readiness, target grounding, and control execution, turning nearzero task success into actionable diagnostics for what VLA skill libraries must learn next: robustness to the messy chained-state distribution that clean demonstrations underrepresent. I. INTRODUCTION Long-horizon household tasks require robots to maintain semantic state across many visually grounded actions. Making microwave popcorn, for example, means navigating to the kitchen, opening the microwave, inserting the bag, closing the door, and activating the appliance. No single languageconditioned action solves such a task; it requires composing many skills while preserving the conditions that make the next skill executable. A natural architecture is to let an agent orchestrate visionlanguage-action (VLA) skills. In this view, VLA policies act as general-purpose, language-conditioned visuomotor tools, while the agent layer handles task decomposition, state tracking, verification, recovery, and evidence collection. Compared with hand-engineered task and motion planners or symbolic tool APIs[20, 8], this boundary promises broader coverage: new objects and appliances need not each come with a new symbolic controller. The challenge is that learned skills exposea less explicit interface. They may complete the local behavior requested by the agent without producing a state that is usable by the next skill. We call this the semantic handoff problem (Sec. II) . A skill can satisfy its own postcondition yet leave the robot, objects, or camera views in a state from which the following skill cannot start. For example, a navigation skill before a grasp should not be considered complete merely because the target object is visible somewhere in the scene; the object should be close enough and positioned plausibly for grasping. Similarly, opening a door may satisfy an open-door postcondition while m

FAQs

What is the semantic handoff problem in agent-orchestrated skill composition?

A skill may satisfy its own local postcondition yet leave the robot, objects, or camera views in a state where the next skill cannot start. The boundary between skills is therefore not explicitly guaranteed by local success.

How does the proposed execution harness decide whether to advance, retry, or replan?

Each typed skill call includes a step budget and an expected handoff-aware postcondition. Progress is verified using a multi-view VLM verifier on head and wrist observations, and the agent advances, retries, or replans based on the verification result.

Why do some skills succeed from clean snapshots but still fail during composed rollouts?

The paper contrasts human-curated clean skill-boundary snapshots with chained terminal states produced by earlier skills. Composed execution can stall because next-skill readiness, target grounding, or control execution are not met by the messy chained-state distribution.

Jul 16, 2026

Upload by Rowan