Harness VLA Steering Frozen VLAs into Reliable Manipulation Primitives via Memory Guided Agents

Jul 16, 2026

Language-conditioned manipulation needs both contact-rich control and robust planning across language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models deliver strong local visuomotor skills but degrade under deployment perturbations like semantic retargeting, goal rebinding, spatial-layout shifts, and unstable contacts. LLM coding agents provide semantic reasoning, yet fixed analytic primitives struggle with irregular grasping and articulated interactions. Harness VLA introduces a memory-augmented agentic framework that reuses a frozen VLA as a retryable contact primitive while composing it with analytic primitives learned from execution traces, improving performance on multiple benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages38

Reading Time1 h 36 min

Abstract

Introduction

arXiv :2607 .08448v 1 [ cs .RO] 9 Jul 2026 Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Yixian Zhang 1 ,∗ Huanming Zhang 1 ,∗ Feng Gao2 Xiao Li3 Zhihao Liu4 Chunyang Zhu5 Jiaxing Qiu5 Yuchen Yan5 Jiyuan Liu6 Wenhao Tang 1 Zhengru Fang7 Yi Nie 1 ,2 Changxu Wei 1 Yu Wang 1 Wenbo Ding 1 Chao Yu 1 ,† 1Tsinghua University 2 Striding AI 3Purdue University 4Institute of Automation, Chinese Academy of Sciences 5Infinigence AI 6Zhongguancun Academy 7Hong Kong University of Science and Technology ∗ Equal contribution. †Corresponding author: [zoeyuchao@gmail.com](zoeyuchao@gmail.com) Website: [https://harnessvla.github.io/](https://harnessvla.github.io/) Figure 1: Harness VLA system overview. Given a task description, RGB-D observations, and robot state, the agentic planner selects structured calls from a fixed primitive library rather than emitting low-level actions directly. The library exposes the frozen VLA as VLA ACT for contact-rich behaviors and uses analytic primitives such as MOVE TO, ROTATE, and SET GRIPPER for perception-conditioned staging, transport, posture adjustment, and release. Task Specific Memory stores successful command traces from reference-seed exploration for few-shot re-grounding, while Global Memory stores reusable success rules and failure models. The right panels summarize gains over the relevant strongest baselines, and the bottom strip illustratesa rollout that alternates sparse VLA invocations with analytic control. Abstract Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulatedobject interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25 .4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. 1 Introduction A long-standing goal of robotic manipulation is a system that reliably executes free-form natural-language instructions across changing objects, layouts, and embodiments. Two dominant paradigms approach this goal from opposite directions. End-to-end Vision-Language-Action (VLA) models learn contact-rich visuomotor control directly from robot trajectories, while LLM coding agents use language-model reasoning to compose explicit perception-and-control APIs. Each paradigm is powerful, but each assigns the wrong component too much responsibility: monolithic VLAs must absorb language grounding, long-horizon composition, and low-level control inside a single policy, whereas coding agents must realize physically delicate interactions through hand-designed or agent-generated APIs. Figure 2 visualizes our response: use analytic primitives to

FAQs

What problem does Harness VLA address in language-conditioned robotic manipulation?

It addresses failures of end-to-end VLA policies under deployment perturbations and the limitations of purely analytic primitives for irregular contact-rich manipulation tasks.

How does Harness VLA combine the frozen VLA with analytic primitives?

The planner exposes the frozen VLA as a retryable contact-rich primitive (VLA ACT) and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release.

What role do task-specific memory and global memory play?

Task-specific memory stores successful command traces for few-shot re-grounding, while global memory stores reusable success rules and failure models to guide planning across attempts.

Jul 16, 2026

Upload by River Wang