Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

Jul 16, 2026

Assembly action understanding is essential for effective human-robot collaborative assembly, yet it is difficult because of subtle motion, fine-grained hand–object interactions, and actions that vary in motion or partners. The work adapts vision-language models using Compositional Context Fine-Tuning (CCFT), decomposing actions into Verb/Object/Tool elements and fine-tuning with templated VQA pairs for near-deterministic outputs. A Layer-Partitioned Alternating Training (LP-AT) strategy assigns disjoint layer groups to element-specific LoRA adapters to reduce cross-task interference under limited data. HA-ViD-VQA and IKEA-ASM-VQA datasets support experiments showing consistent gains over action recognition baselines and interpretable element-level predictions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Problem and motivation

Proposed approach: CCFT

Proposed training approach: LP-AT

Dataset reformulation and evaluation

Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos Hao Zheng∗ , 1 ,4 , Jinyi Huang4 , Tiantian Zheng 1 ,2 , Xun Xu4 , Tuka Alhanai 1 ,3 arXiv :2607 . 10797v1 [ cs .CV] 12 Jul 2026 Abstract—Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand–object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated questionanswering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through elementspecific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications. Code and dataset are released at [https://github.com/x-labs-xyz/CCFT](https://github.com/x-labs-xyz/CCFT). I. INTRODUCTION Building on the strong textual reasoning of large language models (LLMs), vision–language models (VLMs) have advanced multimodal understanding by integrating visual and linguistic cues [1],[2] . Multimodal perception enables VLMs to tackle a broader range of real-world challenges, positioning them as promising foundations for human-robot collaboration (HRC), which demands highly precise environmental understanding and physical interaction. Assembly action understanding represents a critical component of collaborative robotics, as robots must comprehend complex assembly actions to effectively assist humans and acquire manipulation skills through demonstration [3],[4] . However, assembly action understanding poses unique challenges for robotic perception: subtle and intricate actions, distinct actions with similar motions, identical actions with distinct motions, and nuanced hand-object interactions. While VLMs offer considerable potential for advancing complex assembly action understanding, this application domain ∗ Hao Zheng is corresponding author: [h.zheng@nyu.edu](h.zheng@nyu.edu); 1Department of Computer Engineering, New York University Abu Dhabi, UAE; 2Center for Quantum and Topological Systems, New York University Abu Dhabi, UAE; 3 Center for AI and Robotics, NYUAD, UAE; 4 Department of Mechanical and Mechatronics Engineering, The University of Auckland, New Zealand. H.Z.: [hzhe951@aucklanduni.ac.nz](hzhe951@aucklanduni.ac.nz); J.H: [jhua658@aucklanduni.ac.nz](jhua658@aucklanduni.ac.nz); X.X: [x.xu@auckland.ac.nz](x.xu@auckland.ac.nz. T.A)[. T.A](x.xu@auckland.ac.nz. T.A) acknowledges support by CAIRand CQTS funded by Tamkeen NYUAD Research Institute Award CG010 and CG008, respectively. remains largely unexplored. Most existing VLM research targets general scene understanding, supporting only basic video summarization or simple question-answering (QA) tasks [5], and lacking specialized adaptions for recognizing complex assembly actions. Moreover, HRC applications demand deterministic outputs, which conflicts with the generative nature of VLM outputs. This paper proposes two complementary technical ingredients to address these challenges. First, a Compositional Context Fine-Tuning (CCFT) method is proposed to decompose assembly actions into semantic elements—Verb, Object, and T

FAQs

What is CCFT in this work?

CCFT decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes a vision-language model to recognize each element using templated VQA question-answer pairs for near-deterministic predictions.

How does LP-AT improve multi-task learning with limited data?

LP-AT assigns disjoint model layer groups to different action elements and trains element-specific LoRA adapters by alternating updates, limiting gradient interference across tasks while keeping parameter-efficient training.

What datasets are introduced or created, and why?

The work reformulates existing assembly video datasets (HA-ViD and IKEA-ASM) into compositional VQA datasets (HA-ViD-VQA and IKEA-ASM-VQA) with templated QA pairs and element-level annotations to validate compositional action understanding.

Jul 16, 2026

Upload by Clementine