TACTIC Tactile and Vision Conditioned Contact Centric Control for Whole-Arm Manipulation

Jul 16, 2026

Whole-arm manipulation requires managing direct, contact-rich interactions by distributing contact across multiple links as contacts form, slide, and break. Existing learning pipelines face coupled motion–force effects, partially observed contact under occlusion, and physical inconsistencies when contact configurations shift out of the training distribution. TACTIC introduces a receding-horizon controller that uses a contact-centric hybrid predictive model fusing RGB-D, distributed tactile sensing, and a compact proximity representation. It couples learned latent dynamics with analytical contact Jacobians, enabling MPC planning with contact-aware action sampling. Experiments in simulation outperform prior methods and real-world tests validate performance on multi-contact tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Introduction

Challenges in whole-arm contact

Limitations of existing approaches

arXiv :2607 .09218v1 [ cs .RO] 10 Jul 2026 TACTIC: Tactile and Vision Conditioned Contact Centric Control for Whole-Arm Manipulation Rishabh Madan 1 , Angchen Xie2 , Samantha Saak 1 , Andres Blanco 1 , Dohyeok Lee 1 , Sarah Grace Brown 1 , Yunting Yan 1 , Mark Zolotas3 , Jose Barreiros3 , Tapomayukh Bhattacharjee 1 1 Cornell University 2 Carnegie Mellon University 3Toyota Research Institute Fig. 1: TACTIC enables robots to perform contact-rich manipulation by utilizing the entire arm surface. While end-effector-centric control is often infeasible and unsafe under high payloads, TACTIC fuses multimodal observations such as RGB-D, proximity masks, and distributed tactile sensing into a contact-centric representation to enable whole-arm manipulation. We demonstrate its effectiveness on manipulation tasks, including side rollover and limb repositioning on a life-size manikin. Abstract—Whole-arm manipulation involves direct contact with the environment while the robot completes a task by distributing contact across multiple links as contacts form, slide, and break. This setting breaks common implicit assumptions in many learning-based manipulation pipelines: arm configuration tightly couples motion and contact forces, contact state is partially observed under occlusion, and purely learned rollouts can become physically inconsistent under distribution shift because many multi-link contact configurations are sparsely represented in the data. To address this, we propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a recedinghorizon controller for whole-arm manipulation. TACTIC uses a contact-centric hybrid predictive model that combines RGBD, distributed tactile sensing, and a compact 2D proximity representation. The model couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling rollouts of future contact configurationsand interaction forces. TACTIC integrates these rollouts into a sampling-based MPC planner with contact-aware action sampling: contact Jacobian-based projections steer sampled action sequences toward force-modulating directions, and objectives defined over predicted proximity and interaction forces trade task progress against whole-arm force regulation. We evaluate TACTIC in simulation against state-of-the-art model-based and model-free methods, and perform ablations that isolate the contribution of each design choice. Across experiments, TACTIC consistently outperforms other methods. We further demonstrate real-world performance on a robot with distributed tactile sensing across three whole-arm manipulation tasks that require multi-contact trajectories: turning over and repositioninga manikin, and goal-reaching in a 3D dynamic maze. Website: [emprise.cs.cornell.edu/tactic](emprise.cs.cornell.edu/tactic). I. INTRODUCTION Recent progress in robot learning spanning vision-languageaction (VLA) models [1], diffusion-based policies [2], and world models [3] has improved performance across a wide range of manipulation tasks [4] . By leveraging large datasets, these methods acquire priors over scenes and skills and often yield desired behavior with minimal task-specific engineering. However, a gap remains: these approaches struggle in contact-rich manipulation where success requires reasoning about and regulating interaction forces. This gap is especially pronounced in whole-arm manipulation, where the robot deliberately uses multiple arm links to make, break, and regulate contact while executing a task. Whole-arm contact is fundamental to human manipulation: people use the forearm and upper arm to brace, hold, guide, and reposition objects and bodies, distributing forces over a larger area to improve stability and comfort [5] . Robots require similar capabilities in contact-rich settings such as assistive caregiving and physical human–robot interaction (pHRI) . These challenges also arise when manipulating large, unwieldy,

FAQs

Why is whole-arm manipulation harder than end-effector-focused control?

Whole-arm manipulation must reason about multiple simultaneous contacts, where arm configuration tightly couples motion with contact forces and contact state is hard to observe under occlusion.

What problem does TACTIC address in learning-based manipulation under distribution shift?

Purely learned rollouts can become physically inconsistent because multi-link contact configurations are sparsely represented in training data, leading to unreliable contact and force predictions.

How does TACTIC enable reliable contact-centric control?

TACTIC uses a receding-horizon controller with a contact-centric hybrid predictive model that fuses RGB-D, distributed tactile sensing, and a compact proximity representation, coupling learned latent dynamics with analytical contact Jacobians for MPC planning.

Jul 16, 2026

Upload by Seraphina