Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation

Jul 16, 2026

Upload by Aurora

Touch is a key modality for embodied agents, yet scaling tactile commonsense reasoning to realistic open-world settings is limited by two bottlenecks: tactile datasets are too small and lack diverse formats, and tactile signals are redundant and action-specific while existing models often treat all frames uniformly. TouchThinker addresses both data and representation issues by building TouchThinker-1M, a million-scale multi-source visuotactile dataset, and TouchThinker-Bench for open-world evaluation, plus action-aware modeling to improve representation efficiency and reasoning quality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Abstract

Introduction

Motivation and key bottlenecks

Data-side contributions

Representation-side contributions

Proposed TouchThinker framework

: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation arXiv :2606 . 1 1637v 3 [ cs .AI] 12 Jul 2026 Kailin Lyu1,3 , Di Wu1 , Pengwei Zhang1 , Yuhang Zheng2 , Yingxin Lai4 , Long Xiao1 , Kangyi Wu5 , Pengna Li5 , Chen Gao2 , Lianyu Hu6 , Xiaobin Hu2,B , Jie Hao1,B , Ce Hao3,B , Weihao Yuan7 , Shuicheng Yan2,B 1Institute of Automation, Chinese Academy of Sciences 2National University of Singapore, 3Zhongguancun Academy, 4Xiamen University 5Xi’an Jiaotong University, 6Nanyang Technological University, 7Nanjing University BCorresponding authors Figure 1: The core contributions of TouchThinker. (1) We propose an action-aware tactile encoder to enable efficient tactile representation learning. Upon it, we construct TouchThinker-1M, a million-scale, multi-source Visuotactile dataset that expands the scope of tactile reasoning. (2) We further introduce TouchThinker-Bench to support multi-dimensional evaluation of open-world tactile reasoning. (3) Across multiple tasks, TouchThinker achieves significant improvements over prior methods. Abstract Touch is a key modality for embodied agents to understand the physical world. Although recent work has incorporated tactile signals into language systems for tactile commonsense reasoning, scaling such systems to realistic openworld settings remains challenging due to two key bottlenecks: (1) current tactile reasoning datasets remain limited in format and scale, providing insufficient supervision for reasoning from tactile observations to physical commonsense and hindering the learning of transferable tactile commonsense; (2) Tactile signals are inherently redundant and action-specific, yet existing methods often overlook these properties, resulting in inefficient representations with limited semantic expressiveness. To address these limitations, we propose TouchThinker, a tactile-language framework that scales tactile commonsense reasoning to the open world from both data and representation perspectives. First, we construct TouchThinker-1M, a million-scale, multi-source tactile reasoning dataset covering 415 objects, 8 scenarios, and 7 sensor types, providing a solid data foundation for open-world generalization. We further introduce TouchThinker-Bench, an openworld benchmark with more realistic and diverse tasks. Then, we propose action-aware modeling mechanism to improve tactile representation efficiency and enable efficient reasoning. Experimental results demonstrate that TouchThinker achieves competitive performance against state-of-the-art models across multiple datasets. Our code and dataset will be made available at: [https://github.com/](https://github.com/)[ ](https://github.com/)lvkailin0118/TouchThinker. 1 Introduction Among human senses, touch is fundamental for perceiving and interacting with the physical world. It provides physical cues that vision alone cannot reliably capture, including material properties, surface texture, and contact state, thereby supporting physical reasoning and action decisions (Li et al., 2024b) . For example, when a person touches a hard grain of rice, they can infer that it is under-ripe; similarly, when touching a soft sponge, they can deduce that it is suitable for wiping based on the context. Therefore, incorporating tactile information into commonsense reasoning frameworks and developing tactile reasoning in open-world scenarios are crucial for advancing embodied intelligence. Several recent studies have begun to integrate tactile perception with large language models (Yu et al., 2024, 2025 ; Xie et al., 2026 ; Cheng et al., 2026), enabling robots to perform tactile understanding tasks under natural language instructions. Despite these advances, existing approaches still struggle to achieve reliable and robust tactile reasoning in open-world environments, primarily due to limitations in data and representation. (1) On the data side, existing public tactile reason

FAQs

How does TouchThinker improve tactile representation beyond prior approaches?

It introduces action-aware modeling to learn more efficient tactile representations, reducing noise from low-information frames and aligning action types with question semantics.

Jul 16, 2026

Upload by Aurora