[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84374-en":3,"doc-seo-84374-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84374,13056703020460,"Valentina","https://ap-avatar.wpscdn.com/avatar/be000253dac470eee5d?_k=1778207105932848923",8,"Research & Report","Do Egocentric Video-Language Models Capture Both Hand-and Object-Centric Cues?","Hand–object interaction (HOI) recognition depends on accurately modeling both hand manipulations and object transformations. Existing video-language models often exploit shortcuts from spurious correlations in hands, objects, and surrounding context instead of reasoning from hand/object appearance and dynamics. This work introduces a learning paradigm combining hand–object masked training with an HOI-dynamics-aware decoder that learns hand- and object-centric embeddings via auxiliary predictions. It also proposes Cue-Isolated HOI (CI-HOI) and the DEHOI testbed, enabling controlled cue-specific evaluation through inpainting. Experiments show improved performance and transferred gains across standard and robot-related benchmarks.","arXiv :2607 .085 14v 1 [ cs .CV] 9 Jul 2026  \nDo Egocentric Video-Language Models Capture Both Hand-and Object-Centric Cues?  \nMasatoshi Tateno 1, Alexandros Stergiou2, Risa Shinoda 1,  \nYoichi Sato 1†, and Dima Damen3†  \n1 The University of Tokyo 2 University of Twente 3 University of Bristol  \nAbstract. Hand–object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand–object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and objectcentric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CIHOI), a new evaluation that assesses models’ ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand-and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand-and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.  \nKeywords: Hand-Object Interaction · Action Recognition · Masked Pre-training  \n1 Introduction  \nHand-Object Interaction (HOI) recognition is essential for understanding human actions and their effects on the environment. It enables applications such as AR/VR assistance and skill transfer for robotic manipulation. HOI inherently couples hand manipulations with object transformations, and accurate HOI understanding requires capturing both complementary aspects.  \nHowever, existing video-language models often confuse visually similar actions [34] that share hand postures and object categories but differ in their  \n† Equal Supervision.  \n2 M. Tateno et al.  \nStandard HOI Evaluation  Prediction  \nOriginal Video  \nCue-Isolated HOI (CI-HOI)  \n􀀁 HandCentric Video  \n􀀂 ObjectCentric Video  \n􀀇 Model fails to capture object transformation (dough deformation)  \nFig. 1: Cue-Isolated HOI. CI-HOI requires verb prediction representing HOI dynamics from isolated hand-or object-related observations. In this example, recognition failure can be attributed to the incorrect recognition of dough transformation.  \nunderlying interactions (e.g., in Fig. 1 top, the model recognizes knead as take) . This suggests that current models tend to rely on spurious correlations among hands, objects, and environmental context, rather than reasoning from the appearance and dynamics of the hands and objects themselves.  \nTo address this limitation, we propose a robust representation learning framework comprising two key components. First, a hand–object masked training strategy that masks patches corresponding to hands or objects, enabling the model to independently reconstruct patches by exploiting the remaining visible cues. Second, we introduce an HOI-dynamics-aware (HDA) decoder that explicitly learns hand-and object-centric embeddings, in addition to a video-level embedding, through a multi-task objective. The objective supervises hand–object locations and object/action categories, and aligns the video-level embedding with narrations describing the entire interaction.  \nTo systematically evaluate cue-specific reasoning ability, we introduce a new Cue-Isolated HOI (CI-HOI) evaluation that assesses action recognition mo","cbCaifZ69Co98rpF","https://ap.wps.com/l/cbCaifZ69Co98rpF","pdf",4511029,1,26,"English","en",105,"# Introduction\n## Hand–Object Interaction recognition and limitations of existing models\n## Proposed learning paradigm: hand–object masked training and HDA decoder\n## Cue-Isolated HOI (CI-HOI) evaluation and DEHOI testbed\n# Contributions and experimental outcomes","[{\"question\":\"What problem do the authors address in HOI recognition for video-language models?\",\"answer\":\"They address the tendency of existing models to rely on spurious correlations rather than reasoning from the visual appearance and dynamics of hands and objects themselves.\"},{\"question\":\"What are the two main components of the proposed learning paradigm?\",\"answer\":\"The approach uses hand–object masked training to reconstruct information from partial hand/object observations, and an HOI-dynamics-aware decoder that learns hand- and object-centric embeddings via auxiliary predictions.\"},{\"question\":\"How does CI-HOI evaluate cue-specific reasoning, and what is DEHOI used for?\",\"answer\":\"CI-HOI tests whether a model can predict HOI action verbs from isolated hand-related or object-related visual cues. DEHOI provides inpainted videos that disentangle hand and object observations to support this evaluation.\"}]",1784195180,66,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"do-egocentric-video-language-models-capture-both-hand-and-object-centric-cues","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/do-egocentric-video-language-models-capture-both-hand-and-object-centric-cues/84374/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem do the authors address in HOI recognition for video-language models?","Question",{"text":74,"@type":75},"They address the tendency of existing models to rely on spurious correlations rather than reasoning from the visual appearance and dynamics of hands and objects themselves.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What are the two main components of the proposed learning paradigm?",{"text":79,"@type":75},"The approach uses hand–object masked training to reconstruct information from partial hand/object observations, and an HOI-dynamics-aware decoder that learns hand- and object-centric embeddings via auxiliary predictions.",{"name":81,"@type":72,"acceptedAnswer":82},"How does CI-HOI evaluate cue-specific reasoning, and what is DEHOI used for?",{"text":83,"@type":75},"CI-HOI tests whether a model can predict HOI action verbs from isolated hand-related or object-related visual cues. DEHOI provides inpainted videos that disentangle hand and object observations to support this evaluation.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]