[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84540-en":3,"doc-seo-84540-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84540,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","Learning When to Listen: Gated Affect Fusion for Human Motion Prediction","Human motion forecasting in unconstrained real-world videos is difficult because future behavior is ambiguous and observations are noisy and multimodal. Facial affect can provide behavioral cues, yet its practical value and limitations inside forecasting models remain unclear. The study introduces a systematic, affect-conditioned pipeline using MediaPipe body pose trajectories and HSEmotion facial affect representations, together with the Gated Affect Transformer (GAT) for adaptive cross-modal fusion. Multi-horizon evaluations show naive early concatenation harms accuracy, while gating suppresses cross-modal noise; facial affect is useful only within short-to-medium windows (about 30 frames).","Learning When to Listen: Gated Affect Fusion for Human Motion Prediction  \nJingni Huang  \nUniversity of Oxford  \n[jingni.huang@kellogg.ox.ac.uk](jingni.huang@kellogg.ox.ac.uk)  \n[jingnih@gmail.com](jingnih@gmail.com)  \narXiv :2607 .00296v 1 [ cs .CV] 1 Jul 2026  \nAbstract  \nHuman motion forecasting in unconstrained real-world videos remains challenging due to the ambiguity of future behaviors and the presence of noisy multimodal observations. While facial affect potentially provides complementary behavioral cues, its practical utility and mechanistic boundaries within motion forecasting frameworks remain poorly understood. In this work, we present a systematic study investigating the utility and temporal limitations of affect-conditioned forecasting in-the-wild. We establish a rigorous multimodal pipeline combining MediaPipe body pose trajectories with HSEmotion facial affect representations, and introduce the Gated Affect Transformer (GAT) to dynamically regulate cross-modal information flow.  \nThrough extensive multi-horizon evaluations under a strict subject-wise protocol, we demonstrate that naive early cross-modal concatenation consistently degrades forecasting accuracy relative to pose-only baselines. Conversely, our proposed gating mechanism stabilizes crossmodal integration by adaptively controlling the affective stream. Crucially, controlled counterfactual experiments using shuffled and randomized affect inputs reveal that the learned gate successfully suppresses unstructured crossmodal noise while remaining responsive to plausible affective signals. Furthermore, our empirical results indicate that facial affect features provide bounded, horizondependent predictive cues strictly within short-to-medium windows (e.g., 30 frames), whereas long-term trajectories remain predominantly governed by intrinsic kinematic continuity. Our findings provide empirical evidence that facial affect should be regarded as a complementary behavioral cue rather than a dominant driver of future motion, offering practical guidance for selective multimodal fusion in unconstrained human motion forecasting.  \n1. Introduction  \nHuman motion forecasting is a fundamental problem in computer vision with applications in autonomous driving,  \nhuman–robot interaction, and virtual environments. Existing forecasting methods predominantly rely on historical body kinematics to extrapolate future motion trajectories. However, human movement is also influenced by rich behavioral signals, including facial affect, which often reflects communicative intent and precedes expressive upper-body gestures. Whether such affective cues genuinely improve future motion prediction—and under what conditions they remain useful—remains largely unexplored.  \nFigure 1 conceptually illustrates the central hypothesis investigated in this work. Rather than relying solely on observed body pose, we investigate whether synchronized facial affect provides complementary behavioral information for motion forecasting. However, integrating affect into a predictive pipeline is non-trivial. Facial expressions are inherently transient and noisy in unconstrained videos, and naive multimodal fusion can easily destabilize the learned kinematic representation, sometimes degrading performance even relative to pose-only baselines.  \nTo address these limitations, we introduce the Gated Affect Transformer (GAT), a novel multimodal framework designed to systematically study how continuous, frame-level facial affect representations can modulate future upperbody gestures through an adaptive gated fusion mechanism. Our approach leverages a dual-stream architecture: a kinematic stream utilizing normalized 2D coordinate sequences extracted via MediaPipe, and an affective stream capturing 8-dimensional emotion distribution vectors generated by HSEmotion. Instead of relying on rigid, hard-coded fusion boundaries, we design a learnable gating mechanism that continuously modulates the influence of the affect","cbCaihoVr64Bpfto","https://ap.wps.com/l/cbCaihoVr64Bpfto","pdf",765497,1,7,"English","en",105,"# Introduction\n## Problem Background and Motivation\n## Proposed Gated Affect Transformer (GAT)\n## Contributions","[{\"question\":\"Why is human motion forecasting difficult in unconstrained videos?\",\"answer\":\"Future behavior is ambiguous and the input observations are noisy and multimodal. These factors make it hard to reliably extrapolate future motion trajectories from past signals.\"},{\"question\":\"What is the core idea of the Gated Affect Transformer (GAT)?\",\"answer\":\"GAT uses an adaptive gating mechanism to dynamically regulate how affective embeddings influence the kinematic stream at each timestamp. The gate dampens unreliable affect and amplifies affect when it is predictive.\"},{\"question\":\"Under what conditions does facial affect improve motion prediction?\",\"answer\":\"Facial affect provides bounded, horizon-dependent predictive cues primarily within short-to-medium time windows (e.g., around 30 frames). Over longer horizons, predictions are dominated more by intrinsic kinematic continuity.\"}]",1784196524,18,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"learning-when-to-listen-gated-affect-fusion-for-human-motion-prediction","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/learning-when-to-listen-gated-affect-fusion-for-human-motion-prediction/84540/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why is human motion forecasting difficult in unconstrained videos?","Question",{"text":74,"@type":75},"Future behavior is ambiguous and the input observations are noisy and multimodal. These factors make it hard to reliably extrapolate future motion trajectories from past signals.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What is the core idea of the Gated Affect Transformer (GAT)?",{"text":79,"@type":75},"GAT uses an adaptive gating mechanism to dynamically regulate how affective embeddings influence the kinematic stream at each timestamp. The gate dampens unreliable affect and amplifies affect when it is predictive.",{"name":81,"@type":72,"acceptedAnswer":82},"Under what conditions does facial affect improve motion prediction?",{"text":83,"@type":75},"Facial affect provides bounded, horizon-dependent predictive cues primarily within short-to-medium time windows (e.g., around 30 frames). Over longer horizons, predictions are dominated more by intrinsic kinematic continuity.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,118,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":119,"slug":120},30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]