[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85482-en":3,"doc-seo-85482-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85482,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",8,"Research & Report","Emotion Recognition in Signers","Emotion recognition in signers faces two core obstacles: grammatical facial expressions overlap with affective facial expressions, and supervised data for training is scarce. This work tackles both issues in a crosslingual setup using a new benchmark, eJSL, for Japanese Sign Language signers, and the large BOBSL dataset with subtitles. eJSL contains 1,092 video clips from two signers covering 78 utterances across seven emotion states, plus neutral. Experiments show caption-based textual emotion cues reduce data scarcity, temporal segment selection strongly affects results, and hand-motion features improve recognition beyond facial cues, yielding stronger baselines than spoken-language LLMs.","Emotion Recognition in Signers  \nKotaro Funakoshi  \nFIRST, Institute of Integrated Research Institute of Science Tokyo [funakoshi@first.iir.isct.ac.jp](funakoshi@first.iir.isct.ac.jp)  \nYaoxiong Zhu  \nICT, School of Engineering Institute of Science Tokyo [zhuyaoxiong@lr.first.isct.ac.jp](zhuyaoxiong@lr.first.isct.ac.jp)  \narXiv :2512 . 15376v2 [ cs .CV] 13 Jul 2026  \nAbstract  \nRecognition of signers’ emotions suffers from one theoretical challenge and one practical challenge, namely, the overlap between grammatical and affective facial expressions and the scarcity of data for model training. This paper addresses these two challenges in a crosslingual setting using our eJSL dataset, a new benchmark dataset for emotion recognition in Japanese Sign Language signers, and BOBSL, a large British Sign Language dataset with subtitles. In eJSL, two signers expressed 78 distinct utterances with each of seven different emotional states, resulting in 1,092 video clips.  \nWe empirically demonstrate that 1) textual emotion recognition in spoken language mitigates data scarcity in sign language, 2) temporal segment selection has a significant impact, and 3) incorporating hand motion enhances emotion recognition in signers. Finally we establish a stronger baseline than spoken language LLMs (Qwen 2 .5 and GPT-4o) .  \n1 Introduction  \nEmotion recognition is a core topic not only in natural language processing (Yun et al., 2024) but also in affective computing and human-computer interaction (Zeng et al., 2009 ; El Ayadi et al., 2011), enabling more natural and empathetic systems. Such systems are equally or more important for social minorities. Recently, more light is shed on sign language (Long et al., 2024 ; Yin et al., 2024 ; Wanget al., 2025), however, automatic emotion recognition in signers has not been explored at all. To our best knowledge, the single contribution in this direction is the EmoSign dataset for American Sign Language (ASL) (Chua et al., 2025) .  \nIn this paper, we introduce eJSL 1 , a new benchmark dataset for emotion recognition in Japanese Sign Language JSL) . We asked two signers to express 78 distinct sentences with each of seven different emotional states, resulting in 1,092 video  \n1 [https://dataverse.harvard.edu/dataverse/eJSL](https://dataverse.harvard.edu/dataverse/eJSL)  \nclips. Because human languages are highly contextdependent, any linguistic expression potentially can be expressed with any emotion. In this dataset, thus, the task is recognizing the emotions of signing signers rather than that of signed contents.  \nHere, the arising challenge is that emotion expressions in signers are further complicated because facial expressions convey both grammatical and affective information (Brentari, 1999 ; Wilbur, 2000) . For example, eyebrow movement can signal a yes/no question (Pfau and Quer, 2010) or express surprise (Valli and Lucas, 2000), creating ambiguity for emotion recognition models trained on non-signers.  \nTo address the challenge, we investigate three hypotheses: (1) caption-based weakly labeled data can support effective model fine-tuning,(2) selecting temporal segments less affected by grammatical expressions improve accuracy, and (3) hand gesture features enhance recognition beyond facial features alone. Experiments on multiple datasets validate these hypotheses and offer insights into understanding of emotional communication in signers.  \n2 Emotion Recognition and Sign Language  \nAs discussed, a unique challenge in emotion recognition in signers lies in the overlap between grammatical facial expressions (GFEs) and affective facial expressions (AFEs) . Unlike spoken language, sign language uses non-manual markers such as facial movements and head gestures to encode syntax. These signals often occur simultaneously with AFEs, making their separation critical for accurate understanding of communicative information.  \nTo this end, Silva et al. (2020) annotated their corpus with facial Action Units (AUs) ","cbCaidVa98gj0KVK","https://ap.wps.com/l/cbCaidVa98gj0KVK","pdf",736759,1,9,"English","en",105,"# Abstract\n# Introduction\n# Emotion Recognition and Sign Language\n# Datasets\n## eJSL","[{\"question\":\"What are the main challenges in emotion recognition for signers addressed in the paper?\",\"answer\":\"The paper targets two challenges: overlap between grammatical and affective facial expressions, and limited data available for supervised model training.\"},{\"question\":\"What is the eJSL dataset and how is it constructed?\",\"answer\":\"eJSL is a benchmark dataset for emotion recognition in Japanese Sign Language signers. Two signers produced 78 distinct sentences across seven emotional states plus neutral, totaling 1,092 video clips.\"},{\"question\":\"Which modeling factors improved emotion recognition in the experiments?\",\"answer\":\"Results show that textual emotion recognition from spoken-language captions mitigates data scarcity, temporal segment selection significantly impacts accuracy, and incorporating hand motion improves recognition beyond facial information alone.\"}]",1784203922,23,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"emotion-recognition-in-signers","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/emotion-recognition-in-signers/85482/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What are the main challenges in emotion recognition for signers addressed in the paper?","Question",{"text":75,"@type":76},"The paper targets two challenges: overlap between grammatical and affective facial expressions, and limited data available for supervised model training.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the eJSL dataset and how is it constructed?",{"text":80,"@type":76},"eJSL is a benchmark dataset for emotion recognition in Japanese Sign Language signers. Two signers produced 78 distinct sentences across seven emotional states plus neutral, totaling 1,092 video clips.",{"name":82,"@type":73,"acceptedAnswer":83},"Which modeling factors improved emotion recognition in the experiments?",{"text":84,"@type":76},"Results show that textual emotion recognition from spoken-language captions mitigates data scarcity, temporal segment selection significantly impacts accuracy, and incorporating hand motion improves recognition beyond facial information alone.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]