[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123247-en":3,"doc-seo-123247-105":30,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123247,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",6,"Technology","Multimodal Machine Learning for Sign Language Prediction - Research Study","Sign language recognition benefits many real-world applications such as translation tools, interpreting services, video remote interpreting, human-computer interaction, and online hand tracking for communication. Multimodal data can combine sources like video and sensors, including non-verbal emotional cues such as facial expressions and body posture. By integrating multimodal information with emotions, the study aims to better capture user intent and improve model performance. The work proposes collecting differentiated datasets covering individual signs and emotions, training multimodal and deep learning algorithms, and evaluating performance on test data.","6th International Conference on Nanotechnologies and Biomedical Engineering Proceedings of ICNBME-2023, September 20–23, 2023, Chisinau, Moldova Volume 2: Biomedical Engineering and New Technologies for Diagnosis, Treatment, and  \nRehabilitation  \nMultimodal Machine Learning for Sign Language  \nPrediction  \nYassèr Khalafaoui, Nistor Grozavu, Basarab Matei,  \nNicoleta Rogovschi  \n[https://doi.org/10.1007/978-3-031-42782-4_26](https://doi.org/10.1007/978-3-031-42782-4_26)  \nAbstract  \nNumerous applications, including translation tools, interpreting services, video remote interpreting, human-computer interaction, online hand tracking of human communication in desktop settings, real-time multi-person recognition systems, games, virtual reality settings, robot controls, and natural language communications, benefit from sign language recognition advantages. Multimodal data contains information from different sources such as video, sensors, electrocardiograms (ECGs), while emotions refer to the non-verbal cues that accompany language use, such as facial expressions and body posture. Integrating these additional sources of information helps to better understand the user’s intent, which improves the performance of the sign language recognition model. To build such a model, a set of multimodal data and emotions must be collected. This data set should be differentiated and cover different individual/isolated signs, emotions and body gestures. The model is designed to integrate multimodal data and emotions, which would involve combining different machine and deep learning algorithms adapted to different types of data. In addition, the model will need to be trained to recognize the different emotions that accompany sign language. Once the model is trained, it can be tested on the test dataset to assess its performance and also plan for a test on real data (with signing people). In this paper we propose a study to use the multi-modal machine learning for sign recognition language.  \nKeywords: sign language, multimodal data, multimodal machine learning (MML)  \n6th International Conference on Nanotechnologies and Biomedical Engineering Proceedings of ICNBME-2023, September 20–23, 2023, Chisinau, Moldova Volume 2: Biomedical Engineering and New Technologies for Diagnosis, Treatment, and  \nRehabilitation  \nReferences  \n1. Bengio,Y., Courville,A.,Vincent, P.: Representation learning: a reviewand newperspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828. IEEE Computer Society (2013)  \n2. Guo,W.,Wang, J.,Wang, S.: Deep multimodal representation learning: a survey. IEEE Access 7, 63373–63394 (2019). [https://doi.org/10.1109/ACCESS.2019.2916887](https://doi.org/10.1109/ACCESS.2019.2916887)  \n3. Yan, A.,Wang,W., Ren, Y., Geng, H.: A clustering algorithm for multi-modal heterogeneous big data with abnormal data. Front. Neurorobot. 15, 64 (2021)  \n4. Pedrycz, W., Hirota, K.: A consensus-driven fuzzy clustering. Pattern Recogn. Lett. 29(9), 1333– 1343 (2008)  \n5. Zhao, B., Kwok, J.T., Zhang, C.: Multiple Kernel clustering. In: Proceedings of the 2009 SIAM International Conference on Data Mining, pp. 638–649. Society for Industrial and Applied Mathematics (2009)  \n6. Bickel, S., Scheffer, T.: Multi-view clustering. Proc. ICDM 4(2004), 19–26 (2004)  \n7. Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1145–1153 (2017)  \n8. Yan, S., Xia, Y., Smith, J.S., Lu,W., Zhang, B.:Multiscale convolutional neural networks for hand detection. Appl. Comput. Intell. Soft Comput. 2017, 1–13 (2017)  \n9. Rao, G., Syamala, K., Kishore, P.V.V., Sastry, A.S.C.S.: Deep convolutional neural networks for sign language recognition. In: Conference on Signal Processing and Communication Engineering Systems (SPACES), India (2018)  \n10. Koller, O., Ney, H., Bowden, R.: Deep learning of mouth shapes for sign languag","cbCaioqLgnZ65pBT","https://ap.wps.com/l/cbCaioqLgnZ65pBT","pdf",171124,1,3,"English","en",105,"# Abstract\n# Keywords\n# References","[{\"question\":\"What types of data are considered for the sign language prediction model?\",\"answer\":\"The study integrates multimodal information from different sources such as video and sensors, and also incorporates emotional non-verbal cues like facial expressions and body posture.\"},{\"question\":\"Why is emotion information included in multimodal sign language recognition?\",\"answer\":\"Emotion-related non-verbal cues help capture the user’s intent alongside the linguistic content, which can improve recognition model performance.\"},{\"question\":\"How is the proposed model built and evaluated?\",\"answer\":\"The approach requires collecting differentiated multimodal datasets with varied signs, emotions, and gestures, training multimodal/deep learning algorithms for both signs and emotions, and then testing on a dataset to assess performance and plan evaluation on real signing data.\"}]","Multimodal Machine Learning for Sign Language Prediction - Research Study | PDF",1785815440,8,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":28},"multimodal-machine-learning-for-sign-language-prediction-research-study","",{"@graph":36,"@context":84},[37,53,67],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":21},"https://docshare.wps.com/document/technology/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/multimodal-machine-learning-for-sign-language-prediction-research-study/123247/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What types of data are considered for the sign language prediction model?","Question",{"text":74,"@type":75},"The study integrates multimodal information from different sources such as video and sensors, and also incorporates emotional non-verbal cues like facial expressions and body posture.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Why is emotion information included in multimodal sign language recognition?",{"text":79,"@type":75},"Emotion-related non-verbal cues help capture the user’s intent alongside the linguistic content, which can improve recognition model performance.",{"name":81,"@type":72,"acceptedAnswer":82},"How is the proposed model built and evaluated?",{"text":83,"@type":75},"The approach requires collecting differentiated multimodal datasets with varied signs, emotions, and gestures, training multimodal/deep learning algorithms for both signs and emotions, and then testing on a dataset to assess performance and plan evaluation on real signing data.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,112,117,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":110,"slug":111},50,"technology",{"id":113,"doc_module":4,"doc_module_name":46,"category_name":114,"show_sort_weight":115,"slug":116},7,"Healthcare",40,"healthcare",{"id":29,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":119,"slug":120},"Research & Report",30,"research-report",{"id":122,"doc_module":4,"doc_module_name":46,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":46,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":46,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]