[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123985-en":3,"doc-seo-123985-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123985,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","A Machine Learning Approach for Predicting Upper Limb Motion Intentions with Multimodal Data in Virtual Reality","Interactive virtual rehabilitation has advanced rapidly, yet objective measurement of upper-limb therapy outcomes remains constrained by difficulties in tracking progress and sustaining patient motivation. This study proposes a machine learning approach for objectively assessing user performance and motion intentions using multimodal data in a virtual reality user study with non-clinical participants. Reaching tasks are performed while collecting eye-gaze, KinArm motion, and wearable sleeve sensor resistance signals, then modeled with a two-step architecture using gaze-based segment prediction and LSTM directional prediction.","arXiv :2405 . 13023v1 [ cs .HC] 15 May 2024  \nA Machine Learning Approach for Predicting Upper Limb Motion Intentions with Multimodal Data in Virtual Reality  \nPavan Uttej Ravva Pinar Kullu  \nMohammad Fahim Abrar Roghayeh Leila Barmaki  \nUniversity of Delaware, Newark, DE, USA  \n[ravva@udel.edu](ravva@udel.edu)[pkullu@udel.edu](pkullu@udel.edu)[ ](pkullu@udel.edu)[fahim@udel.edu](fahim@udel.edu)[rlb@udel.edu](rlb@udel.edu)  \nAbstract  \nOver the last decade, there has been significant progress in the field of interactive virtual rehabilitation. Physical therapy (PT) stands as a highly effective approach for enhancing physical impairments. However, patient motivation and progress tracking in rehabilitation outcomes remain a challenge. This work addresses the gap through a machine learning-based approach to objectively measure outcomes of the upper limb virtual therapy system in a user study with non-clinical participants. In this study, we use virtual reality to perform several tracing tasks while collecting motion and movement data using a KinArm robot and a custom-made wearable sleeve sensor. We introduce a two-step machine learning architecture to predict the motion intention of participants. The first step predicts reaching task segments to which the participant-marked points belonged using gaze, while the second step employs a Long ShortTerm Memory (LSTM) model to predict directional movements based on resistance change values from the wearable sensor and the KinArm. We specifically propose to transpose our raw resistance data to the time-domain which significantly improves the accuracy of the models by 34 .6% . To evaluate the effectiveness of our model, we compared different classification techniques with various data configurations. The results show that our proposed computational method is exceptional at predicting participant’s actions with accuracy values of 96.72% for diamond reaching task, and 97.44% for circle reaching task, which demonstrates the great promise of using multimodal data, including eye-tracking and resistance change, to objectively measure the performance and intention in virtual rehabilitation settings.  \nData and Code Availability This paper uses two types of data: gaze data collected from the HTC Vive Pro Eye tracker and resistance data collected from a wearable sleeve sensor made of carbon nanotubes. Detailed explanations of the data collection process are provided in later sections. The data and code generated during the current study are publicly accessible via the following GitHub repository link.  \nInstitutional Review Board (IRB) The research has been sanctioned by the Institutional Review Board (IRB) of the University of Delaware, ensuring compliance with all ethical norms concerning research with human participants. This encompasses the protection of participant confidentiality and the mitigation of any potential harm. Ethical approval was obtained on Feb 3, 2022 under protocol Number 1982585-1, and applies for the duration of the project.  \n1. Introduction  \nOver the past years, the use of virtual reality (VR) has increased significantly. VR involves various technologies that create an engaging, simulated digital world. Users can interact within this environment, which responds to their movements, fostering a sense of presence in the virtual environment. Moreover, there have been substantial developments in these interactive virtual settings specifically designed for the motor skills rehabilitation. These improvements are especially beneficial for patients with cognitive issues and those undergoing orthopedic rehabilitation.  \nIn addition, technology advancements in roboticsand artificial intelligence progress the rehabilitation process for upper extremity patients (Ara´ujo et al. , 2020) . Patients impacted by from stroke need to perform reaching and stretching movements to regain  \n© 2024 P.U. Ravva, P. Kullu, M.F. Abrar & R.L. Barmaki.  \nPredicting Hand Motion Intentions with Multimoda","cbCait8dFCw3hWLl","https://ap.wps.com/l/cbCait8dFCw3hWLl","pdf",1359779,1,13,"English","en",105,"# Abstract\n## Study setup and data collection\n## Two-step machine learning architecture\n## Model evaluation results\n# Introduction\n## Motivation and background\n## Framework: VR with endpoint robotics\n## Participants and reaching tasks\n## Multimodal signals: gaze and resistance data","[{\"question\":\"What problem does the proposed work address in virtual rehabilitation?\",\"answer\":\"It targets the challenge of objectively measuring rehabilitation outcomes and motion intentions while also improving progress assessment during upper-limb virtual therapy.\"},{\"question\":\"What data modalities are used in the study?\",\"answer\":\"The study collects gaze data during VR tasks and motion-related signals from a KinArm robot, along with resistance change values from a custom wearable sleeve sensor.\"},{\"question\":\"How does the machine learning architecture predict motion intentions?\",\"answer\":\"It uses two steps: first predicting which reaching task segments correspond to participant-marked points using gaze, then applying an LSTM model to predict directional movements from wearable resistance changes and KinArm signals.\"}]","A Machine Learning Approach for Predicting Upper Limb Motion Intentions with Multimodal Data in Virtual Reality | PDF",1785819636,33,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"a-machine-learning-approach-for-predicting-upper-limb-motion-intentions-with-multimodal-data-in-virtual-reality","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/a-machine-learning-approach-for-predicting-upper-limb-motion-intentions-with-multimodal-data-in-virtual-reality/123985/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the proposed work address in virtual rehabilitation?","Question",{"text":75,"@type":76},"It targets the challenge of objectively measuring rehabilitation outcomes and motion intentions while also improving progress assessment during upper-limb virtual therapy.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What data modalities are used in the study?",{"text":80,"@type":76},"The study collects gaze data during VR tasks and motion-related signals from a KinArm robot, along with resistance change values from a custom wearable sleeve sensor.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the machine learning architecture predict motion intentions?",{"text":84,"@type":76},"It uses two steps: first predicting which reaching task segments correspond to participant-marked points using gaze, then applying an LSTM model to predict directional movements from wearable resistance changes and KinArm signals.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]