[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119000-en":3,"doc-seo-119000-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119000,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","Machine-Learning-Enabled Gestural Interaction in Mixed Reality - Doctoral Dissertation","Mixed Reality (MR) blends real and digital environments to create an integrated interactive space, enabling new computing and interaction paradigms on devices such as Microsoft HoloLens and Meta Quest. Broad everyday adoption remains limited, largely due to the absence of frictionless interaction. This dissertation investigates controller-free, bare-hand gestural interaction in MR, proposing machine-learning-enabled systems for mid-air gesture text entry and for command elicitation through gesture recognition. It addresses data sparsity with generative skeleton synthesis, personalizes keyboard layout via multi-objective Bayesian optimization, improves gesture-trajectory decoding under tracking latency, and supports real-time key-gesture spotting with an interactive application for nonexperts.","Machine-Learning-Enabled Gestural Interaction in Mixed Reality  \nJunxiao Shen  \nDepartment of Engineering  \nUniversity of Cambridge  \nThis dissertation is submitted for the degree of  \nDoctor of Philosophy  \nTrinity College September 2023  \nDeclaration  \nThis thesis is the result of my own work and includes nothing which is the outcome of work done in collaboration except as declared in the preface and specified in the text. It is not substantially the same as any work that has already been submitted, or, is being concurrently submitted, for any degree, diploma or other qualification at the University of Cambridge or any other University or similar institution except as declared in the preface and specified in the text. It does not exceed the prescribed word limit for the relevant Degree Committee  \nJunxiao Shen  \nSeptember 2023  \nAbstract  \nMixed Reality (MR) is a term used to describe the seamless blending of a physical environment with a digitally generated environment, creating an integrated space where real and virtual elements coexist and interact. It has introduced the possibility of new computing and interaction paradigms for users with the use of commercial headsets such as Microsoft HoloLens and Meta Quest. However, various factors have prevented these products from becoming widely adopted as everyday devices. The lack of frictionless interactions is one of the preventive factors. The thesis focuses on gestural interactions that do not require the use of controllers and are instead performed with bare hands. However, gestural interactions in MR are still in their infancy, with potential for exploration and development. The direction of gestural interaction in this thesis is partitioned into two parts: mid-air gesture keyboard for text entry, and gesture recognition for eliciting commands. They are similar to the keyboard and mouse used on personal computers. Machine learning has significantly advanced various technologies; thus the central hypothesis in this thesis is that Machine learning enables fast and accurate gestural interaction systems in Mixed Reality.  \nThe design of machine-learning-enabled gestural interaction systems faces many challenges. Standard machine learning models require a significant amount of data for training so that complex patterns can be recognized from the data. Owing to the limited adoption of MR devices and considering that novel interaction systems are proposed, acquiring large amounts of data can be time-consuming and challenging. Thus, data sparsity poses the first challenge, which leads to Research Question 1 : How to use generative machine learning models to synthesize skeleton gestural data? This thesis proposesa novel model, the Imaginative Generative Adversarial Network (GAN), to automatically synthesize skeleton-based hand gesture data aimed at data augmentation for training gesture classification models. The results demonstrate that the proposed model trains quickly and can enhance classification accuracy compared to conventional data augmentation strategies. Furthermore, this model is extended to generate trajectory data for mid-air gesture keyboards, compare it with other generative models, and discuss the comparative advantages of each model.  \nDesigning a mid-air gesture keyboard involves both user interface and user experience design. MR introduces a vast design space through its high-degree of interactivity and expansive display area. Consequently, the optimal design for a mid-air gesture keyboard size varies among users due to differences in users’ gesture motions and preferences. Thus, Research Question 2 is: How to use machine learning-based optimization methods to adaptively personalize the size of a mid-Air gesture keyboard? This thesis proposes a multi-objective Bayesian optimization approach for adapting the layout size of a mid-air gesture keyboard to individual users. The results demonstrates that this  \nprocess can achieve a 14.4% improvement in speed and a 13.","cbCaiiBg5QDlGJBC","https://ap.wps.com/l/cbCaiiBg5QDlGJBC","pdf",34509062,1,170,"English","en",105,"# Abstract\n## MR and gestural interaction motivation\n## Generative models for data augmentation\n## Machine-learning personalization of mid-air keyboard size\n## Gesture-trajectory decoding for intended text\n## Real-time gesture recognition and nonexpert usability","[{\"question\":\"What central idea does the dissertation explore about mixed reality gestural interaction?\",\"answer\":\"It tests the hypothesis that machine learning enables fast and accurate gestural interaction systems in mixed reality, especially for controller-free bare-hand input.\"},{\"question\":\"How does the thesis address the data-sparsity challenge for gesture systems?\",\"answer\":\"It proposes a generative model, the Imaginative Generative Adversarial Network (GAN), to synthesize skeleton-based hand gesture data for data augmentation during training.\"},{\"question\":\"What techniques are proposed for mid-air gesture text entry and real-time recognition?\",\"answer\":\"For text entry, it introduces a gesture trajectory decoding model and a multi-objective Bayesian optimization approach to adapt keyboard layout size per user; for recognition, it proposes a key gesture spotting architecture with real-time activation and evaluation on multiple datasets.\"}]","Machine-Learning-Enabled Gestural Interaction in Mixed Reality - Doctoral Dissertation | PDF",1785721634,428,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"machine-learning-enabled-gestural-interaction-in-mixed-reality-doctoral-dissertation","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/machine-learning-enabled-gestural-interaction-in-mixed-reality-doctoral-dissertation/119000/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What central idea does the dissertation explore about mixed reality gestural interaction?","Question",{"text":75,"@type":76},"It tests the hypothesis that machine learning enables fast and accurate gestural interaction systems in mixed reality, especially for controller-free bare-hand input.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the thesis address the data-sparsity challenge for gesture systems?",{"text":80,"@type":76},"It proposes a generative model, the Imaginative Generative Adversarial Network (GAN), to synthesize skeleton-based hand gesture data for data augmentation during training.",{"name":82,"@type":73,"acceptedAnswer":83},"What techniques are proposed for mid-air gesture text entry and real-time recognition?",{"text":84,"@type":76},"For text entry, it introduces a gesture trajectory decoding model and a multi-objective Bayesian optimization approach to adapt keyboard layout size per user; for recognition, it proposes a key gesture spotting architecture with real-time activation and evaluation on multiple datasets.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]