[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-126821-en":3,"doc-seo-126821-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":11,"language":21,"language_code":22,"site_id":23,"html_lang":22,"table_of_contents":24,"faqs":25,"seo_title":26,"seo_description":14,"update_tm":27,"read_time":28},126821,1099523885074,"Ivy","https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc",8,"Research & Report","When XR Meets AI - Integrating Interactive Machine Learning with an XR Musical Instrument","This paper presents Netz, an XR musical instrument (XRMI) that integrates artificial intelligence with extended reality to improve expressive control. Netz uses deep learning to map physical hand gestures to musical parameters, reducing the impact of sensing errors and enabling customizable control schemes. A participatory design process with a professional keyboard player and music producer guided development across exploration, making, and performance & refinement phases. Interactive machine learning addressed challenges in hand-pose classification and enabled personalized gesture control. Musical performance tasks evaluated playability and expressivity, and interview-based thematic analysis indicates that the IML system strengthens musical interaction. Future work will test wider musician populations to assess generalisability.","Audio Engineering Society  \nConference Paper  \nPresented at the AES International Symposium on AI and the Musician 2024 June 6–8, Boston, MA, USA  \nThis conference paper was selected based on a submitted abstract and 750 -word precis that have been peer reviewed by at least two qualified anonymous reviewers. The complete manuscript was not peer reviewed. This conference paper has been reproduced from the author's advance manuscript without editing, corrections, or consideration by the Review Board. The AES takes no responsibility for the contents. This paper is available in theAES E-Library ([http://www.aes.org/e-lib](http://www.aes.org/e-lib)), all rights  \nreserved. Reproduction of this paper, or any portion thereof, is not permitted without direct permission from the Journal of the Audio Engineering Society.  \nWhen XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument  \nMax Graf1 and Mathieu Barthet 1  \n1 Queen Mary University of London  \nCorrespondence should be addressed to Max Graf ([max.graf@qmul.ac.uk](max.graf@qmul.ac.uk))  \nABSTRACT  \nThis paper explores the integration of artificial intelligence (AI) with extended reality (XR) through the development of Netz, an XR musical instrument (XRMI) designed to enhance expressive control using deep learning techniques. Netz implements algorithms to map physical gestures to musical controls, offering customisable control schemes that enhance gesture interpretation accuracy and elevate the overall musical experience. The instrument was developed through a participatory design process involving a professional keyboard player and music producer. The process spanned three phases with corresponding design sessions: exploration, making, and performance & refinement. Initial challenges with traditional computational approaches to hand-pose classification were overcome by incorporating an interactive machine learning (IML) system, enabling personalised gesture control. A set of musical performance tasks encompassing melodies and chord progressions were used to assess the instrument's playability and expressivity in collaboration with our musician partner. Thematic analysis of reflective interviews revealed that the IML system enhanced musical interaction, suggesting AI's potential to improve XR musical performance. Future work will involve a wider range of musicians to assess the generalisability of our findings.  \n1 Introduction  \nIn the evolving landscape of digital music, the merging of extended reality (XR), encompassing augmented, mixed, and virtual reality, and AI holds the potential to unlock new avenues for musical expression and instrument interaction. This paper describes the development of Netz, an XR musical instrument (XRMI), which exemplifies this synergy by utilising deep learning to personalise gesture control for expressive musical interaction. Netz is a hand-controlled XR instrument that  \nuses machine learning (ML) to refine the translation of musicians' gestures into musical controls. The use of ML reduces the impact of technological sensing errors and empowers expressive control in XR music creation. Ontologically, our AI approach focuses on augmenting musicians' control over the instrument, providing them with new creative possibilities. This differs from AI aimed at automating tasks traditionally done by musicians, such as generative modelling for composition or performance. XR head-mounted displays (HMDs) blend physical and virtual spaces through  \nhead-tracking, stereoscopy and real-world scene understanding, enabling new forms of instrument design and interaction. Zellerbach and Roberts broadly defined an XRMI as an “embodied system for expressive musical performance, characterized by the relationships between the performer, the virtual, and the physical environment” [1] . Out of the many current issues in interaction design for XR, we want to highlight two in particular: sensing-based technological faults [2] and accurate and timely con","cbCaieV6x7kYfbpI","https://ap.wps.com/l/cbCaieV6x7kYfbpI","pdf",271184,1,"English","en",105,"# Abstract\n# Introduction\n# Background\n# Interactive Machine Learning Pipeline\n# Evaluation and Musical Tasks\n# Participatory Design Process\n# Discussion and Future Work","[{\"question\":\"What is Netz and what problem does it address in XR musical performance?\",\"answer\":\"Netz is an XR musical instrument that enhances expressive hand gesture control. It addresses inaccuracies and glitches caused by sensing and tracking limitations in XR devices.\"},{\"question\":\"How does interactive machine learning contribute to Netz’s gesture control?\",\"answer\":\"Interactive machine learning refines the translation of musicians’ gestures into musical controls. It improves hand-pose recognition and enables personalized gesture-to-parameter mapping.\"},{\"question\":\"How was Netz developed and evaluated with musicians?\",\"answer\":\"Netz was created through a participatory design process with a keyboard player and music producer across exploration, making, and performance \\u0026 refinement phases. Evaluation used musical performance tasks (melodies and chord progressions) and reflective interviews analyzed thematically.\"}]","When XR Meets AI - Integrating Interactive Machine Learning with an XR Musical Instrument | PDF",1785935002,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":23,"language":22,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"when-xr-meets-ai-integrating-interactive-machine-learning-with-an-xr-musical-instrument","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/when-xr-meets-ai-integrating-interactive-machine-learning-with-an-xr-musical-instrument/126821/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":22,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What is Netz and what problem does it address in XR musical performance?","Question",{"text":74,"@type":75},"Netz is an XR musical instrument that enhances expressive hand gesture control. It addresses inaccuracies and glitches caused by sensing and tracking limitations in XR devices.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does interactive machine learning contribute to Netz’s gesture control?",{"text":79,"@type":75},"Interactive machine learning refines the translation of musicians’ gestures into musical controls. It improves hand-pose recognition and enables personalized gesture-to-parameter mapping.",{"name":81,"@type":72,"acceptedAnswer":82},"How was Netz developed and evaluated with musicians?",{"text":83,"@type":75},"Netz was created through a participatory design process with a keyboard player and music producer across exploration, making, and performance & refinement phases. Evaluation used musical performance tasks (melodies and chord progressions) and reflective interviews analyzed thematically.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":23},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":28,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":28,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]