When XR Meets AI - Integrating Interactive Machine Learning with an XR Musical Instrument

Aug 5, 2026

Upload by Ivy

This paper presents Netz, an XR musical instrument (XRMI) that integrates artificial intelligence with extended reality to improve expressive control. Netz uses deep learning to map physical hand gestures to musical parameters, reducing the impact of sensing errors and enabling customizable control schemes. A participatory design process with a professional keyboard player and music producer guided development across exploration, making, and performance & refinement phases. Interactive machine learning addressed challenges in hand-pose classification and enabled personalized gesture control. Musical performance tasks evaluated playability and expressivity, and interview-based thematic analysis indicates that the IML system strengthens musical interaction. Future work will test wider musician populations to assess generalisability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Abstract

Introduction

Background

Interactive Machine Learning Pipeline

Evaluation and Musical Tasks

Participatory Design Process

Discussion and Future Work

Audio Engineering Society Conference Paper Presented at the AES International Symposium on AI and the Musician 2024 June 6–8, Boston, MA, USA This conference paper was selected based on a submitted abstract and 750 -word precis that have been peer reviewed by at least two qualiﬁed anonymous reviewers. The complete manuscript was not peer reviewed. This conference paper has been reproduced from the author's advance manuscript without editing, corrections, or consideration by the Review Board. The AES takes no responsibility for the contents. This paper is available in theAES E-Library ([http://www.aes.org/e-lib](http://www.aes.org/e-lib)), all rights reserved. Reproduction of this paper, or any portion thereof, is not permitted without direct permission from the Journal of the Audio Engineering Society. When XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument Max Graf1 and Mathieu Barthet 1 1 Queen Mary University of London Correspondence should be addressed to Max Graf ([max.graf@qmul.ac.uk](max.graf@qmul.ac.uk)) ABSTRACT This paper explores the integration of artiﬁcial intelligence (AI) with extended reality (XR) through the development of Netz, an XR musical instrument (XRMI) designed to enhance expressive control using deep learning techniques. Netz implements algorithms to map physical gestures to musical controls, offering customisable control schemes that enhance gesture interpretation accuracy and elevate the overall musical experience. The instrument was developed through a participatory design process involving a professional keyboard player and music producer. The process spanned three phases with corresponding design sessions: exploration, making, and performance & reﬁnement. Initial challenges with traditional computational approaches to hand-pose classiﬁcation were overcome by incorporating an interactive machine learning (IML) system, enabling personalised gesture control. A set of musical performance tasks encompassing melodies and chord progressions were used to assess the instrument's playability and expressivity in collaboration with our musician partner. Thematic analysis of reﬂective interviews revealed that the IML system enhanced musical interaction, suggesting AI's potential to improve XR musical performance. Future work will involve a wider range of musicians to assess the generalisability of our ﬁndings. 1 Introduction In the evolving landscape of digital music, the merging of extended reality (XR), encompassing augmented, mixed, and virtual reality, and AI holds the potential to unlock new avenues for musical expression and instrument interaction. This paper describes the development of Netz, an XR musical instrument (XRMI), which exempliﬁes this synergy by utilising deep learning to personalise gesture control for expressive musical interaction. Netz is a hand-controlled XR instrument that uses machine learning (ML) to reﬁne the translation of musicians' gestures into musical controls. The use of ML reduces the impact of technological sensing errors and empowers expressive control in XR music creation. Ontologically, our AI approach focuses on augmenting musicians' control over the instrument, providing them with new creative possibilities. This differs from AI aimed at automating tasks traditionally done by musicians, such as generative modelling for composition or performance. XR head-mounted displays (HMDs) blend physical and virtual spaces through head-tracking, stereoscopy and real-world scene understanding, enabling new forms of instrument design and interaction. Zellerbach and Roberts broadly deﬁned an XRMI as an “embodied system for expressive musical performance, characterized by the relationships between the performer, the virtual, and the physical environment” [1] . Out of the many current issues in interaction design for XR, we want to highlight two in particular: sensing-based technological faults [2] and accurate and timely con

FAQs

What is Netz and what problem does it address in XR musical performance?

Netz is an XR musical instrument that enhances expressive hand gesture control. It addresses inaccuracies and glitches caused by sensing and tracking limitations in XR devices.

How does interactive machine learning contribute to Netz’s gesture control?

Interactive machine learning refines the translation of musicians’ gestures into musical controls. It improves hand-pose recognition and enables personalized gesture-to-parameter mapping.

How was Netz developed and evaluated with musicians?

Netz was created through a participatory design process with a keyboard player and music producer across exploration, making, and performance & refinement phases. Evaluation used musical performance tasks (melodies and chord progressions) and reflective interviews analyzed thematically.

Aug 5, 2026

Upload by Ivy