Interactive Machine Learning for Movement Interaction in VR - paper title

Aug 4, 2026

Full body movement is a central interaction channel for virtual reality, and it can influence both realism and emotional experience. Designing effective movement interactions is difficult because human movement knowledge is tacit and embodied, making explicit programming unreliable. Interactive Machine Learning offers an example-driven alternative by learning movement-to-effect mappings. The paper introduces IntearctML, an IML-based movement interaction design platform, and demonstrates it through a case study building Dolittle VR, a movement-centric experience for interacting with abstract animals.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages4

Reading Time10 min

AccessFree to download

Introduction

Core role of movement in VR

Tacit movement knowledge and IML motivation

InteractML

Library overview and data processing

Workflow: bodystorming, training, and testing

Node-based configuration

Dolittle VR

Gesture-and-dance interaction design

Interactive Machine Learning for Movement Interaction in VR Tom Lawrence Tianyuan Zhang Clarice Hilton Marco Gillies* Department of Computing Goldsmiths, University of London London, UK ABSTRACT Full body movement is a powerful way of interacting with virtual reality experiences. Not only does it reproduce real world interactions, it can also have positive effects on emotions. However, designing effective movement interaction can be hard, as our knowledge about how we move is tacit and embodied, meaning that we can move without knowing exactly how we make those movements. This makes it hard to explicitly program movement interaction. Interactive Machine Learning (IML) is an alternative approach in which movement interaction is designed and implementing by providing examples of movement. This paper presents IntearctML, a movement interaction design platform based on IML, as well as a case study of using it to create a VR experience called Dolittle VR. Index Terms: Virtual Reality, Movement Interaction, Interactive Machine Learning. 1 INTRODUCTION Mel Slater’s theory of Virtual Reality[10] places movement at the core of the VR experience. The two core illusions, Place Illusion and Plausibility Illusion, are based on reproducing the sensorimotor contingencies[9] that relate our movement and perception in the real world. This means that our interactions with VR should mirror our interactions with the real world, i.e. they should use the movement of our full bodies. This is already common place in VR where head movement is key to viewing the world[10], walking to how we navigate it[11], and our hands to how we interact with objects[1] . There are multiple other benefits of using movement interaction[5], such as making use of real world skills[7] or embodied cognition[8] . One of the challenges of developing movement interaction techniques is that much of our movement knowledge is embodied and tacit[4], meaning that we can perform a task such as riding a bicycle without being able to put into words exactly how we do it. If we cannot put our movement knowledge into worlds, we are unlikely to be able to express it in program code. Machine learning methods can address this issue, because they make it possible to implement movement interaction by giving examples of movements, rather than explicit coding. However, traditional machine learning is a batch process requiring the gathering of large amounts of data in a long process that conflicts with the rapid prototyping typical of interaction design. The use of Interactive Machine Learning[2] techniques that aim to reproduce the iterative workflow of design process, have also been shown to benefit the designers of movement interaction by allowing them to design in more embodied ways[3] . This paper presents InteractML, a platform for interactive machine learning based movement interaction design. It also presentsa case study of using InteractML to develop DolittleVR a virtual reality experience that relies on movement as a way of interacting with abstract virtual animals. *[m.gillies@gold.ac.uk](m.gillies@gold.ac.uk) Figure 1: The InteractML process. 2 INTERACTML InteractML[6] is an interactive machine learning library built for the 3D game engine Unity. It can build interaction using any real time data however it is specifically tailored for movement interaction. It allows users to design movement interaction by actually performing the movements. Real time movement data can be processed by the machine learning model to produce responsive effects in the environment, this is particularly useful for virtual reality applications where user motion is a key feature. It is built for iterative and collaborative interaction design, where users can ideate as they train the model. This should produce expressive movement interaction owing to the creative freedom in the workflow. Another strength of InteractML is that interaction is built from real movement data which is

FAQs

Why is movement interaction important in VR?

Movement sits at the core of the VR experience by supporting sensorimotor contingencies that drive Place Illusion and Plausibility Illusion. Using full-body movement helps interactions mirror real-world behavior.

What problem does interactive machine learning address for movement interaction design?

Movement knowledge is tacit and hard to translate into explicit code. IML enables implementing interactions by learning from examples of performed movements rather than relying on manually programmed rules.

How does InteractML support an interactive design workflow?

InteractML supports iterative collaboration by letting designers bodystorm movements, train a model with real movement data, and test the resulting interaction by performing it again. It is configured through a visual node-based graph in Unity.

Aug 4, 2026

Upload by River Wang