A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments - initial case study

Aug 4, 2026

Upload by Noah

Investigating children’s embodied learning in mixed-reality environments requires analyzing complex multimodal data to interpret learning and coordination behaviors during collaborative scientific simulations. Traditional Interaction Analysis workflows depend on hours of video inspection, which slows researchers’ ability to extract learning patterns. This study reduces researcher effort by combining machine learning and multimodal learning analytics to support Interaction Analysis and streamline understanding of students’ engagement through movement, gaze, and affect. An initial case study evaluates the feasibility of representing students’ state, actions, gaze, affect, and movement on a timeline, focusing on a photosynthesis learning scenario.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Introduction

Embodied learning and its data challenges

Interaction Analysis and the need for AI support

Multimodal data sources for embodied learning

AI-in-the-loop and study contributions

arXiv :2405 .06203v1 [ cs .AI] 10 May 2024 A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments Joyce Fonteles 1[0000−0001−9862−8960], Eduardo Davalos 1[0000−0001−7190−7273], Ashwin T. S. 1[0000−0002−1690−1626], Yike Zhang 1[0000−0003−3503−2996], Mengxi Zhou2[0009−0003−6902−0325], Efrat Ayalon 1[0009−0006−6679−2452], Alicia Lane 1[0009−0002−1589−621X], Selena Steinberg2[0000−0003−0032−8957], Gabriella Anton 1[0000−0002−0117−053], Joshua Danish2[0000−0001−5119−5897], Noel Enyedy 1[0000−0001−7662−5654], and Gautam Biswas 1[0000−0002−2752−3878] 1 Vanderbilt University, Nashville TN 37240, USA {joyce.h.fonteles,[gautam.biswas}@Vanderbilt.edu](gautam.biswas}@Vanderbilt.edu) 2 Indiana University, Bloomington IN 47405, USA Abstract. Investigating children’s embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination behaviors. Learning scientists have developed Interaction Analysis (IA) methodologies for analyzing such data, but this requires researchers to watch hours of videos to extract and interpret students’learning patterns. Our study aims to simplify researchers’ tasks, using Machine Learning and Multimodal Learning Analytics to support the IA processes. Our study combines machine learning algorithms and multimodal analyses to support and streamline researcher efforts in developing a comprehensive understanding of students’ scientific engagement through their movements, gaze, and affective responses in a simulated scenario. To facilitate an effective researcher-AI partnership, we present an initial case study to determine the feasibility of visually representing students’ states, actions, gaze, affect, and movement on a timeline. Our case study focuses on a specific science scenario where students learn about photosynthesis. The timeline allows us to investigate the alignment of critical learning moments identified by multimodal and interaction analysis, and uncover insights into students’ temporal learning progressions. Keywords: Multimodal learning analytics · Embodied learning · Machine learning · Interaction analysis. 1 Introduction Embodied learning aligns with the natural ways in which humans perceive, interact, and learn from the world around them. By engaging the body in the learning process, we create richer, more immersive educational experiences where our actions, movements, and interactions contribute significantly to how we understand 2 Fonteles et al. and internalize concepts [6] . It allows students to actively explore and embody knowledge through perception, awareness, and exploration of their environment. Embodiments not only enhance retention and a deeper understanding of abstract or complex concepts; it leverages the power of immersive experiences to make education more engaging and impactful [10] . Embodied learning data analysis presents a great challenge due to the complexity of monitoring student groups spatially and temporally. Conventional educational settings focus mostly on verbal communication and digital system interactions. Meanwhile, embodied learning necessitates the capture of non-verbal cues and body movements in 3D space, along with conversations and simulation logs [6] . Interaction Analysis (IA) is one of the main approaches employed by learning scientists because it can unravel deep insights and nuanced interactions captured in video data [13] . IA yields valuable insights, but its manual processes are time-consuming and demand substantial human resources. Therefore, recent advances in Machine Learning (ML) and Multimodal Learning Analytics (MMLA) make it easier to leverage algorithms to support human analysis, with the idea that the combination will allow researchers and educators to gain a nuanced understanding of how learners engage with content, facilitating feedback,

FAQs

What problem does the study address in embodied learning research?

Embodied learning generates complex multimodal data, and traditional Interaction Analysis requires time-consuming manual video watching to extract learning and coordination patterns.

How does the study reduce researchers’ workload?

It applies machine learning and multimodal learning analytics to support Interaction Analysis, streamlining how researchers interpret students’ engagement using movement, gaze, and affect.

What does the initial case study evaluate?

It tests the feasibility of visually representing students’ state, actions, gaze, affect, and movement on a timeline, aligned with learning moments identified by multimodal and interaction analysis in a photosynthesis scenario.

Aug 4, 2026

Upload by Noah