Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

Jul 16, 2026

Turn-taking prediction is essential for social robots in human-human conversation, especially mediator roles where the robot must anticipate conversational dynamics rather than respond only to pauses. The Multimodal Voice Activity Projection (MM-VAP) framework extends audio-only VAP to synchronized audio-visual inputs while keeping its self-supervised future-projection objective. Pretrained audio-visual backbones are adapted using Low-Rank Adaptation for multimodal turn-taking, followed by inter-speaker attention and a semantic consistency loss over a 256-state output space. Experiments on NoXi/NoXi+J and evaluation on Haru EDR support improved mediation-oriented turn-taking performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Turn-taking in human communication

Challenges in current dialogue and robot systems

Conversation structure: TCUs and TRPs

Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders arXiv :2607 .07294v 1 [ cs .RO] 8 Jul 2026 Antonio Cano 1 ,2 , Guillermo Prez 1 , Luis Merino2 and Randy Gomez3 Abstract—Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot must anticipate conversational dynamics rather than merely react to pauses. This work presents a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs while preserving its self-supervised future-projection objective. The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks and adapts them through Low-Rank Adaptation to the multimodal turn-taking problem. After independent speaker encoding, an inter-speaker attention stage models the relational dynamics required to project future voice activity. In addition, a semantic consistency loss is introduced to regularize the 256-state output space according to higher-level dialogue activity patterns. Experiments on NoXi and NoXi+J showed improvements over the current baselines, particularly for some turn-taking events. Additional evaluation on the Haru EDR corpus further supported the suitability of this direction for mediation-oriented human-robot interaction. The source codes and pretrained models are available at [https://github.com/acano15/MM-VAP](https://github.com/acano15/MM-VAP). I. INTRODUCTION Turn-taking is the universal mechanism of human communication for structuring spoken interaction and coordinating speaker and listener roles [1] . To achieve effective Human–Robot Interaction (HRI), conversational robots must account for this human-like coordination across verbal and nonverbal behavior. In social robotics, interaction is inherently turn-based [2], so accurate interpretation of conversational flow and timing is necessary to enable fluid and effective communication. This work extends previous research [3] within the Haru social robot project, specifically when Haru acts as an embodied social mediator that supports human-human interactions [4]–[6] . In this scenario, Haru is not conceived as one of the main interlocutors, but instead as a background social mediator (Fig. 1) that helps regulate conversational flow, balance participation, manage silences, and support socio-emotional behaviors such as active listening, engagement, and empathy. This mediator role motivates the present 1 Antonio Cano and Guillermo Perez are with 4i Intelligent Insights, Seville, Spain. a.cano@4i.ai, [g.perez@4i.ai](g.perez@4i.ai) [3](3 Antonio Cano is with Universidad de Sevilla)[ Antonio Cano is with Universidad de Sevilla](3 Antonio Cano is with Universidad de Sevilla), [Seville](Seville), [Spain](Spain). [acano4@us.es](acano4@us.es) 2 Luis Merino is with Universidad Pablo de Olavide, Seville, Spain. [lmercab@upo.es](lmercab@upo.es) 3 Randy Gomez is with Honda Research Institute Japan, Saitama, Japan. [r.gomez@jp.honda-ri.com](r.gomez@jp.honda-ri.com) Fig. 1: Haru operating as social mediator in two-party interaction setting [8] . research toward enabling Haru to better understand and manage the ongoing interaction, with particular emphasis on how the conversational floor evolves over time and anticipate when it may change. Nevertheless, current dialogue systems are not yet sufficiently accurate at managing turn-taking, and this remains an active area of research because users frequently experience the effects of such failures. Typical problems include interruptions or overlaps due to speaking before or after a turn has ended, unnaturally long delays caused by late responses, and incorrect answers resulting from misunderstanding the user’s input. Furthermore, because human communication is inherently multimodal [7], these limitations, toge

FAQs

What problem does MM-VAP address for social robots?

MM-VAP targets turn-taking prediction in social robots, enabling anticipation of conversational dynamics rather than reacting only after pauses.

How does MM-VAP extend audio-only VAP?

It introduces a multimodal audio-visual voice activity projection framework that keeps the original self-supervised future-projection objective while incorporating synchronized visual information.

What training components help model future voice activity and dialogue dynamics?

The method uses pretrained audio-visual backbones adapted via Low-Rank Adaptation, an inter-speaker attention stage after independent speaker encoding, and a semantic consistency loss to regularize the 256-state output space.

Jul 16, 2026

Upload by Seraphina