Breaking the 15% Barrier: A Real-World Data-Driven System for Proactive Social Robot Triggered by User Nonverbal Cues

Jul 16, 2026

Upload by Aurelia

Retail service robots often use cascaded speech pipelines (STT–LLM–TTS), but many customer interactions start through nonverbal actions such as approaching, waving, pointing, or showing items. This work examines a real-world store deployment with a teleoperated humanoid robot and finds that 15.3% of robot utterances are triggered by users’ nonverbal behaviors rather than spoken input. Frequent service-relevant cues are defined and recognized online via a real-time, multi-person, multi-label video model. An LLM-based dialogue framework conditions responses on cue tokens and can use vision-language evidence when items are shown, enabling proactive turn-taking without hand-crafted rules.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Problem: limitations of audio-only dialogue pipelines

Real-time recognition requirement (about 1 fps)

Contributions

Data collection and quantification of nonverbal-triggered turns

Real-time multi-label recognition and proactive dialogue framework

System Overview (Fig. 1)

Identified nonverbal triggers

Breaking the 15% Barrier: A Real-World Data-Driven System for Proactive Social Robot Triggered by User Nonverbal Cues Yuga Yano 1 ,2 , Yuki Okafuji2 ,3 , Ryo Miyoshi2 ,3 , Sanae Yamashita2 ,3 , Yoshiki Ohira2 ,3 arXiv :2607 . 11633v1 [ cs .RO] 13 Jul 2026 Abstract—Service robots in retail stores increasingly rely on cascaded speech pipelines (STT–LLM–TTS), yet many customer–robot interactions are initiated or guided by nonverbal behaviors such as approaching, waving, pointing, or showing items. This paper studies such cues in a real-world store deployment with a teleoperated humanoid robot and shows that anon-negligible portion of robot turns are triggered by nonverbal behaviors rather than spoken input, revealing a limitation of audio-only dialogue systems. In a 6-day in-the-wild deployment, 15.3% of robot utterances were initiated by users’ nonverbal behaviors rather than spoken input. Based on an analysis of observed customer behaviors, we define a set of frequent, service-relevant nonverbal cues and develop a real-time multiperson, multi-label recognizer that runs online from video. We then propose a dialogue framework that conditions LLM-based utterance generation on recognized nonverbal cue tokens, and optionally leverages a vision–language model when items are shown, enabling proactive robot responses without hand-crafted rules. We evaluate the approach offline on nonverbal-triggered turns and demonstrate an online prototype that reacts to users’nonverbal cues in real time. I. INTRODUCTION Service robots capable of autonomous customer-service dialogue are being introduced in retail stores to reduce the workload of store staff and improve the customer experience [1]–[3] . Recently, advances in large language models (LLMs) such as ChatGPT [4] and Gemini [5], as well as speech-to-text (STT) and text-to-speech (TTS) technologies, have significantly improved the verbal interaction capabilities of service robots. Meanwhile, many nonverbal behaviors—such as waving and pointing—often trigger interactions [6] . In contrast, widely deployed spoken dialogue systems are often implemented as a cascaded pipeline: the input speech is transcribed by STT, the transcript is fed into an LLM to generate a response, and the response is synthesized by TTS [7] . In such systems based on only audio input, spoken dialogue systems cannot incorporate users’ nonverbal behaviors as inputs, which leads to missed opportunities for engagement. While many studies of human activity recognition have focused on developing dedicated models, recent advances in large-scale infrastructure models have made it possible to describe detailed user behavior using vision language models (VLMs) [8]–[10] . However, such VLM-based approaches require a high-performance GPU for real-time recognition, and significant challenges remain before deployment as a 1 Kyushu Institute of Technology, Fukuoka, Japan 2 CyberAgent, Tokyo, Japan 3 The University of Osaka, Osaka, Japan For correspondence: [yano.yuuga158@mail.kyutech.jp](yano.yuuga158@mail.kyutech.jp) Coming / waving Showing items Fig. 1. In this study, we (a) collect dialogue data from a remote-controlled robot installed in a physical store,(b) analyze utterances resulting from the user’s nonverbal behavior, and (c) propose an autonomous dialogue system based on that analysis. We build a dialogue robot that can respond to the user’s nonverbal behavior in real time. customer-service dialogue system. In this study, real-time recognition refers to a speed of at least 1 fps. This is because humans generally expect a response delay of approximately one second to user actions [11]; therefore, human activity recognition must be completed within this time frame, even when the time required for speech generation and voice synthesis is excluded. Therefore, it is essential to analyze nonverbal behaviors that play important roles in customer-service interactions and to develop a dialogue framework that

FAQs

What percentage of the robot’s utterances are triggered by users’ nonverbal behaviors in the real-world deployment?

In the 6-day in-the-wild deployment, 15.3% of robot utterances were initiated by users’ nonverbal behaviors rather than spoken input.

Why do audio-only cascaded speech pipelines limit service-robot interactions?

Because they rely on STT output and cannot incorporate nonverbal behaviors (e.g., waving, pointing, showing items) as inputs, causing missed engagement opportunities.

How does the proposed system enable proactive robot responses from nonverbal cues?

It defines frequent service-relevant nonverbal cues, recognizes them online with a real-time multi-person, multi-label video recognizer, and conditions LLM-based utterance generation on the recognized cue tokens; it can also leverage a vision-language model when items are shown.

Jul 16, 2026

Upload by Aurelia