From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping

Jul 16, 2026

Upload by Patrick

Improvements in human–robot interaction technical performance do not necessarily translate into differences that users notice during live use. This study tests whether a 15-point increase in end-to-end task success (from 75% to 90%) produces consistent, measurable changes in user perception. A baseline combines Whisper, Florence-2, LLaMA 3.1, and an interval Type-2 fuzzy controller, while an improved setup swaps perception and language modules for Grounding DINO + SAM and Qwen 3.5 9B. In a within-subject tabletop grasping study with 24 participants, the improved system was preferred by 70.83% and scored significantly higher on speed, reliability, and competence/fluency.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract

Introduction

From Technical Metrics to User Perception: A User Study of a Multimodal Human–Robot Interaction System for Object Detection and Grasping Jian Song, Tian Zi, Shen Guanting arXiv :2607 .00530v 1 [ cs .RO] 1 Jul 2026 Abstract—Improvements in the technical performance of human–robot interaction (HRI) systems do not automatically translate into differences that human users can detect during live interaction. This paper investigates whether a 15-percentage-point gain in end-to-end task success (from 75% in a multimodal baseline system to 90% in an improved configuration identified through a prior ablation study) is sufficient to produce consistent and measurable differences in user perception. The baseline system combines Whisper for speech recognition, Florence-2 for open-vocabulary object detection, LLaMA 3.1 for action extraction, and an interval Type-2 fuzzy logic controller for motion execution. The improved configuration replaces the perception and language modules with Grounding DINO + SAM and Qwen 3.5 9B, respectively, while retaining the same controller. A withinsubject user study with 24 participants compared both systemson the same tabletop object-grasping task. After interacting with each configuration, participants rated perceived speed, reliability, and overall competence and fluency on a 7-point Likert scale. Results show that 17 out of 24 participants (70.83%) preferred the improved system (exact binomial test, p = 0.043, h = 0.43), and all three perceptual constructs were rated significantly higher for the improved configuration after Holm correction, with large to very large effect sizes (p ¡ 0.001). These findings confirm that the identified technical improvements are perceptible to users in direct interaction and underscore the importance of complementing benchmark evaluation with usercentred evidence when assessing robotic manipulation pipelines. Index Terms—Human-Robot Interaction, User Study, Object Detection, Large Language Models, Fuzzy Logic, Robotic Manipulation I. INTRODUCTION Human–robot interaction (HRI) has moved steadily from rigid, task-specific automation toward systems that can interpret human intent, adapt to changing contexts, and act in away that feels natural to non-expert users. This shift has been driven by applications in healthcare, domestic assistance, education, logistics, and collaborative manufacturing, where robots are expected to operate not only safely but also transparently and intuitively alongside people [1]–[4] . Across these settings, the accurate inference of human intention directly conditions system efficiency, operational safety, and user satisfaction [5], [6], and has therefore become a core research problem in modern HRI [7] . This requirement The authors are with the Dalian University of Technology, Dalian, [China.](China. {jian.song.robotics)[ {](China. {jian.song.robotics)[jian.song.robotics](China. {jian.song.robotics), zi.tian.robotics, [shen.guanting.99](shen.guanting.99}@gmail.com. The first author is the)[}](shen.guanting.99}@gmail.com. The first author is the)[@gmail.com](shen.guanting.99}@gmail.com. The first author is the)[. The first author is the](shen.guanting.99}@gmail.com. The first author is the)[ ](shen.guanting.99}@gmail.com. The first author is the)corresponding author. becomes especially critical when the robot operates in safetysensitive domains or in close physical proximity to people: in robot-assisted surgery, misinterpretation of the operator’s intent can have severe consequences for patient safety [8], while in collaborative transport and handover tasks, errors in intent inference can cause coordination failures and physical harm [9]–[11] . When the robot must additionally respond to spoken instructions, visual context, and physical constraints simultaneously, the design problem becomes inherently multimodal and must balance perception, language understanding, and control under uncertainty. Recent progress in foundation models

FAQs

What core question does the paper address about HRI improvements?

Whether a technical gain in end-to-end task success leads to consistent, measurable improvements in what users perceive during live interaction.

How did the authors modify the system from baseline to improved configuration?

They replaced the perception and language modules: Florence-2 and LLaMA 3.1 were substituted with Grounding DINO + SAM and Qwen 3.5 9B, while keeping the same interval Type-2 fuzzy logic controller.

What study results were found about user perception and preferences?

Seventeen of 24 participants (70.83%) preferred the improved system, and all three perceptual measures—perceived speed, reliability, and overall competence and fluency—were rated significantly higher for the improved configuration.

Jul 16, 2026

Upload by Patrick