ERR@HRI 3.0 Challenge Multimodal Detection of Errors and Anticipation in Human Robot Interactions

Jul 16, 2026

As robots are integrated into everyday human environments, reliably detecting and responding to errors is central to maintaining user trust and interaction quality. While machine learning improves error detection, many methods generalize poorly because they are tied to narrow contexts, controlled settings, or pre-extracted features. ERR@HRI 3.0 provides two crowdsourced, naturalistic video datasets for end-to-end modeling of error detection and prevention, enabling tasks for bystander reaction detection and anticipatory outcome prediction under real-world variability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Abstract

Introduction

Background and challenges in robot error detection

Multimodal behavioral signals for error inference

Limitations of existing approaches

Motivation for anticipatory detection

ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions Maria Teresa Parreira [mb2554@cornell.edu](mb2554@cornell.edu)[ ](mb2554@cornell.edu)Cornell University Ithaca, NY, USA Micol Spitale Politecnico di Milano Milan, Italy Maia Stiber Microsoft Research Redmond, WA, USA Shiye Cao Johns Hopkins University Baltimore, MD, USA Amama Mahmood Johns Hopkins University Baltimore, MD, USA Chien-Ming Huang Johns Hopkins University Baltimore, MD, USA arXiv :2607 . 11570v1 [ cs .RO] 13 Jul 2026 Hatice Gunes University of Cambridge Cambridge, UK ABSTRACT As robots become increasingly integrated into human environments, their ability to detect and respond to errors remains critical for maintaining user trust and interaction quality. While recent advances in machine learning have improved error detection capabilities, most approaches are limited to specific contexts, controlled settings, or pre-extracted features, limiting their generalizability and applicability to real-world conditions. To address this challenge, the third edition of the ERR@HRI Challenge (ERR@HRI 3.0) provided researchers with two complementary datasets that enable end-to-end innovation in methods for both detecting and preventing errors in human-robot interaction. The challenge offered raw, non-anonymized video data from naturalistic settings: (1) the Bystander Affect Detection (BAD) dataset, containing webcam recordings of 45 participants’ spontaneous reactions to robot and human failure scenarios; and (2) the Bad Idea dataset, featuring 29 participants’ anticipatory facial responses while predicting action outcomes before failures occur. Both datasets were collected via crowdsourcing, capturing the inherent variability of real-world conditions—diverse lighting, camera angles, participant positioning, and environmental contexts. This naturalistic variability, while challenging, provides an authentic testbed for developing robust error detection systems. Participants developed multimodal machine learning models for bystander reaction detection (Track 1) and anticipatory outcome prediction (Track 2), with an optional cross-dataset generalization track (Track 3) . Three teams submitted valid models, all of which surpassed our convolutional neural network baselines. This paper describes the datasets, tasks, baselines, and results ofERR@HRI 3.0, and discusses implications for building generalizable, context-aware, and anticipatory error detection systems for human-robot interaction. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s) . ICMI’26, 5–9 October 2026, Napoli, Italy © 2026 Copyright held by the owner/author(s) . ACM ISBN 978-x-xxxx-xxxx-x/YY/MM [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn) Wendy Ju Cornell University Ithaca, NY, USA KEYWORDS Robot Failure, Error Detection, Human-Robot Interaction, Multimodal Interaction, Benchmarking, Anticipation ACM Reference Format: Maria Teresa Parreira, Micol Spitale, Maia Stiber, Shiye Cao, Amama Mahmood, Chien-Ming Huang, Hatice Gunes, and Wendy Ju. 2026. ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in HumanRobot Interactions. In Proceedings of 28th ACM International Conference on Multimodal Interaction (ICMI’26) . ACM, New York, NY, USA, 5 pages. [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn) 1 INTRODUCTION Robot errors – deviations from expected or intended behavior [8] – are not merely technical malfunctions but social events that can disrupt interaction flow, diminish user trust, an

FAQs

Why is detecting robot errors important in human-robot interactions?

Robot errors are social events that can disrupt interaction flow, reduce user trust, and degrade collaboration quality. Detecting them reliably helps maintain interaction quality and user confidence.

What limitations affect many existing human-robot error detection approaches?

They often depend on task- or domain-specific knowledge, which limits generalization, and they frequently require users to explicitly report problems, causing delays. Many also rely on pre-extracted features and controlled laboratory conditions rather than raw, naturalistic signals.

What datasets and tracks does ERR@HRI 3.0 provide for modeling error detection and anticipation?

ERR@HRI 3.0 provides two complementary crowdsourced, non-anonymized video datasets: BAD for spontaneous bystander reactions to failures and Bad Idea for anticipatory facial responses predicting outcomes before failures. The challenge includes tracks for bystander reaction detection, anticipatory outcome prediction, and an optional cross-dataset generalization track.

Jul 16, 2026

Upload by Theodora