[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86265-en":3,"doc-seo-86265-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},86265,687197207919,"Theodora","https://ap-avatar.wpscdn.com/avatar/a000253d6f5f7c60be?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779446848396160552",8,"Research & Report","ERR@HRI 3.0 Challenge Multimodal Detection of Errors and Anticipation in Human Robot Interactions","As robots are integrated into everyday human environments, reliably detecting and responding to errors is central to maintaining user trust and interaction quality. While machine learning improves error detection, many methods generalize poorly because they are tied to narrow contexts, controlled settings, or pre-extracted features. ERR@HRI 3.0 provides two crowdsourced, naturalistic video datasets for end-to-end modeling of error detection and prevention, enabling tasks for bystander reaction detection and anticipatory outcome prediction under real-world variability.","ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions  \nMaria Teresa Parreira  \n[mb2554@cornell.edu](mb2554@cornell.edu)[ ](mb2554@cornell.edu)Cornell University Ithaca, NY, USA  \nMicol Spitale  \nPolitecnico di Milano Milan, Italy  \nMaia Stiber Microsoft Research Redmond, WA, USA  \nShiye Cao  \nJohns Hopkins University Baltimore, MD, USA  \nAmama Mahmood  \nJohns Hopkins University Baltimore, MD, USA  \nChien-Ming Huang  \nJohns Hopkins University Baltimore, MD, USA  \narXiv :2607 . 11570v1 [ cs .RO] 13 Jul 2026  \nHatice Gunes  \nUniversity of Cambridge Cambridge, UK  \nABSTRACT  \nAs robots become increasingly integrated into human environments, their ability to detect and respond to errors remains critical for maintaining user trust and interaction quality. While recent advances in machine learning have improved error detection capabilities, most approaches are limited to specific contexts, controlled settings, or pre-extracted features, limiting their generalizability and applicability to real-world conditions. To address this challenge, the third edition of the ERR@HRI Challenge (ERR@HRI 3.0) provided researchers with two complementary datasets that enable end-to-end innovation in methods for both detecting and preventing errors in human-robot interaction. The challenge offered raw, non-anonymized video data from naturalistic settings: (1) the Bystander Affect Detection (BAD) dataset, containing webcam recordings of 45 participants’ spontaneous reactions to robot and human failure scenarios; and (2) the Bad Idea dataset, featuring 29 participants’ anticipatory facial responses while predicting action outcomes before failures occur. Both datasets were collected via crowdsourcing, capturing the inherent variability of real-world conditions—diverse lighting, camera angles, participant positioning, and environmental contexts. This naturalistic variability, while challenging, provides an authentic testbed for developing robust error detection systems. Participants developed multimodal machine learning models for bystander reaction detection (Track 1) and anticipatory outcome prediction (Track 2), with an optional cross-dataset generalization track (Track 3) . Three teams submitted valid models, all of which surpassed our convolutional neural network baselines. This paper describes the datasets, tasks, baselines, and results ofERR@HRI 3.0, and discusses implications for building generalizable, context-aware, and anticipatory error detection systems for human-robot interaction.  \nPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s) .  \nICMI’26, 5–9 October 2026, Napoli, Italy  \n© 2026 Copyright held by the owner/author(s) .  \nACM ISBN 978-x-xxxx-xxxx-x/YY/MM [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn)  \nWendy Ju  \nCornell University  \nIthaca, NY, USA  \nKEYWORDS  \nRobot Failure, Error Detection, Human-Robot Interaction, Multimodal Interaction, Benchmarking, Anticipation  \nACM Reference Format:  \nMaria Teresa Parreira, Micol Spitale, Maia Stiber, Shiye Cao, Amama Mahmood, Chien-Ming Huang, Hatice Gunes, and Wendy Ju. 2026. ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in HumanRobot Interactions. In Proceedings of 28th ACM International Conference on Multimodal Interaction (ICMI’26) . ACM, New York, NY, USA, 5 pages. [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn)  \n1 INTRODUCTION  \nRobot errors – deviations from expected or intended behavior [8]  \n– are not merely technical malfunctions but social events that can disrupt interaction flow, diminish user trust, an","cbCaiqZumDyQsKpI","https://ap.wps.com/l/cbCaiqZumDyQsKpI","pdf",500043,4,1,5,"English","en",105,"# Abstract\n# Introduction\n## Background and challenges in robot error detection\n## Multimodal behavioral signals for error inference\n## Limitations of existing approaches\n## Motivation for anticipatory detection","[{\"question\":\"Why is detecting robot errors important in human-robot interactions?\",\"answer\":\"Robot errors are social events that can disrupt interaction flow, reduce user trust, and degrade collaboration quality. Detecting them reliably helps maintain interaction quality and user confidence.\"},{\"question\":\"What limitations affect many existing human-robot error detection approaches?\",\"answer\":\"They often depend on task- or domain-specific knowledge, which limits generalization, and they frequently require users to explicitly report problems, causing delays. Many also rely on pre-extracted features and controlled laboratory conditions rather than raw, naturalistic signals.\"},{\"question\":\"What datasets and tracks does ERR@HRI 3.0 provide for modeling error detection and anticipation?\",\"answer\":\"ERR@HRI 3.0 provides two complementary crowdsourced, non-anonymized video datasets: BAD for spontaneous bystander reactions to failures and Bad Idea for anticipatory facial responses predicting outcomes before failures. The challenge includes tracks for bystander reaction detection, anticipatory outcome prediction, and an optional cross-dataset generalization track.\"}]",1784209907,13,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"errhri-30-challenge-multimodal-detection-of-errors-and-anticipation-in-human-robot-interactions","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/errhri-30-challenge-multimodal-detection-of-errors-and-anticipation-in-human-robot-interactions/86265/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is detecting robot errors important in human-robot interactions?","Question",{"text":75,"@type":76},"Robot errors are social events that can disrupt interaction flow, reduce user trust, and degrade collaboration quality. Detecting them reliably helps maintain interaction quality and user confidence.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What limitations affect many existing human-robot error detection approaches?",{"text":80,"@type":76},"They often depend on task- or domain-specific knowledge, which limits generalization, and they frequently require users to explicitly report problems, causing delays. Many also rely on pre-extracted features and controlled laboratory conditions rather than raw, naturalistic signals.",{"name":82,"@type":73,"acceptedAnswer":83},"What datasets and tracks does ERR@HRI 3.0 provide for modeling error detection and anticipation?",{"text":84,"@type":76},"ERR@HRI 3.0 provides two complementary crowdsourced, non-anonymized video datasets: BAD for spontaneous bystander reactions to failures and Bad Idea for anticipatory facial responses predicting outcomes before failures. The challenge includes tracks for bystander reaction detection, anticipatory outcome prediction, and an optional cross-dataset generalization track.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":22,"slug":137},19,"General","general"]