EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

Jul 16, 2026

Upload by Logic

Vision-language models (VLMs) are proposed as runtime safety guards for embodied agents, where reliable monitoring must distinguish genuinely unsafe moments from routine yet superficially alarming activity. EgoSafetyBench introduces a diagnostic egocentric video benchmark with 1,200 robotview scenarios, annotated every half-second. Two tracks evaluate streaming guard behavior: situational hazards and visual-channel text mismatches. Contrastive ladders isolate single deciding cues to measure true physical-safety reasoning. Experiments on ten VLMs show hazard video recognition yet systematic failures on contextual hazards, and misleading in-scene signs harm both accuracy and intervention calibration.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Abstract

Introduction

Runtime safety guards for embodied agents

Limitations of binary safety benchmarks

EGOSAFETYBENCH design and annotation axes

Contrastive ladder methodology

Evaluation setup and findings

EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards arXiv :2607 .00218v1 [ cs .CV] 30 Jun 2026 Siddhant Panpatil 1 * Arth Singh 1∗ Mijin Koo 1 ,2 Chaeyun Kim 1 ,2 Haon Park1 ,2 Dasol Choi 1† 1AIM Intelligence 2 Seoul National University {sid, arth, mijinkoo, chaeyun, haon, [dasol.choi](dasol.choi}@aim-intelligence.com)[}](dasol.choi}@aim-intelligence.com)[@aim-intelligence.com](dasol.choi}@aim-intelligence.com) HuggingFace GitHub Abstract Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genuinely unsafe situations while avoiding unnecessary intervention on routine but superficially alarming activity, a distinction that binary safety benchmarks obscure. We introduce EGOSAFETYBENCH, an egocentric video benchmark of 1,200 robotview scenarios annotated at half-second granularity, to evaluate VLMs as streaming guards across two tracks. The situational track (800 scenarios) spans four families, from routine and safe-but-suspicious scenes to obvious and contextual hazards. The visual-channel track (400 scenarios) targets in-scene text—a sign, sticker, or label visible in the scene—that can misrepresent the physical situation, pairing each misleading sign with a truthful version to test both whether a guard flags the text as misleading and whether the text corrupts its physical-safety judgment. Both tracks use contrastive ladders: near-identical scenarios differing only in a single visible deciding cue, so a correct call must hinge on that cue rather than the overall scene type. We evaluate ten open- and closed-source VLMs. We find that while guards reliably recognize videos containing hazards, they often miss specific hazardous moments, particularly contextual hazards. Furthermore, misleading inscene signs degrade all tested guards: vulnerable models miss up to a third of hazards, while robust models overintervene on safe content. Matched controls reveal that apparent safety robustness often reflects indiscriminate alarming rather than true physical reasoning. *Equal contribution.†Corresponding author. 1. Introduction Embodied agents powered by vision-language models (VLMs) [8, 16, 29] are increasingly deployed in humancentric environments [25] . This highlights the need for runtime safety guards [5, 15] that monitor the robot’s egocentric stream and interrupt unsafe behavior before harm occurs. Such a guard faces two competing requirements: it must detect genuinely unsafe situations, including context-dependent risks, while allowing routine activity that is inherently benign despite alarming surface features. For instance, a kitchen knife is not inherently dangerous; safety depends entirely on the dynamic relation between the robot’s action and the scene context. However, most embodied-safety benchmarks [9, 18, 20, 23, 27] collapse this nuanced problem into binary safe/unsafe classification. This conflates two distinct failures: over-flagging safe scenes, and missing contextual hazards. Consequently, binary evaluation cannot distinguish whether a model is a reliable runtime guard or merely a surface-cue detector. To expose these distinctions, we introduce EGOSAFETYBENCH, an egocentric video benchmark evaluated chunkby-chunk in an online, future-blind setting that approximates a streaming guard. Each scenario is rendered from a humanoid robot’s first-person perspective and annotated at half-second granularity. Our annotation framework decouples evaluation along two independent axes (Figure 1) . The first is a situational four-family taxonomy, from plainly safe (S1) and resolved-confound safe (S2) to obvious (U1) and contextual (U2) hazards. The second is a visual-channel mismatch (VCM) axis, marking whether an in-scene channel (a sign, sticker, or screen) misrepresents the situation [4, 28] . Importantly, every misleading channel is paired with a truthful co

FAQs

What problem does EgoSafetyBench address for embodied VLMs in runtime safety?

It addresses the need for a guard that catches genuinely unsafe situations while not over-intervening on routine activity that only looks alarming on the surface.

How does EGOSAFETYBENCH structure the benchmark cases?

It provides 1,200 egocentric robotview scenarios annotated at half-second granularity, split into a situational track (800) and a visual-channel text track (400).

What is the purpose of the contrastive ladder design?

Scenarios differ only by a single visible deciding cue, forcing correct predictions to rely on the safety-relevant relationship rather than general scene correlations.

What key evaluation trends are reported for tested VLMs?

Models often detect videos containing hazards but frequently miss specific hazardous moments, especially contextual hazards; misleading in-scene signs also degrade performance and can trigger over-intervention.

Jul 16, 2026

Upload by Logic