SteelBench Evaluating Vision Language Models in Real World Industrial Environments

Jul 16, 2026

Upload by Aria

Existing video benchmarks for action recognition often rely on consumer videos, egocentric recordings, or simulated industrial settings, where subjects are clearly visible and procedures are controlled. SteelBench targets real operational industrial CCTV conditions with distant workers, dust, steam, low light, glare, occlusion, and concurrent activities. The benchmark provides 1,345 densely annotated clips from 149 operational hours and 10,024 candidate videos, including per-worker actions, PPE attributes, spatial context, and safety-rule compliance. It also introduces a provenance-aware audit protocol to quantify label influence and reveal evaluation bias, showing limited model accuracy and frequent incorrect safety judgments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages40

Reading Time1 h 41 min

Abstract

Introduction

Industrial surveillance evaluation challenges

Dataset and annotation design

Provenance-aware audit protocol and findings

arXiv :2607 .05264v 1 [ cs .CV] 6 Jul 2026 SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments Suryanarayana Reddy Yarrabothula 1 ,2 Manisha Chawla 1 , Gagan Raj Gupta 1 , Kunal Sinha3 , Sashank Lekkala 1 , Ashirvadhan Dosapati 1 , Saikamal Nannuri 1 , Katragadda Ajay RamaSwamy Chowdary Gowtham 1 1Indian Institute of Technology Bhilai, 2 Steel Authority of India Limited, 3VIT Vellore [yarrabothula@iitbhilai.ac.in](yarrabothula@iitbhilai.ac.in) Abstract Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test visionlanguage models under the visual and procedural conditions of real industrial CCTV, where workers appear as distant figures amid dust, steam, low light, glare, occlusion, and overlapping activities. We introduce STEELBENCH, a diagnostic benchmark for industrial surveillance that jointly evaluates per-worker activity recognition, safety-rule reasoning, and annotation provenance. SteelBench contains 1,345 densely annotated clips, curated from 149 hours of operational plant footage and 10,024 candidate clips using temporal deduplication, class balancing, and visibility-aware stratified sampling. Each clip includes dense per-worker action labels, PPE attributes, spatial context, and safety-rule annotations. Because model-assisted annotation can shape the labels later used for model evaluation, SteelBench includes a provenance-aware audit protocol. The protocol measures label influence, evaluates sensitivity to ground-truth provenance, and reports a human reference from expert-reviewed labels. Applying this audit, we find that unaudited VLM-sourced ground truth can inflate same-family model accuracy by up to 17 percentage points. Across nine VLMs from four architectural families, the best model reaches only 42.6% action accuracy, compared with an 84.6% human benchmark. Performance also fragments across recognition, robustness, calibration, and safety reasoning. Even when models predict the correct action, 37-58% of cases still yield incorrect safety judgments, and no model passes more than 2 of 5 diagnostic checks. SteelBench shows that real industrial activity understanding requires provenance-aware and failure-modespecific evaluation rather than leaderboard accuracy alone. The dataset is publicly available on Hugging Face. 1 Introduction Vision-language models (VLMs) are routinely evaluated on action recognition, visual question answering, and scene understanding, but predominantly on data where subjects are clearly visible, centrally framed, and recorded under controlled or simulated conditions. Video benchmarks such as Kinetics [1], ActivityNet [2], and AVA [3] capture human activity at close range and high resolution, while egocentric datasets such as Ego4D [4] and Assembly101 [5] focus on detailed hand-object interactions. Many existing industrial video benchmarks use simulated or controlled environments [6], while related safety datasets often evaluate narrower tasks, such as PPE detection ∗ Corresponding author. Email: [yarrabothula@iitbhilai.ac.in](yarrabothula@iitbhilai.ac.in) Preprint from curated images [7–9] . None of these datasets, to our knowledge, combines dense per-worker action annotation, PPE assessment, spatial context, and safety-rule compliance from real operational industrial CCTV. Workers appear as 80–200 pixel figures at 7–10 meters. Visibility conditions arise from natural plant operations, including occlusion from worker activities and equipment placement, low light, dust from material handling, steam from cooling, and glare from molten metal. Multiple workers often perform different activities simultaneously. These conditions also make annotation difficult. When visual evidence is ambiguous, annotators may defer to VLM-generated pre-fills, creating a dependency between model-assisted label construction and model evaluation that most benchmark

FAQs

What problem does SteelBench address in existing vision-language model evaluations?

SteelBench addresses the gap between benchmark conditions and real industrial CCTV, where workers are distant and scenes include dust, steam, low light, glare, occlusion, and simultaneous activities that make reliable annotation difficult.

What does each SteelBench clip include for annotation?

Each clip provides dense per-worker action labels, PPE attributes, spatial context, and safety-rule annotations, using a schema designed for multi-worker scenes.

How does SteelBench evaluate the reliability of model-assisted labels?

SteelBench adds a provenance-aware audit protocol that measures label influence and sensitivity to ground-truth provenance, including a human reference from expert-reviewed labels to detect evaluation inflation.

Jul 16, 2026

Upload by Aria