Devil in the Lens: Analyzing and Defending Physical Prompt Injection Against Vision-Language Models on Wearable Devices

Jul 16, 2026

Upload by Ezra

Vision-Language Models (VLMs) are being deployed on wearable devices such as smart glasses to support multimodal perception and AI-assisted decision-making. Existing research on visual prompt injection into digital images does not fully address the security risks created when physical environments interact with wearable intelligence. This work models Physical Prompt Injection where malicious scene text uses a high-priority visual channel to hijack VLM behavior, disrupting tasks and steering outputs toward profane, biased, or untruthful results. Experiments across 200+ environments evaluate 6 threat vectors on 12 VLMs, with attack success up to 96% (simulated) and 60% (real), and propose mask-based filtering and semantic-vector internal detection defenses.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Motivation and safety concerns

Prompt injection background and related work

Threat overview and problem statement

Devil in the Lens: Analyzing and Defending Physical Prompt Injection Against Vision-Language Models on Wearable Devices Yaxin Li [yli849@connect.hkust-gz.edu.cn](yli849@connect.hkust-gz.edu.cn)[ ](yli849@connect.hkust-gz.edu.cn)Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China Hao Wang Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China Yanda Shao Beijing University of Posts and Telecommunications Beijing, China Shuhao Zhang Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China Yan Long∗ [yanlong@hkust-gz.edu.cn](yanlong@hkust-gz.edu.cn)[ ](yanlong@hkust-gz.edu.cn)Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China arXiv :2607 . 10269v 1 [ cs .CR] 11 Jul 2026 Abstract Vision-Language Models (VLMs) are rapidly deployed on humanfacing wearable devices such as smart glasses to enable multimodal perception and AI-assisted decision-making. While prior research has demonstrated the risks of visual prompt injection into digital image inputs of VLMs, the unique security challenges posed by the increasing integration between physical environments and wearable intelligence, such as those embodied in VLM-enabled AI glasses, remain underexplored. Toward understanding and modeling such threats, our work characterizes how malicious textual information embedded in physical environments introduces a highpriority visual channel for indirect prompt injection, where scene texts that hinder or evade human perception could hijack VLM models’ behavior. Such Physical Prompt Injection Attacks can not only disrupt normal tasks of VLM-enabled wearable devices, but also steer models to produce profane, biased, or even untruthful outputs. Using physically captured photos from AI glasses in over 200 real-world environments, our analysis identifies 6 representative threat vectors of physically injected prompts, and further evaluates their impacts on 12 VLM models. Results show that these attacks consistently manipulate model outputs across integrity-and safety-critical tasks, achieving attack success rates of up to 96% and 60% in simulated and real-world settings. Our analysis confirms that multiple models exhibit excessive blind trust in environmental text, ignoring the actual visual context and producing completely opposite summaries or directives. We further propose two targeted defense strategies, including a mask-based external filter and a semantic-vector-based internal detector, to effectively reduce the success rate and safety impact of these attacks. ∗ Corresponding author Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission [and/or a fee. Request permissions from permissions@acm.org](and/or a fee. Request permissions from permissions@acm.org). Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s) . Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn) CCS Concepts • Security and privacy → Usability in security and privacy. Keywords large vision-language model, multi-modal, visual attack, prompt injection attack ACM Reference Format: Yaxin Li, Hao Wang, Yanda Shao, Shuhao Zhang, and Yan Long. 2026. Devil in the Lens: Analyzing and Defending Physical Prompt Injection Against Vision-Language Models on Wearable Devices. In . ACM, New York, NY, USA, 10 pages. [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nn

FAQs

What is the main threat studied in the paper?

The paper studies Physical Prompt Injection attacks where malicious textual information embedded in physical environments hijacks Vision-Language Models via scene text.

How do these attacks affect wearable VLM behavior?

They can disrupt normal tasks and steer models to produce unsafe, profane, biased, or even untruthful outputs, including guidance that contradicts the actual visual context.

What defenses does the paper propose to reduce attack success and safety impact?

It proposes a mask-based external filter and a semantic-vector-based internal detector to lower the success rate and mitigate harmful effects of physical prompt injection.

Jul 16, 2026

Upload by Ezra