Open Models Open Risks Measuring Unsafe Generation in Text-to-Image Models In the Wild

Jul 16, 2026

Existing safety research on text-to-image (T2I) jailbreaks mostly relies on controlled in-the-lab experiments using a limited set of canonical models, leaving the real safety state of the fast-growing in-the-wild T2I ecosystem unclear. This gap is driven by detector-oriented metrics that target controlled evaluation and by unsafe risks that may stem from both adversarial prompting and unsafe release practices or risky derivatives. The study conducts a large-scale empirical jailbreak analysis across 200+ Hugging Face models.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Introduction

In-the-Wild vs In-the-Lab Safety

Research Questions

Refined Evaluation Metrics

Large-Scale Evaluation Setup

Findings and High-Risk Identification

Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild Peilin Han [hanpeilin6788@gmail.com](hanpeilin6788@gmail.com)[ ](hanpeilin6788@gmail.com)Xidian University China Jingchun Zhang Xidian University China Yang Liu Xidian University China Teng Li Xidian University China Yilong Yang Xidian University China Jianfeng Ma Xidian University China Zhuo Ma Xidian University China arXiv :2607 .07827v 1 [ cs .CR] 8 Jul 2026 Abstract Existing safety studies on text-to-image (T2I) jailbreaks are largely conducted in controlled in-the-lab settings, typically on a small number of canonical models. As a result, the current safety status of the rapidly growing in-the-wild T2I ecosystem remains unclear. This uncertainty is amplified by two factors: existing detector-based metrics are designed for controlled evaluation, and in-the-wild risks may arise not only from adversarial prompting, but also from unsafe release practices and unsafe model derivatives. In this paper, we present a large-scale empirical study of in-thewild T2I safety through the lens of jailbreak. We first show that detector-only jailbreak metrics substantially overestimate practical risk over in the wild due to semantic drift and generation artifacts, and we introduce Advanced ASR to better capture semantically valid and visually plausible unsafe generation. Using this refined metric, we evaluate 200+ in-the-wild T2I models from Hugging Face under three representative jailbreak attacks. Our results show that many downstream models retain a non-trivial degree of safety even without explicit post-hoc safeguards, indicating that safety degradation in the wild is neither universal nor uniform. At the same time, we identify a set of high-risk models, including explicitly NSFW-oriented releases as well as seemingly benign models whose unsafe behavior is only exposed through systematic evaluation. We further trace these models to their release context and report high-risk cases to Hugging Face. Keywords Text-to-Image Model, Jailbreak, Not Safe For Work, In the Wild 1 Introduction With the rapid advancement of diffusion-based architectures, multimodal generative models, particularly Text-to-Image (T2I) systems, have been widely adopted in real-world applications[5, 22, 30] . These models are capable of generating high-quality and visually coherent content, leading to a rapidly expanding user base. However, models developed in controlled laboratory environments (i.e., in-the-lab) are typically optimized for general-purpose objectives. Such designs are often insufficient to accommodate diverse and evolving user requirements. In practice, communities exhibit a strong demand for customization capabilities, including support for specific artistic styles and domain-specific generation tasks[3, 25] . To address this limitation, open model ecosystems have emerged on platforms such as Hugging Face and ModelScope [7, 21] . These platforms lower the barrier to model access and modification, enabling users to fine-tune and redistribute customized T2I models. Models deployed in the wild are often released with weakened, optional, or entirely removed safety mechanisms. However, the relaxation or removal of safety mechanisms exposes new attack surfaces. Among them, jailbreak-based manipulation[4, 9, 16, 28, 29] has emerged as an effective strategy to bypass content safeguards. By crafting specific prompts or conditioning inputs, adversaries can induce T2I models to generate unsafe or Not Safe for Work (NSFW) content that violate predefined safety policies. Our work. In this paper, we systematically study the current safety status of in-the-wild T2I models through the following three research questions: (1) RQ1: In-the-wild Jailbreak Metrics. What is the difference between lab and wild? What limitations arise from these existing evaluation metrics? How to accurately evaluate effective real-world risk? (2) RQ2: S

FAQs

Why are current text-to-image jailbreak safety results unclear in real-world (in-the-wild) settings?

Most studies use controlled in-the-lab evaluations with a small set of canonical models, and detector-based metrics are designed for those controlled conditions. Real-world risks can also emerge from unsafe release practices and unsafe model derivatives, not only adversarial prompts.

What problem do detector-only jailbreak metrics have when measuring in-the-wild risk?

Detector-only metrics substantially overestimate practical risk because they can be unreliable in the wild due to semantic drift and generation artifacts. The study notes that detectors may flag images as unsafe even when they do not satisfy real-world NSFW objectives.

How does the paper evaluate in-the-wild safety and what does it conclude about model safety variation?

It introduces an advanced ASR-style refined metric (AASR) to better capture semantically valid and visually plausible unsafe generation, then evaluates 200+ in-the-wild T2I models with three representative jailbreak attacks. Results indicate safety degradation is neither universal nor uniform: many downstream models retain non-trivial safety, while high-risk models are identified and traced for reporting.

Jul 16, 2026

Upload by Clementine