PromptGuard - Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models - Abstract

Aug 29, 2026

Uploaded by Aurora

Text-to-image (T2I) models are vulnerable to misuse, especially for generating not-safe-for-work (NSFW) content, creating serious ethical and safety risks. The work introduces PromptGuard, a content moderation method inspired by system prompts in large language models, adapted to T2I where no direct behavioral interface exists. PromptGuard optimizes a universal safety soft prompt in the model’s textual embedding space to moderate NSFW inputs while preserving benign image realism. Experiments across three datasets show strong NSFW mitigation with minimal impact on safe output quality, including reported speed and defense improvements.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Motivation and misuse risks for T2I models

Existing defenses: model alignment vs content moderation

PromptGuard overview and key idea

Challenges and research questions

Enabling safety guidance without model modification

Achieving universal moderation across NSFW categories

Method direction (soft prompt design)

PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models arXiv :2501 .03544v 1 [ cs .CV] 7 Jan 2025 Lingzhi Yuan 1 , *, Xinfeng Li2 ,5 ,∗ , Chejian Xu3 , Guanhong Tao4 , Xiaojun Jia2 , Yihao Huang2 , Wei Dong2 , Yang Liu2 , XiaoFeng Wang5 , Bo Li 1 ,3 1 University of Chicago, 2 Nanyang Technological University, 3 University of Illinois at Urbana–Champaign, 4 The University of Utah, 5 Indiana University Bloomington [lingzhiyxp@gmail.com](lingzhiyxp@gmail.com) , [lxfmakeit@gmail.com](lxfmakeit@gmail.com) , [lbo@illinois.edu](lbo@illinois.edu) [https://prompt-guard.github.io](https://prompt-guard.github.io) Abstract Text-to-image (T2I) models have been shown to be vulnerable to misuse, particularly in generating not-safe-for-work (NSFW) content, raising serious ethical concerns. In this work, we present PromptGuard, a novel content moderation technique that draws inspiration from the system prompt mechanism in large language models (LLMs) for safety alignment. Unlike LLMs, T2I models lack a direct interface for enforcing behavioral guidelines. Our key idea is to optimize a safety soft prompt that functions as an implicit system prompt within the T2I model’s textual embedding space. This universal soft prompt (P∗) directly moderates NSFW inputs, enabling safe yet realistic image generation without altering the inference efficiency or requiring proxy models. Extensive experiments across three datasets demonstrate that PromptGuard effectively mitigates NSFW content generation while preserving highquality benign outputs. PromptGuard achieves 7.8 times faster than prior content moderation methods, surpassing eight state-of-the-art defenses with an optimal unsafe ratio down to 5.84%. Warnings: This paper contains NSFW imagery and discussions that some readers may find disturbing, distressing, and/or offensive. 1. Introduction Text-to-image (T2I) models, such as Stable Diffusion [26], have marked a transformative leap in generative AI, enabling highly realistic and creative images based solely on textual prompts. However, the misuse ofT2I models to generate not-safe-for-work (NSFW) content, such as sexual, violent, political, and disturbing images, has raised significant *co-first authors; Work done during Lingzhi’s internship at the University of Chicago. Figure 1 . Unlike existing moderation frameworks that rely on additional models to check or detoxify NSFW content, PromptGuard presents an efficient, universal soft prompt, P∗ , inspired by the system prompt mechanism in LLMs, to directly moderates NSFW inputs and generate safe yet realistic content. ethical concerns [12, 16, 41, 45] . The Internet Watch Foundation reports that thousands of AI-generated child sexual abuse images are shared on the dark web [29] . Besides, misleading political images and racially biased content have been frequently disseminated on social media, which may incite people’s emotions and even influence elections and social stability [32] . To prevent such misuse, there is an urgent demand for T2I service providers to adopt effective defense mechanisms. Current safeguards against NSFW content generation can be typically classified into two categories: model alignment and content moderation. Model alignment directly modifies the T2I model, aiming to remove learned NSFW textual concepts [10, 11] or visual representations [22, 34] from the model by fine-tuning or retraining its parameters [5, 18] . While effective, these methods might reduce the model’s capability to generate non-offensive, intended imagery [34, 50] . On the other hand, content moderation often uses proxy models that inspect unsafe textual inputs [20] or visual outputs [27] during generation or employs a prompt modifier [46] that utilizes aligned large lan- guage models (LLMs) to rephrase input prompts for safer image creation. Although these methods avoids the risk of unintended removal of benign concepts as in model alignment, their relia

FAQs

What problem does PromptGuard address for text-to-image (T2I) models?

It addresses the misuse of T2I models to generate NSFW content, aiming to reduce unsafe outputs while keeping benign generation effective.

How does PromptGuard work without modifying the T2I model architecture?

It optimizes a universal safety soft prompt that acts like an implicit system prompt within the T2I model’s textual embedding space, guiding both benign and NSFW prompts away from unsafe regions.

What evidence is provided that PromptGuard is effective?

The abstract states extensive experiments on three datasets showing PromptGuard mitigates NSFW generation while preserving high-quality benign outputs, along with reported efficiency and defense-strength improvements.

Aug 29, 2026

Uploaded by Aurora

PromptGuard - Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models - Abstract_Page 1

Open in WPS to view all 16 pages