Devil is in the Detail - Towards Injecting Fine Details of Image Prompt in Image Generation

Aug 27, 2026

Large-scale text-to-image diffusion models can generate diverse, high-quality images from prompts, yet detailed user intent often fails to transfer—especially textures and other fine-grained attributes. Image prompts address this by conditioning generation on user-provided images, typically by modifying self-attention through key/value integration. This paper identifies two recurring flaws that prevent faithful image-prompt reflection, then introduces a conflict-free guidance strategy and a stratiﬁed attention mechanism to balance realism with prompt fidelity, validated across multiple tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Abstract

Introduction

Motivation: image prompts for fine details

Self-attention modification for conditioning

Classifier-free guidance conflicts

Method overview

Conflict-free guidance

Stratiﬁed attention

This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conﬂict-free Guidance and Stratiﬁed Attention Kyungmin Jo Jooyeol Yun Jaegul Choo Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Korea {bttkm, blizzard072, [jchoo](jchoo}@kaist.ac.kr)[}](jchoo}@kaist.ac.kr)[@kaist.ac.kr](jchoo}@kaist.ac.kr) Figure 1 . Our method faithfully reﬂects the details of image prompts (e.g., eye makeup) for diverse image generation across various tasks (Tasks A-C) in a training-free manner. In contrast, existing methods often fail to incorporate ﬁne details of the image prompt. Even with training, IP-Adapter [19] and SSR-Encoder [21] struggle to properly represent the color and texture of the image prompt. RIVAL [22] and StyleAligned [4] also fail to generate high-ﬁdelity images that faithfully reﬂect the image or text prompts due to a trade-off between the realism of the generated results and alignment with the image prompt. Extra results are in the Supplementary (Suppl.) . Abstract While large-scale text-to-image diffusion models enable the generation of high-quality, diverse images from text prompts, these prompts struggle to capture intricate details, such as textures, preventing the user intent from being reﬂected. This limitation has led to efforts to generate images conditioned on user-provided images, referred to as image prompts. Recent work modiﬁes the self-attention mechanism to impose image conditions in generated images by replacing or concatenating the keys and values from the image prompt. This enables the self-attention layer to work like a cross-attention layer, generally used to incorporate text prompts. In this paper, we identify two common issues in existing methods of modifying self-attention that hinder diffusion models from reﬂecting the image prompt. By addressing these issues, we propose a novel method that generates images that properly reﬂect the details of image prompts. First, existing approaches often neglect the importance of image prompts in classiﬁer-free guidance, which directs the model towards the intended conditions and away from those undesirable. Speciﬁcally, current methods use image prompts as both desired and undesired conditions, causing conﬂicting signals. To resolve this, we propose conﬂict-free guidance by using image prompts only as desired conditions, ensuring that the generated image faithfully reﬂectsthe image prompt. In addition, we observe that the two most common self-attention modiﬁcations involve a tradeoff between the realism of the generated image and alignment with the image prompt, achieved by selectively using keys and values from both images. Speciﬁcally, selecting more keys and values from the image prompt improves alignment, while selecting more from the generated image enhances realism. To balance both, we propose an alternative self-attention modiﬁcation method, Stratiﬁed Attention, which jointly uses keys and values from both images rather than selecting between them. Through extensive experiments across three distinct image generation tasks, we demonstrate that the proposed method outperforms existing image-prompting models in faithfully reﬂecting the image prompt. 1. Introduction The emergence of large-scale text-to-image diffusion models [11, 13] has enabled users to generate diverse images from text. However, describing complex scenes using only text prompts can be complicated, leading to ongoing efforts to incorporate user-provided image conditions to better reﬂect user intent [14, 17, 19, 21] . As part of these efforts, modifying the self-attention mechanism in pre-trained textto-image diffusion models has recently attracted interest in areas such as image editing

FAQs

为什么仅靠文本提示难以捕捉细节？

文本提示难以描述纹理等细粒度属性，导致用户意图无法完整反映到生成结果中。图像提示能提供这些精细信息。

现有自注意力修改方法存在哪些主要问题？

文中指出两类常见问题会阻碍扩散模型对图像提示的反映，尤其是在与分类器无关引导相关的机制上产生冲突信号。

本文提出的关键方法是什么？

通过冲突无关引导（只将图像提示用于期望条件）消除负向冲突，并提出分层注意力（Stratiﬁed Attention）联合使用来自两路的 keys/values，以在真实感与提示一致性之间取得平衡。

Aug 27, 2026

Uploaded by Felix Montgomery