Stage-Aware Adaptation and Distribution Calibration for Subject-Driven Personalized Text-to-Image Generation

Jul 16, 2026

Upload by Eden

Subject-driven personalized text-to-image generation adapts a pretrained diffusion model using only a few reference images to learn a specific subject identity, follow novel text prompts, and keep sample diversity. Existing low-rank personalization and adapter methods fail to separate capacity needs across diffusion denoising stages, and inference-time identity-only candidate selection can shrink visual diversity. The framework combines SPaRa stage-aware low-rank training with DCAL distribution-calibrated candidate selection. On SDXL and DreamBooth 30-subject protocols, DCAL improves multiple identity- and text-related metrics while exposing diversity tradeoffs, guiding evaluation beyond identity similarity alone.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Stage-Aware Adaptation and Distribution Calibration for Subject-Driven Personalized Text-to-Image Generation Wenyan Xu 1 ,∗ Alizer Wong2 ,3 ,∗ 1 School of Computer Science, Guangdong University of Technology 2 School of Computer Science, Peking University 3ManXis [3223004777@mail2.gdut.edu.com](3223004777@mail2.gdut.edu.com) [aliiiiezer@gmail.com](aliiiiezer@gmail.com) ∗Equal contribution arXiv :2607 .07 173v 1 [ cs .CV] 8 Jul 2026 Abstract Subject-driven personalized text-to-image generation requires a pretrained diffusion model to acquire a specific subject from a few reference images while preserving subject identity, following novel text prompts, and maintaining sample diversity. Existing optimization-based methods instantiate subject adaptation through full fine-tuning, textual embedding optimization, or low-rank parameter updates; PaRa further constrains personalization from the perspective of parameter rank reduction. However, a uniform low-rank constraint or a uniform adapter strength cannot explicitly distinguish the capacity requirements of different denoising stages. Moreover, inference-time candidate selection driven mainly by identity similarity may compress the selected samples in the visual representation space. We decompose the problem into two complementary components: SPaRa denotes training-side stage-aware low-rank adaptation, DCAL denotes inference-side distribution-calibrated candidate selection, and SPaRa–DCAL denotes the combined framework. Theoretical analysis shows that timestep-dependent scaling controls the effective perturbation magnitude of a low-rank adapter, while identity-biased candidate selection restricts the radius of selected features around the reference center under explicit conditions. Auditable experiments under the SDXL and DreamBooth 30-subject protocol show that DCAL improves 1-LPIPS, CLIP-I, DINO-I, and CLIP-T on a fixed LoRA candidate pool, while revealing a clear tradeoff with CLIP/DINO pairwise diversity and pairwise LPIPS. These results indicate that personalized generation should be evaluated through identity consistency, text alignment, and representation diversity rather than identity metrics alone. 1 Introduction Text-to-image diffusion models have evolved from generalpurpose image synthesis systems into controllable genera- Figure 1 . Motivation. Stage-agnostic adaptation and identity-only selection may both limit personalization quality. tive infrastructure for real content production. Latent Diffusion Models and SDXL substantially improve visual fidelity, semantic compositionality, and cross-style transfer under natural-language conditions, which enables generative systems to support e-commerce material creation, intellectualproperty assets, personal digital avatars, advertising design, and interactive visual authoring [1, 2] . Practical generation targets, however, are rarely abstract categories alone. Many applications require a model to reproduce a concrete subject with stable identity, such as a particular pet, backpack, toy, commodity, or character. Subject-driven personalized text-to-image generation therefore asks a model to capture the fine-grained appearance that distinguishes one instance from other category members using only a few reference images, and then reuse those subject-specific attributes under new prompts describing unseen scenes, poses, materials, and styles [3, 4] . This ability determines whether diffusion models can move beyond open-domain synthesis toward user-level customization and asset-level reuse. As illustrated in Fig. 1, few-shot subject personalization is difficult because reference images provide the desired identity and incidental factors at the same time. Backgrounds, camera viewpoints, lighting conditions, occlusions, and local compositions are entangled with subject appearance in the small reference set. Excessive adaptation can absorb these incidental correlations, causing generated images to remain near th

FAQs

What problem does stage-aware adaptation target in subject-driven personalized text-to-image generation?

It addresses the mismatch between personalization capacity requirements across different diffusion denoising stages, which uniform low-rank constraints or adapter strengths cannot explicitly handle.

How does the proposed framework combine SPaRa and DCAL?

SPaRa performs training-side stage-aware low-rank adaptation, while DCAL performs inference-side distribution-calibrated candidate selection; SPaRa–DCAL is evaluated as the combined framework.

What tradeoffs does DCAL introduce according to the experiments?

DCAL improves identity- and text-alignment related metrics on a fixed LoRA candidate pool, while revealing a clear tradeoff with diversity measures such as CLIP/DINO pairwise diversity and pairwise LPIPS.

Jul 16, 2026

Upload by Eden