EvolveDirector - Approaching Advanced Text-to-Image Generation with Large Vision-Language Models

Aug 27, 2026

Uploaded by Fahsai

Recent generation models can produce high-quality images, yet many rely on proprietary datasets and keep parameters private, restricting reproducibility and downstream benefits. EvolveDirector explores whether an open text-to-image model can be trained with public resources by using public APIs of advanced models to create text-image pairs. Experiments show generated data helps match advanced capabilities, but large-scale samples (10M+) are costly. The framework uses pre-trained large vision-language models to iteratively discriminate, expand, delete, and mutate training data, reducing required volume. It also selects best samples across multiple advanced models and the final Edgen model outperforms them.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

AccessFree to download

Introduction

Motivation and Problem Setting

EvolveDirector Framework

Vision-Language Guided Data Evolution

Experimental Findings and Model Comparison

EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models Rui Zhao 1 , Hangjie Yuan2 , Yujie Wei2 ,3 , Shiwei Zhang2 , Yuchao Gu 1 , Lingmin Ran 1 , Xiang Wang2 ,4 , Zhangjie Wu 1 , Junhao Zhang 1 , Yingya Zhang2 , Mike Zheng Shou 1 , ∗ 1 Show Lab, National University of Singapore 2Alibaba Group 3 Fudan University 4 Huazhong University of Science and Technology Abstract Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million or more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates andrefines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The code and model weights are available at [https://github.com/showlab/EvolveDirector](https://github.com/showlab/EvolveDirector). 1 Introduction In the field of AI-generated content, an increasing number of advanced models have showcased their ability to generate realistic and imaginative images, such as Imagen [1], DALL·E 3 [2], Stable Diffusion 3 [3], Midjourney [4] . While these models benefit from publicly available datasets such as ImageNet [5], LAION [6], and SAM [7], they rely more heavily on proprietary, high-quality data collections that surpass the quality of publicly accessible datasets. Models such as Midjourney [4] are particularly noted for deriving substantial benefits from their internal datasets. However, given the significant commercial advantages brought by their impressive capabilities, most advanced models keep their parameters private, hindering reproducibility and democratization. In this paper, we aim to explore training an open-source text-to-image model with public resources to achieve comparable capabilities with the existing advanced models. ∗ Corresponding Author. 38th Conference on Neural Information Processing Systems (NeurIPS 2024) . Four superheroes stand in front of the pyramids, from left to right, Batman, Hulk, Iron Man, and Spider-Man. Three transparent glass bottles on a wooden table. The one on the left has red liquid and the number 1. The one in the middle has blue liquid and the number 2. The one on the right has green liquid and the number 3. In a garden, there sits a white cat on the left and a black dog on the right, with a green water cup placed between the two of them. A captivating book cover features a young woman with an enchanting gaze, dressed in vibrant attire, surrounded by swirling colors, inviting readers into a world of chaos and beauty. A sleek and modern logo design for NeurIPS 2024. The text "NeurIPS 2024" is written in bold, black capital

FAQs

What is EvolveDirector designed to achieve?

EvolveDirector aims to train an open-source text-to-image model using publicly available resources while approaching the capabilities of advanced models. It builds training pairs through public APIs and improves the training process with vision-language guidance.

How does EvolveDirector collect training data from advanced models?

It interacts with advanced models via their publicly accessible APIs to obtain text-image data pairs. These pairs are then used to train a base model, including scenarios with generated data.

Why does the paper use large vision-language models during training?

Large vision-language models continuously evaluate and guide the evolution of the base model. They update and refine the dataset through discrimination, expansion, deletion, and mutation operations, significantly reducing the data volume required.

Aug 27, 2026

Uploaded by Fahsai