TMI Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation

Jul 16, 2026

Large-vocabulary instance segmentation struggles with long-tailed category distributions and fine-grained inter-class ambiguity. Although generative data synthesis can mitigate costly annotations, existing text-to-image pipelines introduce noisy pseudo-labels and underperform on rare classes, while copy-paste image editing can degrade contextual realism. A hybrid T2I-to-I2I framework is presented: text-to-image supplies scene diversity and a teacher-student scheme filters prompt-specified categories for reliable labels. Context-aware editing with VRAIN inserts verified rare-class instances at semantically appropriate locations, producing coherent, natural edits. Results on LVIS show up to +4.0 overall AP and +9.5 rare-class AP improvements, scaling with backbone capacity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

Abstract

Introduction

Problem: long-tailed instance segmentation and data scarcity

Limitations of existing synthetic-data paradigms

Proposed hybrid framework

arXiv :2607 .0820 1v 1 [ cs .CV] 9 Jul 2026 TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation Hyeonseop Song∗ , Seokhun Choi∗ , and Hoseok Do† AI Lab, CTO Division, LG Electronics, Republic of Korea {[hyeonseop.song](hyeonseop.song) , seokhun.choi, [hoseok.do}@lge.com](hoseok.do}@lge.com)[ ](hoseok.do}@lge.com)Project page: [https://seokhunchoi.github.io/TMI](https://seokhunchoi.github.io/TMI) Abstract. Large-vocabulary instance segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. While data synthesis offers a promising alternative, current paradigmshave complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle on rare classes, whereas copy-paste methods compromise contextual realism. To address these issues, we propose a hybrid pipeline coupling T2I generation with context-aware imageto-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories. To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation. On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity. Keywords: Generative Data Synthesis · Image Diffusion Models · Hybrid Data Pipeline · Long-Tailed Instance Segmentation 1 Introduction Instance segmentation is a critical task, underpinning diverse applications from autonomous driving [4, 5] and robotics [16, 46 , 52] to visual understanding [8, 26 , 43] and image editing [21, 25 , 40] . However, achieving robust performance relies on large, densely annotated datasets, which are costly to build. This bottleneck is particularly acute for large-vocabulary benchmarks such as LVIS [9] with their long-tailed category distributions. While methods like re-weighting [33, 34 , 54], balanced sampling [3, 11 , 54], and classifier calibration [29,37 , 38] help alleviate the imbalance, they cannot resolve the underlying data scarcity of rare categories [9] . * Equal contribution. †Corresponding author. 2 H. Song & S. Choi et al. | | T2I-Generation | | | I2I-Editing | | | | --- | --- | --- | --- | --- | --- | --- | | Baseline | (a) | | Static Label | (c) (d) | | “Adda red and yellow striped bait to thefloor…” | | Ours | Adaptive Refine (b) | | | | | | | Data Synthesis Approach | | Label Accuracy | Realism | Scene Diversity | | --- | --- | --- | --- | --- | | T2I | (a) Mosaic | | | | | | (b) Our T2I | | | | | I2I | (c) Copy-Paste | | | | | | (d) Our Edit | | | | | T2I+I2I | (e) Our Hybrid | | | | Real-only MosaicFusion DiverGen X-Paste Ours APall | 34.5 | | | | | | --- | --- | --- | --- | --- | | 34.1 | | | | | | 35.1 | | | | | | 36.7 | | | | | | 38.1 | | | | | APrare | 24.0 | | | | | --- | --- | --- | --- | | 24.4 | | | | | 25.6 | | | | | 29.6 | | | | | 33.9 | | | | Fig. 1: Mutually Complementary T2I-I2I Data Synthesis. (a) Existing T2I methods (e.g ., MosaicFusion [47]) offer diverse scenes but lack label accuracy, while (c) Copy-Paste-based I2I methods (e.g ., X-Paste [53] and DiverGen [6]) offer accurate labels but reduce realism. (e) Our hybrid framework combines two proposed modules:(b) a T2I branch for scene diversity with reliable labels via teacher-student adaptive pseudo-labeling, and (d) context-aware I2I edits for realistic, accurate instance-level supervision.

FAQs

What problem does the paper address in long-tailed instance segmentation?

It addresses the performance limits caused by long-tailed category distributions and fine-grained ambiguity, compounded by the scarcity of densely annotated data for rare categories.

How does the proposed method combine text-to-image and image-to-image?

It uses a T2I branch to generate diverse broad categories and scenes, then employs a teacher-student scheme to retain reliable prompt-specified categories. A context-aware I2I editor then performs realistic edits to insert rare-class instances.

What is VRAIN and how does it improve rare-class augmentation?

VRAIN is a verified rare-class augmentation method that inserts high-confidence rare instances into semantically appropriate locations within in-the-wild scenes. This yields semantically coherent, visually natural edits that reduce domain gaps for targeted augmentation.

Jul 16, 2026

Upload by Theodora