ICME 2026 Academic Text-to-Music Generation Text-to-Audio Batch Sampling Submission

Jul 16, 2026

This work studies batch sampling strategies for text-to-audio music generation when training data are scarce and models are small, focusing on the ICME 2026 Grand Challenge submission. Training samples are clustered by text embeddings or audio embeddings, then grouped into mini-batches to reduce gradient interference caused by heterogeneous data. The study analyzes how modality choice and cluster granularity affect both objective evaluation and perceptual listening outcomes. Text-embedding clustering yields stronger objective metrics, while moderate cluster counts score best overall; more clusters improve coherent musical structure.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Abstract

Introduction

Clustering-based Batch Sampling for Text-to-Audio Music Generation

Overview

Baseline Text-to-Audio Model

UT-AISTimprt submission for ICME 2026 Grand Challenge on Academic Text-to-Music Generation Shunsuke Yoshida∗ , Yu-Hua Chen†, Satoru Fukayama† ∗ University of Tokyo, Japan †National Institute of Advanced Industrial Science and Technology (AIST), Japan arXiv :2607 .0 1669v 1 [ cs . SD] 2 Jul 2026 Abstract—This work investigates the effect of batch sampling strategies during training for text-to-audio music generation under low-data and small-scale model settings. This paper describes our approach and findings for the ICME 2026 Grand Challenge on Academic Text-to-Music Generation. Training data are clustered using either text embeddings or audio embeddings, and samples with similar characteristics are grouped within the same mini-batch to mitigate gradient interference. The effects of modality and cluster granularity on clustering are analyzed. Results show that clustering based on text embeddings achieves better performance on objective evaluation metrics than clustering based on audio embeddings. In addition, different cluster granularity leads to different behaviors across evaluation criteria: a moderate number of clusters performs best on objective metrics, while a larger number of clusters tends to exhibit music with more coherent structure in listening tests. Index Terms—music generation, text-to-audio, batch sampling, clustering I. INTRODUCTION Machine learning for music generation often suffers from a limited amount of publicly available training data compared to text or image domains. As a result, models for music generation are frequently trained on small-scale datasets, particularly in conditional and personalized generation settings. In practice, training small-scale models from scratch remains a realistic and commonly adopted setting. With a limited amount of training data, using small-scale models can help mitigate overfitting, but their representational capacity is limited. Consequently, the way training data are selected and presented during learning has a strong impact on generation performance. Restricting training data to a narrow genre or condition often reduces the diversity of output. On the other hand, naively mixing heterogeneous data can destabilize training of small-scale models due to gradient interference among diverse training data. The limited model capacity of the small-scale models have less flexibility to disentangle heterogeneous generative factors [1], [2] . Similar issues have been reported in the natural language processing (NLP) community, particularly in the context of multi-task learning and instruction tuning [1]–[3] . When multiple heterogeneous tasks or instructions are jointly learned, gradient conflicts can degrade both training efficiency and model performance. To address this problem, recent studies have proposed focusing on batch construction during training rather than on global data mixing ratios [3] . Commonality-aware Instruction Tuning (CommonIT) clusters training samples in advance and constructs each mini-batch from a single cluster, thereby increasing intra-batch homogeneity while preserving diversity across batches. Inspired by CommonIT, this work investigates clusteringbased batch sampling for text-to-audio music generation under low-resource and scratch-training conditions. The conditions are aligned with the ICME 2026 Grand Challenge on Academic Text-to-Music Generation [4] . Since text-toaudio models involve both text and audio modalities, the choice of modality used to define similarity becomes a key decision. In this submission, training samples are clustered using embedding representations derived from either a text encoder or an audio encoder, and mini-batches are constructed from samples within the same cluster. This work reports three main findings: • clustering-based batch sampling improves the performance of small-scale music generation models under lowdata settings, • text-based clustering achieves better performance in objective e

FAQs

What do the results say about choosing text vs. audio embeddings for clustering and about cluster granularity?

Clustering based on text embeddings performs better on objective evaluation metrics than clustering based on audio embeddings. A moderate number of clusters performs best on objective metrics, while a larger number of clusters tends to yield more coherent structure in listening tests.

Jul 16, 2026

Upload by Olivia Brown