The Song Describer Dataset - A Corpus of Audio Captions for Music-and-Language Evaluation

Aug 3, 2026

The Song Describer dataset (SDD) introduces a crowdsourced collection of high-quality audio–caption pairs for evaluating music-and-language models. It provides 1.1k human-written natural language descriptions spanning 706 music recordings, using openly accessible recordings released under Creative Commons licenses. The work benchmarks popular models across music captioning, text-to-music generation, and music-language retrieval, emphasizing cross-dataset evaluation and practical guidance for researchers seeking a broader, more reliable view of model performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Introduction

Evaluation challenges in music-and-language

Song Describer dataset overview

The Song Describer Dataset

Data source and licensing

Caption characteristics and metadata linkage

Key differences vs. MusicCaps and YT8M-MusicTextClips

The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation Ilaria Manco∗1 ,2 , Benno Weck∗3, Seungheon Doh4 , Minz Won5 , Yixiao Zhang 1 , Dmitry Bodganov3 , Yusong Wu6 , Ke Chen7 , Philip Tovstogan3 Emmanouil Benetos 1 , Elio Quinton2 , George Fazekas 1 , Juhan Nam4 1 QMUL, 2UMG, 3UPF, 4 KAIST, 5ByteDance, 6Mila, 7UCSD Abstract We introduce the Song Describer dataset (SDD), a new crowdsourced corpus of high-quality audio-caption pairs, designed for the evaluation of music-andlanguage models. The dataset consists of 1.1k human-written natural language descriptions of 706 music recordings, all publicly accessible and released under Creative Common licenses. To showcase the use of our dataset, we benchmark popular models on three key music-and-language tasks (music captioning, text-tomusic generation and music-language retrieval) . Our experiments highlight the importance of cross-dataset evaluation and offer insights into how researchers can use SDD to gain a broader understanding of model performance. 1 Introduction Multimodal approaches that jointly process audio and language are becoming increasingly important within music understanding and generation, giving rise to a new area of research, which we refer to as music-and-language (M&L) . Several recent works have emerged in this domain, proposing methods to automatically generate music descriptions [21, 7, 9], synthesise music from a text prompt [2, 14, 30, 6], search for music based on language queries [8, 22, 13], and more [20, 17] . However, evaluating M&L models remains a challenge due to a lack of public and accessible datasets with paired audio and language, resulting in the widespread use of private data [21, 22, 23, 14, 2, 13] and inconsistent evaluation practices. To mitigate this, we release the Song Describer dataset (SDD), anew high-quality evaluation dataset of crowdsourced captions paired with openly licensed music recordings. Through our dataset, we allow for the first standardised comparison of models on realistic music-language data and expand the number and variety of datasets available to the research community. We propose SDD as an evaluation-only dataset to promote the development and use of out-of-domain data and counteract the tendency to overfit to a particular dataset [5, 29, 32] . Alongside SDD, there are currently two other human-curated datasets of music-caption pairs: MusicCaps [2] and YT8M-MusicTextClips [25], composed of 5.5k and 4k 10-second clips from AudioSet [11] and YouTube8M [1] respectively. More recently, Doh et al. [7] have released LP-MusicCaps, a dataset of 2.2M music descriptions generated by prompting a large language model with a set of instructions and ground-truth tags. While synthetic data presents an opportunity to artificially scale up training data, limitations such as LLM hallucinations still pose significant challenges to their use in evaluation, where reliable data plays a crucial role. ∗Equal contribution Machine Learning for Audio Workshop (NeurIPS 2023) . Table 1: An overview of the SDD compared to other music-caption datasets, MusicCaps (MC) [2] and YT8MMusicTextClips (MTC) [25] . * denotes the validated subset. | Dataset Annotators | | Text | | | Audio | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | Caption \# | Avg length | Vocab size | Audio \# | Length | Source | Public | | MC | 10 | 5,521 | 54.8 ± 18.7 | 6,144 | 5,521 | 10 sec | [11] | ✗ | | MTC | - | 4,169 | 16.9 ± 4.4 | 2,599 | 4,169 | 10 sec | [1] | ✗ | | SDD* | 114 | 746 | 18.2 ± 7.6 | 1,942 | 547 | 2 min | [4] | ✓ | | SDD | 142 | 1106 | 21.7 ± 12.4 | 2,859 | 706 | 2 min | [4] | ✓ | Table 2: The main task instruction given to the annotators, together with an example of three captions produced by different annotators describing the same track in the dataset. | Instructions | Example caption | | --- | --- | | Write one sentence that describes the track you just listened to: foc

FAQs

What is the Song Describer Dataset (SDD)?

SDD is a crowdsourced dataset of high-quality audio-caption pairs designed to evaluate music-and-language models.

What does SDD contain in terms of scale and captions?

SDD includes 1.1k human-written natural language descriptions for 706 music recordings, with captions crafted as rich single-sentence descriptions covering musical features.

How does SDD support evaluation compared with other music-caption datasets?

SDD offers standardized, openly licensed audio and longer track segments, enabling more reliable cross-dataset model comparisons and evaluation with automatic captioning metrics.

Aug 3, 2026

Upload by Olivia Brown