The NeurIPS 2023 Machine Learning for Audio Workshop - Affective Audio Benchmarks and Novel Data

Aug 4, 2026

Upload by Violet

The NeurIPS 2023 Machine Learning for Audio Workshop convenes machine learning experts across audio-relevant domains, addressing the community’s limited accessibility to large, high-quality datasets for time-dependent modalities. It highlights tasks spanning speech emotion recognition, audio event detection, and broader audio generation and modeling needs. To support researchers facing data constraints, the organizers release multiple open-source datasets and provide proprietary datasets during the workshop. The paper describes four large-scale resources—HUME-PROSODY, HUME-VOCALBURST, MODULATE-SONATA, and MODULATE-STREAM—summarizes current baselines, and encourages evaluation beyond the initial benchmark tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Introduction

Workshop scope and motivation

Provided datasets and benchmark usage

Workshop Audio Datasets

Dataset design for speech emotion and human responses

HUME-PROSODY

HUME-VOCALBURST

MODULATE-SONATA

MODULATE-STREAM

arXiv :2403 . 14048v1 [ cs . SD] 21 Mar 2024 The NeurIPS 2023 Machine Learning for Audio Workshop: Affective Audio Benchmarksand Novel Data Alice Baird* Hume AI New York, USA [alice@hume.ai](alice@hume.ai) Rachel Manzelli* Modulate AI Massachusetts, USA [rachel@modulate.ai](rachel@modulate.ai) Panagiotis Tzirakis Hume AI New York, USA Chris Gagne Hume AI New York, USA Haoqi Li Hume AI New York, USA Brian Kulis Boston University Boston, USA Sadie Allen Boston University Boston, USA Shrikanth S. Narayanan USC-SAIL California, USA Sander Dieleman DeepMind London, UK Alan Cowen Hume AI New York, USA Abstract The NeurIPS 2023 Machine Learning for Audio Workshop brings together machine learning (ML) experts from various audio domains. There are several valuable audio-driven ML tasks, from speech emotion recognition to audio event detection, but the community is sparse compared to other ML areas, e.g., computer vision or natural language processing. A major limitation with audio is the available data; with audio being a time-dependent modality, high-quality data collection is time-consuming and costly, making it challenging for academic groups to apply their often state-of-the-art strategies to a larger, more generalizable dataset. In this short white paper, to encourage researchers with limited access to largedatasets, the organizers 􀀂rst outline several open-source datasets that are available to the community, and for the duration of the workshop are making several propriety datasets available. Namely, three vocal datasets, HUME-PROSODY, HUMEVOCALBURST, an acted emotional speech dataset MODULATE-SONATA, and an in-game streamer dataset MODULATE-STREAM. We outline the current baselineson these datasets but encourage researchers from across audio to utilize them outside of the initial baseline tasks. 1 Introduction Working with audio data in machine learning presents unique challenges compared to 􀀂elds like computer vision. Despite the importance of various key problems in the audio domain, such as textto-speech, voice recognition, source separation, and synthesis, it has received considerably lower attention. However, there has been a recent renaissance in audio research, particularly in the 􀀂eld of synthesis, with the release of several in􀀃uential papers in the last year [1–6] . The relative scarcity of prior research and this recent boom serves as the primary motivations behind organizing the 2023 NeurIPS Machine Learning for Audio (MLA) Workshop. The workshop covers a vast scope of audio-related tasks, including but not limited to speech modeling, speech generation, music generation, denoising of speech and music, data augmentation, acoustic event classi􀀂cation, transcription, source separation, and even multimodal modelling involving audio. *These authors contributed equally to this work. There are prestigious competitions like the Detection and Classi􀀂cation of Acoustic Scenes and Events (DCASE) focusing on audio-driven machine learning [7], and numerous large-scale audio datasets with high-level labeling available in the literature, including Audioset [8], Urban80k [9], and Librispeech [10], along with unique concepts such as bird identi􀀂cation [11], and music genre detection [12] . Despite the availability of such datasets, there still exists a scarcity of openly accessible large-scale datasets particularly tailored for more specialized domains, such as human-computer interaction and human behavior analysis. The lack of specialized datasets presents a considerable hurdle for developing systems that understand more nuanced human characteristics. To address this issue and to support authors submitting their work to MLA, the organizers have proactively provided four datasets to researchers participating in the workshop, namely, the HUME-PROSODY, HUME-VOCALBURST, MODULATE-STREAM and MODULATE-SONATA. These datasets offer a broad spectrum of human states, including both labeled and unlabeled d

FAQs

Why does the workshop emphasize audio datasets and benchmarks?

Audio tasks are constrained by time-dependent, expensive data collection, leaving the community with fewer openly accessible large-scale datasets. The workshop aims to bridge this gap with new resources and established baseline tasks.

Which four datasets are provided for workshop research?

The paper presents HUME-PROSODY, HUME-VOCALBURST, MODULATE-SONATA, and MODULATE-STREAM. Together they cover human states using different interaction settings and include both labeled and unlabeled data where applicable.

What do the datasets enable researchers to study?

They support evaluation and improvement of unsupervised models, particularly for speech emotion recognition and modeling human responses to dynamic events. The datasets also provide labeled benchmarks focused on prosody and vocal bursts when applicable.

Aug 4, 2026

Upload by Violet