Deep learning’s shallow gains - a comparative evaluation of algorithms for automatic music generation

Aug 3, 2026

Deep learning methods are widely regarded as state of the art, yet their advantage for automatic music generation (AMG) using symbolic tokens in a target style has not been conclusively demonstrated. A listening study compares multiple music generation systems across six musical dimensions: stylistic success, aesthetic pleasure, repetition/self-reference, melody, harmony, and rhythm. Models include deep learning algorithms and non-deep methods. In controlled 30-s excerpts, Bayesian non-parametric hypothesis testing evaluates differences and finds deep learning does not clearly outperform classical baselines, while human-composed examples remain significantly ahead.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages39

Reading Time1 h 38 min

AccessFree to download

Introduction

Background and motivation

Research questions

Related work

Musical representations and method categories

Evaluation approach

[eprints@whiterose.ac.uk](eprints@whiterose.ac.uk)[ ](eprints@whiterose.ac.uk)[https://eprints.whiterose.ac.uk](https://eprints.whiterose.ac.uk) Universities of Leeds, Sheffield and York Deposited via The University of York. White Rose Research Online URL for this paper: [https://eprints.whiterose.ac.uk/id/eprint/233324/](https://eprints.whiterose.ac.uk/id/eprint/233324/) Version: Published Version Article: YIN, ZONGYU, REUBEN PARIS, FEDERICO, Stepney, Susan et al. (2023) Deep learning’s shallow gains: a comparative evaluation of algorithms for automatic music generation. Machine Learning. pp. 1785-1822. ISSN: 0885-6125 [https://doi.org/10.1007/s10994-023-06309-w](https://doi.org/10.1007/s10994-023-06309-w) Reuse This article is distributed under the terms of the Creative Commons Attribution (CC BY) licence. This licence allows you to distribute, remix, tweak, and build upon the work, even commercially, as long as you credit the authors for the original work. More information and the full terms of the licence here: [https://creativecommons.org/licenses/](https://creativecommons.org/licenses/) Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk) including the URL of the record and the reason for the withdrawal request. Deep learning’s shallow gains: a comparative evaluation of algorithms for automatic music generation Zongyu Yin1 · Federico Reuben2 · Susan Stepney1 · Tom Collins2,3 Received: 25 June 2021 / Revised: 20 October 2022 / Accepted: 27 January 2023 / Published online: 21 March 2023 © The Author(s) 2023 Abstract Deep learning methods are recognised as state-of-the-art for many applications of machine learning. Recently, deep learning methods have emerged as a solution to the task of automatic music generation (AMG) using symbolic tokens in a target style, but their superiority over non-deep learning methods has not been demonstrated. Here, we conduct a listening study to comparatively evaluate several music generation systems along six musical dimensions: stylistic success, aesthetic pleasure, repetition or self-reference, melody, harmony, and rhythm. A range of models, both deep learning algorithms and other methods, are used to generate 30-s excerpts in the style of Classical string quartets and classical piano improvisations. Fifty participants with relatively high musical knowledge rate unlabelled samples of computer-generated and human-composed excerpts for the six musical dimensions. We use non-parametric Bayesian hypothesis testing to interpret the results, allowing the possibility of finding meaningful non-differences between systems’ performance. We find that the strongest deep learning method, a reimplemented version of Music Transformer, has equivalent performance to a non-deep learning method, MAIA Markov, demonstrating that to date, deep learning does not outperform other methods for AMG. We also find there still remains a significant gap between any algorithmic method and humancomposed excerpts. Keywords Deep learning · Non-parametric Bayesian hypothesis testing · Markov model · Music generation · Comparative evaluation · Listening study Editor: Tijl De Bie. * Zongyu Yin [zongyu.yin@outlook.com](zongyu.yin@outlook.com) Extended author information available on the last page of the article 1 Introduction In the past decade, breakthroughs in artificial intelligence (AI) and deep learning have been established as such through rigorous, comparative evaluations,1 for example, in computer vision (O’Mahony et al., 2019) and automatic speech recognition (Toshniwal et al., 2018) . In the field of automatic music generation (AMG), however, to our knowledge there has been no comparative evaluation to date between deep learning and other methods (Huang et al., 2018 ; Yang et al., 2017 ; Dong et al., 2018 ; Hadjeres et al., 2017 ; Thickstun et al., 2019 ;

FAQs

What does the paper compare in automatic music generation?

It conducts a comparative listening study across multiple AMG systems, including both deep learning and non-deep learning approaches.

Which musical dimensions are used to evaluate the generated excerpts?

The study evaluates stylistic success, aesthetic pleasure, repetition or self-reference, melody, harmony, and rhythm.

What is the main finding about deep learning versus non-deep methods?

The strongest deep learning method (a reimplemented Music Transformer) shows performance equivalent to the non-deep MAIA Markov, indicating deep learning does not yet outperform other methods for AMG.

Aug 3, 2026

Upload by Evangeline