LLMs can read music, but struggle to hear it - An evaluation of core music perception tasks

Aug 4, 2026

Multimodal Large Language Models (MLLMs) claim “musical understanding,” yet most evaluations conflate listening with score reading. This work benchmarks three state-of-the-art LLMs—Gemini 2.5 Pro, Gemini 2.5 Flash, and Qwen2.5-Omni—across Syncopation Scoring, Transposition Detection, and Chord Quality Identification. Variability is analyzed by contrasting audio versus MIDI, comparing zero-shot to few-shot exposure, and testing reasoning strategies including a LogicLM-style schema-and-solver approach. Results show near-ceiling performance on MIDI but substantial accuracy drops on audio, with reasoning and few-shot prompting offering limited gains.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

AccessFree to download

Introduction

Multimodal LLMs and “music understanding” claims

Limitations of existing audio and music benchmarks

Need for listening-based music structure evaluation

Abstract

Benchmark design and evaluated music skills

Sources of variability and reasoning strategies

LLMs can read music, but struggle to hear it. An evaluation of core music perception tasks Brandon James Carone [bcarone@nyu.edu](bcarone@nyu.edu) Department of Psychology, Music and Audio Research Laboratory (MARL), Center for Language, Music, and Emotion (CLaME), New York University Iran R. Roman [i.roman@qmul.ac.uk](i.roman@qmul.ac.uk) School of Electronic Engineering and Computer Science, Queen Mary University of London Pablo Ripoll´es [pripolles@nyu.edu](pripolles@nyu.edu) Department of Psychology, Music and Audio Research Laboratory (MARL), Center for Language, Music, and Emotion (CLaME), New York University Abstract Multimodal Large Language Models (MLLMs) claim “musical understanding,” yet most evaluations conflate listening with score reading. We benchmark three SOTA LLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, and Qwen2.5-Omni) across three core music skills: Syncopation Scoring (rhythm perception), Transposition Detection (melody perception), and Chord Quality Identification (harmony perception) . Moreover, we separate three sources of variability: (i) perceptual limitations (by contrasting audio recordings vs. symbolic MIDI inputs), (ii) exposure to prior examples (zero-vs. few-shot manipulations), and (iii) reasoning strategies (Standalone, Chain of Thought, LogicLM) . For the latter we adapt LogicLM, a framework combining LLMs with symbolic solvers to perform structured reasoning. In LogicLM, LLMs act as perceptual formulators, generating strict, machinecheckable schemas (onset grids, interval sequences) that deterministic solvers execute with self-refinement. Our results reveal a clear perceptual gap: models perform near ceiling on MIDI but show substantial accuracy drops on audio. Reasoning and few-shot prompting offer minimal gains. This is expected for MIDI, where performance reaches saturation, but more surprising for audio, where LogicLM, despite near-perfect MIDI accuracy, remains notably brittle. Among models, Gemini Pro achieves the highest performance across most conditions. Transposition yields the highest accuracies across models, while Chord Identification scores slightly below Syncopation. Overall, current systems reason well over symbols (MIDI) but do not yet “listen” reliably from audio, with reasoning strategies having little impact over accuracy. Our method and dataset make the perception–reasoning boundary explicit and offer actionable guidance for building robust, audio music systems. Keywords: Audio Large Language Models, Multimodal Large Language Models, Music Understanding, Benchmarking and Evaluation, Schema-Guided Reasoning, LogicLM 1. Introduction Recent advances in foundation models have extended their reach beyond text to multimodal architectures that process audio, vision, and language in a unified framework. Models such as Alibaba’s Qwen2.5-Omni, trained on a range of audio tasks (Xu et al., 2025), and Google’s Gemini 2.5 family, which incorporates advanced multimodal integration for real-time interactions (Comanici et al. , 2025), exemplify this new generation. One problem with many multimodal LLMs is that they boast “generic hearing abilities” and “music understanding”, yet struggle with tasks as simple as recognizing a well-known tune when transposed to a different key (i.e., singing the same song at a higher or lower pitch) or played on another © 2026 B.J. Carone, I.R. Roman & P. Ripoll´es. Carone Roman Ripoll´es instrument. For example, after being fed a simple piano rendition of “Happy Birthday” and told that the melody represents that tune regardless of which key it is played in, they fail to recognize it when played in a different key or on another instrument (in our own pilot tests, Gemini 2.5 guesses that the transposed version of “Happy Birthday” played at the same tempo is “Twinkle Twinkle Little Star” and Qwen2.5-Omni responded with “Lose Yourself” by Eminem) . On the other hand, most people with Western enculturation recognize “Happy Birthday” a

FAQs

Which music perception tasks are evaluated in the benchmark?

The benchmark covers Syncopation Scoring (rhythm perception), Transposition Detection (melody perception), and Chord Quality Identification (harmony perception).

How does the study separate perceptual limitations from other sources of variability?

It contrasts audio recordings with symbolic MIDI inputs, tests zero-shot versus few-shot exposure to prior examples, and evaluates reasoning strategies including a LogicLM-style method.

What is the key finding about model performance on MIDI versus audio?

Models perform near the ceiling on MIDI but show substantial accuracy drops on audio, and reasoning strategies produce limited improvements for accuracy.

Aug 4, 2026

Upload by Valentina