MuScriptor: An Open Model for Multi-Instrument Music Transcription

Jul 16, 2026

MuScriptor addresses limits of automatic music transcription models that struggle with realistic, multi-instrument recordings. The work examines how synthetic data can support pre-training but remains insufficient for general-purpose accuracy due to domain shift. It combines pre-training with fine-tuning on real audio and post-training via reinforcement learning using a curated high-quality subset. It also introduces conditioning on instrument presence, and releases MuScriptor as an open-weight model supporting diverse music genres.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract

Introduction

Related Work

MUSCRIPTOR: AN OPEN MODEL FOR MULTI-INSTRUMENT MUSIC TRANSCRIPTION Simon Rouard 1 ,3† Michael Krause2† Axel Roebel3 Carl-Johann Simon-Gabriel2 Alexandre Défossez 1 1 Kyutai 2 Mirelo AI 3 UMR STMS, IRCAM-CNRS Sorbonne Univ. [simon@kyutai.org](simon@kyutai.org) , [michael@mirelo.ai](michael@mirelo.ai) arXiv :2607 .08 168v 1 [ cs . SD] 9 Jul 2026 ABSTRACT Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres. 1. INTRODUCTION The task of Automatic Music Transcription (AMT) consists of converting an audio recording of a piece of music into some kind of symbolic representation, typically MIDI. While significant progress has been made in transcribing single-instrument recordings (specifically for piano [1], guitar [2], and drums [3, 4]), general-purpose transcription for multiple instruments remains a significant challenge. Transcribing multi-instrument music from diverse musical genres requires models to handle a vast range of timbres, overlapping frequencies, and audio effects (such as distortion on electric guitars), across a wide sonic spectrum. A primary bottleneck in building multi-instrument AMT systems is the scarcity of music audio with aligned note annotations. Recent works such as MT3 [5] attempt to solve this by combining small real-world datasets with large-scale synthetic data. For example, MT3 utilizes approximately 1500 hours of synthetic data combined with only 250 hours of (mostly single-instrument) real-world recordings. Although these models perform well on synthetic test sets, their performance often degrades significantly when applied to real-world audio. This suggests a critical domain shift between synthesized MIDI and the complexities of professional music productions. As a consequence, existing models for multi-instrument music transcription are usually too error-prone to be used for downstream applications. However, a general-purpose music transcription model could be an essential tool for musicians and musicologists, as well as enable new work on generative modeling and various music information retrieval tasks like chord or key recognition. In this paper, we investigate the effectiveness of training on synthetic data for music transcription and show that it can be very useful for pre-training, but not sufficient to enable general-purpose transcription. We further collect a large dataset of 170k real music recordings from a wide variety of genres (from classical to heavy metal), featuring audio and aligned note annotations. Through this, weare able to train an effective multi-instrument transcription model, which we release to the public. Figure 1 gives a qualitative impression of the improvements we achieve compared to state-of-the-art AMT models. We forego complex architectural tweaks and instead opt for a simple yet effective decoder-only transformer architecture. We compare two primary training regimes: training exclusively on real data and pre-training on synthetic data followed by real data fine-tuning. Furthermore, we post-train our models with reinforcement learning on a curated subset of 300 high quality transcribed pieces to improve results. Finally, we enable conditioning on instrument presence, allowing users to customize transcription. Overall, we make the f

FAQs

What problem does MuScriptor target in automatic music transcription?

It targets the difficulty of transcribing multi-instrument music in real recordings, where existing models often rely on single-instrument data or produce error-prone outputs in realistic mixes.

How does the paper use synthetic data versus real data?

Synthetic data is used for pre-training, but the paper shows it is not sufficient alone. The model is improved by fine-tuning on real music audio with aligned note annotations.

What post-training method improves alignment and transcription quality?

The model is post-trained with reinforcement learning on a curated subset of 300 high-quality transcriptions to improve alignment.

Jul 16, 2026

Upload by Theodora