Using the Mimi codec for metalinguistic representations - arXiv 2608.15799v1 - paper abstract and study

Sep 15, 2026

Uploaded by Finn

Codec-based audio language models are advancing, yet explainability for tokenised speech remains limited. This paper analyses the 2048-token dictionary in Mimi’s semantic token codebook, focusing on whether Mimi’s ABX experiment reflects the mapping from semantic tokens to phone realisations. The work realigns Mimi representations to TIMIT time-aligned transcriptions, showing token IDs correspond to quadphone, triphone, biphone, phone, and subphone realisations, and discusses implications for allophonic and subphonemic tokenisation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

Codec-based audio language models

Goal and mapping problem

Mimi and its tokenisation scheme

Experimental alignment with TIMIT

Using the Mimi codec for metalinguistic representations Artem Saloev 1 , Erin Pacquetet2 , Nicolas Ballier1 1ALTAE, Université Paris Cité, F-75013 Paris, France 2 SCIAM, 10 rue de Penthièvre, F-75008 Paris, France [artem.saloev@gmail.com](artem.saloev@gmail.com) , [erin.pacquetet@sciam.fr](erin.pacquetet@sciam.fr) , [nicolas.ballier@u-pariscite.fr](nicolas.ballier@u-pariscite.fr) arXiv :2608 . 15799v1 [ cs .CL] 16 Aug 2026 Abstract Codec-based audio language models are developing, but little explainability research has been dedicated to the representation of this typeof speech tokenisation. In this paper, we focus on the dictionary of 2048 tokens used in Mimi’s semantic token codebook, the neural codec of the Moshi language model (Défossez et al., 2024) . We show that the ABX experiment carried out with Mimi fails to capture the mapping of the semantic tokens to phone realisations. By realigning Mimi’s representations to the TIMIT corpus transcriptions (Garofolo et al., 1993), we show that the 2048 tokens IDs of the semantic codebook map to quadphone, triphone, biphone, phone and subphone realisations. We used the TIMIT transcriptions as evidence of the validity of the allophone-based representations of these 80ms semantic token representation and examine some of the theoretical consequences for the tokenisation of speech at allophone and subphonemic level. 1 Introduction Codec-based audio language models have recently emerged as a way to apply Transformer-style sequence modelling to speech by first turning the waveform into a stream of discrete codec tokens. A neural audio codec compresses the signal using multiple codebooks, or layers (typically via residual vector quantisation, RVQ), yielding a small number of tokens per frame; a Transformer is then trained to predict token sequences, and a decoder reconstructs the waveform from those predictions. Working with discrete audio tokens brings benefits: stable long-context modelling, controllable bit-rate and latency (via tokens per second and the number of codebooks used), and, in some designs, faster generation by predicting coarse levels first and refining finer details in parallel. This type of technology is increasingly used for conversational agents or speech-to-speech translations (Labiausse et al., 2025) but little is known about how voicesand speech features are encoded in these neural audio codecs. As conversational agents aim at realtime interactions and reducing latency, this has led to the design of codecs with limited frame rates (in our case study, 80 ms frames) . While this design is undoubtedly motivated by engineering considerations, the decision to discretise a naturally continuous acoustic signal raises a fundamental question: do the learned audio tokens partition the signal in ways that align with the categories posited by phonology (and, more broadly, phonetics)? Answering this question is central both to analysing these models’ internal representations and to assessing whether there is an alternative way to model the phon* level. Our goal in this paper is to test how far the two representations coincide. We focus on Mimi, a recent streaming neural codec that produces multilevel token sequences suitable for spoken-dialogue modelling. Using TIMIT (the standard testbed for the phonetics of American English and its computational modeling), with its time-aligned phonetic transcriptions, we ask whether Mimi’s codebook tokens can be mapped onto the manually identified phonemes. This seemingly simple mapping raises immediate practical difficulties: token frames and phoneme boundaries are not naturally aligned; tokens have a fixed frame rate, whereas phoneme durations vary with context; and some information relevant to segment identity may be distributed across levels. The audio language model Moshi generates and encodes speech as discrete audio tokens using a neural audio codec called “Mimi”1. The Mimi codec quantizes audio waveforms into the

FAQs

How do the authors connect Mimi’s 2048 semantic token IDs to phonetic units?

They realign Mimi’s semantic representations with the TIMIT corpus transcriptions, then examine how token IDs correspond to quadphone, triphone, biphone, phone, and subphone realisations.

Sep 15, 2026

Uploaded by Finn