BlueMagpie TTS：Token高效分词、语言模型与台语口音代码切换语音合成

Jul 16, 2026

Upload by Riley

Off-the-shelf text-to-speech systems struggle with Taiwanese Mandarin because their accent suits other Mandarin variants, their tokenizers over-segment common Taiwan-specific text, and pronunciation degrades at code-switching boundaries between Chinese and English. This work adapts the text side bottom up with PangolinTokenizer, a byte-level BPE tokenizer trained on Taiwan-context data. It achieves the lowest token rate (0.485 tokens/character) and smallest vocabulary among compared tokenizers. Barbet, a billion-parameter Traditional-Chinese LM trained on PangolinTokenizer, leads on a 14-task evaluation and enables BlueMagpie-TTS. A learned bridge connects Barbet to a reused VoxCPM2 acoustic stack, and joint fine-tuning yields lower CER and preferred listening votes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Introduction

Motivation: accent and text-side mismatch for Taiwanese Mandarin

Token fragmentation and its consequences

Code-switching challenges at boundaries

Bottom-up adaptation strategy

Contributions overview

PangolinTokenizer and PangolinBench

Barbet language model

BlueMagpie-TTS system

BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech Ho-Lam Chung, Bo-Xuan Zheng†, Cheng-Chieh Huang†, Cheng-Han Chang†, Jung-Ching Chen†, Lok-Lam Ieong†, Ting-Lin Hsiao†, Yu-Cheng Lee†, Yi-Hsin Chung†, Yu-Kai Guo†, Hung-yi Lee National Taiwan University arXiv :2607 .06054v 1 [ cs . SD] 7 Jul 2026 Abstract—Off-the-shelf TTS systems are poorly adapted to Taiwanese Mandarin. Their accent defaults to other Mandarin variants, their tokenizers over-segment common Taiwanese text, and their pronunciation degrades at code-switching boundaries where Chinese and English alternate within one utterance. These problems share one root: the text side lacks adaptation to the Taiwanese context. We address the text side from the bottom up. PangolinTokenizer, a byte-level BPE tokenizer trained on Taiwancontext data, reaches the lowest token rate (0.485 tokens/character) with the smallest vocabulary among eight tokenizers. Barbet, a billion-parameter Traditional-Chinese language model trained on PangolinTokenizer, serves as the text-semantic frontend and ranks first among the compared billion-parameter models on a 14-task evaluation. BlueMagpie-TTS attaches Barbet to thepretrained acoustic stack of VoxCPM2 through a learned bridge, then jointly fine-tunes the assembled model on target-voice speech. On a 1,000-sentence Taiwan-localized test set, it lowers CER from 11.45% to 4.81% . It stays 25.2% below a baseline that fine-tunes the same acoustic stack to the target voice, so the gain comes from the frontend, not from voice adaptation. In a blind listening study on 500 of these sentences with ten listeners, 65.6% of majority votes prefer BlueMagpie-TTS. Index Terms—text-to-speech, tokenization, code-switching, Taiwanese Mandarin, speech language model I. INTRODUCTION Neural TTS now produces near-human speech for major languages on standard reading benchmarks [1]–[4] . Most of these systems, however, are poorly adapted to Taiwanese Mandarin. Their accent defaults to other regional Mandarin variants, and Taiwanese listeners perceive the mismatch immediately. Their text side is not trained on the terms and writing conventions common in Taiwan, so the model has limited capacity for local usage. These two gaps, the accent and the text representation, reduce the naturalness of off-the-shelf TTS for Taiwanese Mandarin speakers. The text side is where the problem starts. Most TTS systems build on a general multilingual tokenizer [5] . Such a tokenizer represents Taiwan-context text losslessly, but it often fragments common terms into many tokens [6] . A term such as 半導體 costs one token under a Taiwan-context tokenizer, but several under a general multilingual one. Figure 2 shows the effect †Equal contribution. on one sentence. The fragmentation wastes sequence length, raises inference cost, and means the language model trains on a longer, noisier representation of the same content. Code-switching makes it worse. A single spoken sentence in Taiwan often mixes Mandarin with English words, abbreviations, and proper nouns. A speaker says 我用 Transformer 訓練模型, one Mandarin sentence with an English word in the middle. At the switch point, the tokenizer fragments the boundary, the language model cannot plan the prosody transition, and the TTS distorts the English span or breaks the surrounding Mandarin prosody. Chinese–English mixing is common in education, technology, and media, so these failures are not rare. Fixing these problems requires adapting the text side from the bottom up. The first step is a tokenizer trained on Taiwancontext text, so that common terms stay as single tokens and the sequence is short. The second step is a language model trained on this tokenizer, so that the model understands Taiwanese usage and can plan prosody across code-switching boundaries. The third step is connecting this frontend to a TTS acoustic stack, so that the accent and the switch points

FAQs

为什么现成的 TTS 系统不适配台湾口音的台语/普通话？

文中指出现成系统的口音与台湾普通话不匹配、文本端分词对台湾文本过度切分，并且在中文与英文交替的代码切换边界处发音会退化。根因在于文本侧缺少对台湾语境的适配。

PangolinTokenizer 如何改善台湾语境文本的分词效率？

PangolinTokenizer 使用基于字节的 BPE，并在台湾语境数据上训练，使常见台湾相关术语更可能作为单个 token，从而显著降低 token 数率，并在对比中达到最低 token rate（0.485 tokens/character）与最小词表。

BlueMagpie-TTS 的整体构建思路是什么？

系统将 Barbet 作为文本语义前端，通过学习到的桥接模块接入复用的 VoxCPM2 声学堆栈，并对拼接后的模型在目标说话人语音上联合微调。实验表明改进主要来自前端（分词与语言模型）而非仅做声学侧的语音适配。

Jul 16, 2026

Upload by Riley