Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition

Jul 16, 2026

Language model (LM) perplexity (PPL) has long been used as a proxy for automatic speech recognition (ASR) word error rate (WER), often described as an approximately linear relationship in log-log space. Modern end-to-end ASR systems challenge this link because they include internal language modeling, are frequently evaluated without external LMs, and can incorporate neural LMs or LLMs via multiple decoding strategies. This study re-examines PPL–WER linearity, tests external-LM usefulness, analyzes encoder context effects, and evaluates how internal language modeling subtraction alters the observed relation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract

Introduction & Related Work

Classical PPL–WER correlation

External LM integration strategies

Why end-to-end ASR weakens the proxy assumption

Motivation for systematic re-examination

Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition Mohammad Zeineldeen∗†, Albert Zeyer∗†, Haoran Zhang†, Robin Schmitt∗†, Ralf Schl¨uter∗†, Hermann Ney∗† ∗ AppTek. ai GmbH, Aachen, Germany † Machine Learning and Human Language Technology Group, Faculty of Computer Science, RWTH Aachen University Aachen, Germany {zeineldeen, zeyer, schmitt, schlueter, [ney](ney}@ml.rwth-aachen.de)[}](ney}@ml.rwth-aachen.de)[@ml.rwth-aachen.de](ney}@ml.rwth-aachen.de) arXiv :2607 .056 12v 1 [ cs .CL] 6 Jul 2026 Abstract—Language model (LM) perplexity (PPL) has historically been used as a proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. Modern end-toend ASR systems challenge this assumption because they already contain internal language modeling capacity, are often evaluated without external language models, and can now be combined with neural LMs and large language models (LLMs) through different recognition strategies. This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in loglog space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. We further investigate internal language modeling (ILM) in attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder’s internal LM must be considered when interpreting the effect of external LM quality. Index Terms—speech recognition, language models, perplexity, shallow fusion, large language models I. INTRODUCTION & RELATED WORK Language models (LMs) have long been central to automatic speech recognition (ASR) . Perplexity (PPL) is commonly used as an intrinsic LM metric, tracing back to its introduction as a measure of speech task difficulty [1] . However, ASR performance is measured by word error rate (WER), which also depends on the acoustic model, decoding strategy, pruning, and scale tuning. Thus, lower LM perplexity does not automatically translate into lower WER. Earlier studies, mostly using n-gram LMs [2], reported experimental evidence for a correlation between perplexity and WER across several ASR tasks [3]–[5] . The work in [4] systematically studied this question and found that WER and PPL are approximately linearly related in log-log space. The same log-log linear relation was later observed in [6] for hybrid ASR models on Quaero and LibriSpeech dev-clean using both n-gram and long shortterm memory (LSTM) LMs [7] . This observation has been influential because it suggests that relative PPL improvements can be translated into expected WER improvements through a power-law relation. Large-scale ASR studies also support stronger external LMs. The study in [8] showed that increasing LM scale and using large text corpora yields consistent WER reductions. Later work extended LM integration to neural LMs and endto-end ASR. Shallow fusion combines an external LM with the ASR model during beam search [9], while cold fusion incorporates a pretrained LM during training [10] . Several integration strategies were compared in [11] for attentionbased encoder-decoder (AED) ASR and found shallow fusion to be a strong and simple first-pass decoding method across multiple conditions. Neural Transformer LMs [12] have also been applied successfully to hybrid and end-to-end ASR, including lattice rescoring and shallow fusion [13] . Modern end-to-end ASR systems make the PPL-WER relation less direct. In Connectionist temporal classification (CTC) [14], recurrent neural network transducer (RNN-T) [15], and AED [16] systems, acoustic and language modeling components are no longer cleanly separated as in classical hybrid ASR. AED and tra

FAQs

Why might LM perplexity no longer predict ASR word error rate in modern end-to-end systems?

End-to-end ASR models learn internal language modeling from speech-text training data, so external LM effects can interact with the decoder’s internal LM rather than mapping directly to perplexity.

What does the paper investigate about the PPL–WER relationship?

It tests whether external LMs still improve modern end-to-end ASR, whether the PPL–WER relation remains approximately linear in log-log space, and how encoder context length influences the relation.

How does internal language model subtraction affect the observed PPL–WER trend?

The paper shows that ILM subtraction changes the PPL–WER relation, indicating that the decoder’s internal LM must be considered when interpreting how external LM quality impacts WER.

Jul 16, 2026

Upload by Theodore