COALA: Robust Contextualized Speech-Augmented Language Modeling for ASR via Contrastive Regularizer and Biasing Score Estimation

Jul 16, 2026

Contextual biasing integrates external knowledge into automatic speech recognition (ASR) to improve domain-specific entity recognition. This paper proposes COALA (COntextualized ASR Leveraging biAsing scoring), a robust framework for enhancing speech-augmented language models (SLMs) in challenging multi-entity settings. Because SLMs have context-window limits, COALA identifies relevant entities from large biasing lists by mapping SLM latent representations into a discriminative space that scores audio–entity match strength. Experiments on LibriSpeech show consistent gains across biasing-list scales using MPD-Loss and DPD-Loss.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

Methods

Biasing Scoring

Training Objectives

Experiments

COALA: Robust Contextualized Speech-augmented Language Modeling for ASR via Contrastive Regularizer and Biasing Score Estimation Jhih-Rong Guo, Bi-Cheng Yan, Tien-Hong Lo, Berlin Chen National Taiwan Normal University, Taiwan {jhihrong, 80847001s, teinhonglo, [berlin](berlin}@ntnu.edu.tw)[}](berlin}@ntnu.edu.tw)[@ntnu.edu.tw](berlin}@ntnu.edu.tw) arXiv :2607 .08 1 17v 1 [ cs .CL] 9 Jul 2026 Abstract Contextual biasing seeks to integrate external knowledge into automatic speech recognition (ASR) systems to accurately recognize domain-specific entities. In this paper, we propose COALA1 (COntextualized ASR Leveraging biAsing scoring), a robust framework designed to enhance speech-augmented language models (SLMs) in complex multi-entity scenarios. Considering the inherent context-window limitations of SLMs, identifying relevant target entities from a large-scale biasing list is crucial for effective recognition. To this end, COALA maps SLM latent representations into a specialized discriminative space to quantify the matching intensity between audio segments and candidate entities. Furthermore, we address the training collapse in prior study when handling multi-target utterances—where multiple rare words co-occur. Experimental results on the LibriSpeech benchmark demonstrate that COALA consistently achieves superior contextual biasing performance across various biasing list scales. Index Terms: speech recognition, contextual biasing, biasing scoring, contrastive learning 1. Introduction Driven by the monolithic nature and streamlined training process, end-to-end (E2E) automatic speech recognition (ASR) systems [1, 2, 3, 4] have gained widespread attention across both academia and industry. More recently, as witnessed by the remarkable success of large language models (LLMs) in the natural language processing community, efforts in ASR have pivoted toward leveraging LLMs for speech recognition and audio reasoning tasks. Through the synergy of a speech encoder anda language model, speech-augmented language models (SLMs) serve as a promising paradigm for ASR [5, 6, 7, 8, 9] . However, SLMs still struggle with uncommon, domainspecific entities such as contact names, proper nouns, and other named entities. Accordingly, contextual biasing techniques have been widely investigated to incorporate external knowledge into ASR systems, steering ASR hypotheses toward the accurate recognition of target entities and exerting a profound impact on modern speech applications.. Existing literature on contextual biasing can be broadly categorized into two active strands of research, namely, inference-time and trainingtime biasing, based on the stage of external knowledge integration. Inference-time biasing approaches, such as shallow fusion [10, 11] and on-the-fly rescoring [12, 13], typically construct ngram finite state transducers (FSTs) from a curated knowledge dataset, which are then utilized to selectively bias the search 1The source code and experimental setup for this study are available at [https://github.com/Guo0911/COALA](https://github.com/Guo0911/COALA). Figure 1: Distribution of utterances by number of target entities on the LibriSpeech corpus. space and dynamically boost the emission probability of target entities during the decoding stage, often triggered by specific prefixes (e.g., ’call’ or ’play’) . On a separate front, trainingtime biasing methods, such as attention-based biasing adapters [14, 15] and trie-based pointer generators [16, 17], which are trained to align acoustic features with contextual information via location-aware attention or neural shortcuts, effectively internalizing the recognition ability to prioritize target entities from custom biasing lists in an E2E manner. Despite the progress in inference-time and training-time biasing techniques, seamlessly integrating these methods into the evolving architectures of SLMs remains a formidable challenge. As the scale of the biasing list expand

FAQs

What problem does COALA address in speech-augmented language models (SLMs)?

COALA targets the difficulty SLMs face with uncommon, domain-specific named entities in multi-entity scenarios, especially under limited context windows and interference from many distractor entities.

How does COALA choose relevant entities from a large biasing list?

COALA maps SLM latent representations into a specialized discriminative space to quantify the matching intensity between audio segments and candidate entities.

What training objectives does COALA introduce to handle multi-target utterances?

COALA introduces MPD-Loss (Multi-Positive Discriminative Loss) and DPD-Loss (Decoupled Point-wise Discriminative Loss) to resolve gradient conflicts and support stable training when multiple rare words co-occur.

Jul 16, 2026

Upload by Genevieve