Multi-task Learning Is Not Enough: Representational Entanglement in Dual-output Second Language Speech Recognition

Jul 16, 2026

Upload by Patrick

Second-language (L2) speech recognition often requires systems to predict both what is pronounced and what is intended in canonical written form. Multi-task learning (MTL) is commonly used to share representations across outputs, yet controlled experiments on Korean and English show this assumption fails. MTL improves meaning transcription while degrading surface transcription, especially for English, with degradation increasing with surface-meaning divergence measured by Levenshtein edit distance. Encoder and decoder analyses attribute effects to encoder-level representational entanglement and constrained surface dual-output decoding, motivating entanglement-mitigating MTL designs for dual-output L2 ASR.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Abstract

Introduction and Related Work

Method

Problem Formulation

Architecture and Training

Experiments

Multi-task Learning is Not Enough: Representational Entanglement in Dual-output Second Language Speech Recognition Seung Hwan Cho 1 Young-Min Kim 1 2 arXiv :2606 .06065v4 [ cs .CL] 11 Jul 2026 Abstract Second-language (L2) speech recognition often requires transcriptions of pronunciations and intended meanings. Multi-task learning (MTL) is a natural approach because it assumes that shared representations benefit both outputs. However, this paper shows that this assumption does not hold across Korean and English. MTL improves meaning but degrades surface transcription, especially in English, where the degradation scales with surface-meaning divergence measured by Levenshtein edit distance. Encoder analysis links these patterns to encoder-level entanglement, with Korean preserving disentangled representations while English produces nearly identical ones. Cross-output decoder analysis shows that the meaning dual-output decoder adapts with a unique representation, while the surface dualoutput decoder remains constrained by the encoder. These findings motivate the design of MTL frameworks that mitigate encoder-level entanglement to reduce surface degradation in dual-output L2 automatic speech recognition. 1. Introduction and Related Work Human speech exhibits systematic differences between what is actually pronounced (surface-level) and the canonical written form (meaning-oriented) of an utterance. These differences reflect language-specific phonological phenomena, such as coarticulation, phonological reduction, and liaison (Ernestus & Warner, 2011) . The phonological gap is more pronounced in second-language (L2) speech, where speakerspecific deviations are common (Munro, 2021) . Therefore, speech recognition systems for L2 speakers must recover 1Department of Industrial Data Engineering, Hanyang University, Seoul, South Korea 2 School of Interdisciplinary Industrial Studies, Hanyang University, Seoul, South Korea. Correspondence to: Young-Min Kim <[yngmnkim@hanyang.ac.kr](yngmnkim@hanyang.ac.kr) >. Accepted at the 43rd International Conference on Machine Learning Workshop on Machine Learning for Audio, Seoul, South Korea. 2026. Copyright 2026 by the author(s) . both transcription forms from a single acoustic signal to enable targeted feedback in language learning and pronunciation assessment applications (Eskenazi, 2009) . Multi-task learning (MTL) is a natural approach for dualoutput (DO), encompassing auxiliary MTL where one task supports another and joint MTL where tasks are learned with equal importance (Ruder, 2017) . In automatic speech recognition (ASR), joint connectionist temporal classification (CTC)-attention training (Kim et al., 2017 ; Watanabe et al., 2017) and intermediate-layer CTC (Nozaki & Komatsu, 2021) use auxiliary CTC to improve alignment and training stability. Joint MTL approaches generate distinct target sequences from the same acoustic input. Examples include dual-decoder models for ASR and speech translation (Leet al., 2020) and unified diarization-separation-recognition systems (Shakeel et al., 2025) . However, the effectiveness of joint MTL for DO L2 ASR remains unexamined, despite the fact that the two outputs share linguistic content. This paper challenges the assumption that joint MTL benefits both outputs in DO ASR by conducting controlled experiments on Korean and English L2 speech. The results show that joint MTL produces asymmetric output tradeoffs that depend on language, with the source localized to encoder-level representational entanglement. Our contribution is twofold. First, we demonstrate the languagedependent behavior of joint MTL for DO L2 ASR. Second, we identify the underlying mechanisms through encoderand cross-output decoder analyses. These findings motivate the development of structured approaches to mitigate this entanglement. 2. Method To isolate the effect of joint training on shared representations, we compare single-output (SO) models, which are tr

FAQs

What underlying mechanism is proposed to explain the tradeoff between outputs?

The encoder-level analysis links the observed behavior to representational entanglement. Korean tends to preserve disentangled representations, while English produces nearly identical ones, constraining the surface dual-output decoder.

Jul 16, 2026

Upload by Patrick