[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85294-en":3,"doc-seo-85294-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85294,687197207919,"Theodora","https://ap-avatar.wpscdn.com/avatar/a000253d6f5f7c60be?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779446848396160552",8,"Research & Report","Unified Gradient Projection Language-Balanced Continual Learning for Multilingual Low-Resource ASR","Large-scale pretrained ASR models such as Whisper show strong multilingual performance, but fine-tuning for low-resource languages often triggers catastrophic forgetting. Continual learning helps, yet existing methods inadequately control cross-task interference when dominant languages bias optimization. Unified Gradient Projection (UGP) constrains parameter updates using reference gradients from language-balanced replay projected into a unified space, equalizing per-language contributions to reduce bias. Combining gradient-level projection with data-level replay improves both stability and plasticity, enabling effective adaptation with near-zero average forgetting on Whisper-large-v3.","Unified Gradient Projection: Language-Balanced Continual Learning for  \nMultilingual Low-Resource ASR  \nZiang Ren ID ∗, Guodong Lin ID ∗, Yuchen Ai , Kaize Tan , Wei-Qiang Zhang  ∗∗  \nDepartment of Electronic Engineering, Tsinghua University, China  \n[ziangren65@gmail.com](ziangren65@gmail.com) , [wqzhang@tsinghua.edu.cn](wqzhang@tsinghua.edu.cn)  \narXiv :2607 . 1 1 163v 1 [ cs .CL] 13 Jul 2026  \nAbstract  \nLarge-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from languagebalanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.  \nIndex Terms: Continual Learning, Multilingual ASR, LowResource Speech, Gradient Projection  \n1. Introduction  \nThe field of Automatic Speech Recognition (ASR) is undergoing a paradigm shift driven by large-scale foundation models[1, 2] . OpenAI’s Whisper [3], for example, leverages massive weakly supervised data and a unified sequence-to-sequence architecture to achieve remarkable cross-lingual generalization within a single model. Such scaling has significantly narrowed performance gaps across many languages. Nevertheless, performance on numerous low-resource languages remains suboptimal [4, 5], posing a fundamental challenge to the vision of truly universal speech recognition.  \nAdapting foundation models to under-represented languages through fine-tuning can substantially improve recognition accuracy [6] . However, sequential multilingual adaptation inevitably introduces the plasticity–stability dilemma [7]: improving plasticity for newly encountered languages often compromises stability on previously learned ones. As parameters shift to accommodate new linguistic patterns, representations essential to prior languages may be overwritten, leading to catastrophic forgetting [8] . This tension becomes particularly pronounced in large multilingual ASR models [9], where shared representations must support heterogeneous phonetic and linguistic structures.  \nThis work was supported by the National Natural Science Foundation of China under Grant No. 62276153.  \n*These authors contributed equally.  \n**indicates the corresponding author.  \nContinual learning (CL) [10] offers various strategies for the plasticity–stability trade-off, yet existing approaches remain imperfect in large-scale multilingual ASR [11, 12] . Parameterisolation methods such as LoRA [13] retain plasticity but do not explicitly constrain cross-task interference and may reduce parameter sharing across languages [14]; regularization-based methods like EWC [15, 16] promote stability but rely on approximate importance estimation and incur substantial overhead at model scale. Replay-based paradigms, particularly Experience Replay (ER) [17], mitigate forgetting through data rehearsal, yet lack explicit gradient control and may struggle under severe gradient conflicts [12] .  \nAmong these paradigms, gradient-based continual learning regulates interference by constraining update directions. Methods such as GEM [18] and its scalable variant A-GEM [19] preserve prior knowledge by projecting gradients using reference information from previous tasks, with A-GEM approximating multiple constrain","cbCaipfZtPG95QYV","https://ap.wps.com/l/cbCaipfZtPG95QYV","pdf",354654,6,1,5,"English","en",105,"# Introduction\n## Problem: Plasticity–stability and cross-lingual bias\n## Related work: Parameter isolation, regularization, and replay\n## Gradient-based continual learning and its limitations\n## Proposed approach: Unified Gradient Projection (UGP)","[{\"question\":\"Why does fine-tuning low-resource languages cause catastrophic forgetting in multilingual ASR?\",\"answer\":\"Sequential multilingual adaptation creates a plasticity–stability dilemma: learning new language patterns can overwrite representations needed for previously learned languages, especially when gradients from tasks conflict and are biased by dominant languages.\"},{\"question\":\"What is Unified Gradient Projection (UGP) in this work?\",\"answer\":\"UGP constrains parameter updates using language-balanced reference gradients projected into a unified projection space. This reshapes optimization geometry to reduce antagonistic cross-lingual gradient interactions while integrating Experience Replay.\"},{\"question\":\"How does UGP combine gradient projection with data-level replay, and what benefits does it bring?\",\"answer\":\"UGP uses projection to regulate destructive update directions at the optimization level, while Experience Replay mitigates harmful drift on earlier languages. Together, the method supports joint multilingual training that improves stability and plasticity, achieving near-zero average forgetting on Whisper-large-v3.\"}]",1784202305,13,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"unified-gradient-projection-language-balanced-continual-learning-for-multilingual-low-resource-asr","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/unified-gradient-projection-language-balanced-continual-learning-for-multilingual-low-resource-asr/85294/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"Why does fine-tuning low-resource languages cause catastrophic forgetting in multilingual ASR?","Question",{"text":76,"@type":77},"Sequential multilingual adaptation creates a plasticity–stability dilemma: learning new language patterns can overwrite representations needed for previously learned languages, especially when gradients from tasks conflict and are biased by dominant languages.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What is Unified Gradient Projection (UGP) in this work?",{"text":81,"@type":77},"UGP constrains parameter updates using language-balanced reference gradients projected into a unified projection space. This reshapes optimization geometry to reduce antagonistic cross-lingual gradient interactions while integrating Experience Replay.",{"name":83,"@type":74,"acceptedAnswer":84},"How does UGP combine gradient projection with data-level replay, and what benefits does it bring?",{"text":85,"@type":77},"UGP uses projection to regulate destructive update directions at the optimization level, while Experience Replay mitigates harmful drift on earlier languages. Together, the method supports joint multilingual training that improves stability and plasticity, achieving near-zero average forgetting on Whisper-large-v3.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,110,114,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":22,"slug":137},19,"General","general"]