[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86500-en":3,"doc-seo-86500-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86500,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",6,"Technology","AutoNorm Understanding Adaptive Normalization in Transformers through Differentiable Gating","Normalization is essential for stable Transformer training, but choosing between static Layer Normalization (LN) and adaptive alternatives often remains task-dependent. This work studies differentiable normalization gating and shows that, for relatively stationary vision tasks, Gumbel–Softmax gating produces high gradient variance that slows convergence and makes learned gates worse than random selection. For non-stationary language modeling and classification, persistent gating diversity enables stronger layer-wise normalization policies. To address optimization instability, the paper proposes AutoNorm-S (Stabilized) with a gatefreezing schedule, improving NLP benchmarks and remaining competitive on vision.","AutoNorm: Understanding Adaptive Normalization  \nin Transformers through Differentiable Gating  \nPiyush Kaushik Bhattacharyya  \n[piyushbhattacharyya@gmail.com](piyushbhattacharyya@gmail.com)[ ](piyushbhattacharyya@gmail.com)Swastik Singh [singhswastik1103@gmail.com](singhswastik1103@gmail.com)[ ](singhswastik1103@gmail.com)Ayush Ranjan [ranjanayush881@gmail.com](ranjanayush881@gmail.com)  \nDivyanshu Rai [raidivyanshu3232@gmail.com](raidivyanshu3232@gmail.com)  \nKumar Aakash[akashkumar221ank@gmail.com](akashkumar221ank@gmail.com)  \nKrutika Verma[krutika.vermafcs@kiit.ac.in](krutika.vermafcs@kiit.ac.in)  \narXiv :2607 . 10593v 1 [ cs .LG] 12 Jul 2026  \nAbstract—Normalization is a critical component for stabilizing Transformer training, yet the choice between static strategies such as Layer Normalization (LN) and adaptive alternatives remains largely task-dependent. In this paper, we investigate a key optimization challenge in differentiable normalization gating. Our experiments show that, on relatively stationary vision tasks, the high gradient variance introduced by Gumbel–Softmax gating can hinder convergence of the routing mechanism, causing learned gates to underperform simple random selection. In contrast, on non-stationary language modeling and classification tasks, sustained gating diversity enables the model to learn more effective layer-wise normalization policies. Motivated by these observations, we propose AutoNorm-S (Stabilized), a training strategy that mitigates optimization instability through a gatefreezing schedule. AutoNorm-S achieves competitive or improved performance across multiple benchmarks, outperforming adaptive normalization baselines on NLP datasets, including PTBand SST-2, while remaining competitive on standard vision benchmarks. These results suggest that decoupling normalization selection from optimization noise provides a practical and principled approach for adaptive normalization in Transformer architectures.  \nIndex Terms—Dynamic normalization, Transformers, GumbelSoftmax, differentiable gating, training dynamics.  \nI. INTRODUCTION  \nNormalization techniques have become fundamental to the stability and convergence of modern deep learning architectures. By normalizing feature statistics within each layer, techniques such as Batch Normalization (BN) [1] and Layer Normalization (LN) [2] mitigate internal covariate shift and stabilize gradient flow. In the era of Transformers [3], Layer Normalization has emerged as the de facto standard for stable self-attention training across large model depths.  \nHowever, conventional normalization layers are fundamentally static, applying identical transformations regardless of task context or input distribution. This one-sizefits-all approach is increasingly suboptimal as models encounter severe distribution shifts and diverse data modalities. Prior works exploring adaptive normalization—such as FiLM [31], Switchable Normalization [34], and conditional Meta-Normalization—typically rely on external metadata conditioning or expensive architectural searches, lacking a self-  \nsufficient mechanism for on-the-fly adaptation. Recently, Zhu et al. [30] demonstrated that replacing normalization entirely with a learnable tanh-based transformation (Dynamic Tanh; DyT) can match or exceed LN in standard Transformers, raising a natural question: can a model learn, at training time, which normalization strategy is optimal per layer and per task?  \nWhen we initially attempted to answer this using differentiable Gumbel-Softmax gating—a framework we call AutoNorm—we encountered a surprising and counter-intuitive phenomenon: while the learned mechanism worked effectively for linguistic sequences, it consistently failed on vision benchmarks, where a simple Random Selector often outperformed the learned policy. This unexpected failure highlighted a deeper issue in how architectural selection interacts with optimization dynamics.  \nIn this work, we investigate this failure and ide","cbCaikVQn8uVz3bv","https://ap.wps.com/l/cbCaikVQn8uVz3bv","pdf",566979,1,7,"English","en",105,"# Introduction\n# Literature Review\n## Normalization Techniques in Deep Learning","[{\"question\":\"What problem does the paper focus on in adaptive normalization for Transformers?\",\"answer\":\"It examines how differentiable normalization gating affects optimization, particularly whether a learned selector can reliably choose normalization strategies per layer and per task during training.\"},{\"question\":\"Why does the differentiable gating approach work well for language tasks but not for vision tasks?\",\"answer\":\"On stationary vision inputs, Gumbel–Softmax gating introduces high gradient variance early in training, preventing the selector from specializing and leading learned gates to underperform random selection.\"},{\"question\":\"What is AutoNorm-S (Stabilized) and how does it improve training?\",\"answer\":\"AutoNorm-S mitigates optimization instability using a gatefreezing schedule, which improves convergence of the gating mechanism and yields competitive or better performance across NLP benchmarks while staying competitive on standard vision results.\"}]",1784212210,18,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"autonorm-understanding-adaptive-normalization-in-transformers-through-differentiable-gating","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/autonorm-understanding-adaptive-normalization-in-transformers-through-differentiable-gating/86500/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper focus on in adaptive normalization for Transformers?","Question",{"text":75,"@type":76},"It examines how differentiable normalization gating affects optimization, particularly whether a learned selector can reliably choose normalization strategies per layer and per task during training.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why does the differentiable gating approach work well for language tasks but not for vision tasks?",{"text":80,"@type":76},"On stationary vision inputs, Gumbel–Softmax gating introduces high gradient variance early in training, preventing the selector from specializing and leading learned gates to underperform random selection.",{"name":82,"@type":73,"acceptedAnswer":83},"What is AutoNorm-S (Stabilized) and how does it improve training?",{"text":84,"@type":76},"AutoNorm-S mitigates optimization instability using a gatefreezing schedule, which improves convergence of the gating mechanism and yields competitive or better performance across NLP benchmarks while staying competitive on standard vision results.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,117,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":114,"show_sort_weight":115,"slug":116},"Healthcare",40,"healthcare",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":119,"show_sort_weight":120,"slug":121},8,"Research & Report",30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]