[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85769-en":3,"doc-seo-85769-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85769,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","Learning in Curved Weight Space: Exponential-Linear Weight Reparameterization for Improved Optimization","Many neural network operations behave multiplicatively, yet standard training performs additive, linear parameter updates, creating unequal “optimization distances” for weight changes with different magnitudes. Adaptive methods normalize per coordinate but still produce additive steps, yielding mismatched relative perturbations across heterogeneous weight scales. The paper introduces SymExpLin (SEL), combining a sign-aware symmetric-exponential transform with an identity-like linear pathway plus learnable balance parameters. This curved geometry empirically accelerates loss descent and reduces training steps, while folding back into standard linear weights at inference.","arXiv :2607 .09967v 1 [ cs .LG] 10 Jul 2026  \nLearning in Curved Weight Space:  \nExponential-Linear Weight Reparameterization for Improved  \nOptimization  \nEthan Smith  \nCanva Research  \n[ethansmith@canva.com](ethansmith@canva.com)  \nAbstract  \nMany neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations. We introduce SymExpLin (SEL), a weight reparameterization for neural networks that combines a sign-aware symmetric-exponential pathway with an identity-like linear pathway. The symmetricexponential pathway is near-linear for small raw weights but increasingly curved at larger magnitudes. Additive updates in logarithmic space map to magnitude-proportional changes in effective weight space. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway. These components create a curved parameter-space geometry that empirically improves speed of loss descent over standard linear parameterization. We also identify a useful mismatched initialization: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small-model ablations, this improves early optimization and may act as a form of symmetry breaking. We train transformers on OpenWebText over nine width×depth configurations, SEL reaches matched validation loss in 1.32–1.49× fewer training steps, with the largest widths seeing the biggest gains. SEL is practical: in our largest profiled configuration, a 1.44× step reduction and 5.5% per-step overhead correspond to an estimated 1.37 × wall-clock speedup, and after training the parameterization is folded into standard linear weights with no inference cost.  \n1 Introduction  \nStandard neural network training optimizes parameters in an additive fashion with linear update steps. Yet many computations within a network are naturally relative. Doubling or halving a normalization gain, a feature amplitude, or the singular value of a matrix reflects analogous changes in logarithmic space. However, they have unequal optimization distances in linear weight space. More specifically, moving from 1 to 0.5 has half the linear distance of moving from 1 to 2. This is not to say we should necessarily think of all aspects of the network in a multiplicative fashion. Activation functions alongside biases consider absolute scale tobe important. Nevertheless, the optimization employed in training has mostly played into the linear/absolute scale nature of networks. When relative changes weights are to be achieved through additive updates, we should expect training to push a subset of parameters far from initialization whenever the task demands large multiplicative factors, while leaving many others near their starting scale.  \nAs shown in Figure 1, we may observe this effect in pretrained language models, which develop heavytailed weight distributions. Many parameters remain near initialization scale, while a small fraction move far into the tails. In our Qwen3-4B-Instruct analysis (Figure 1), the maximum weight is roughly 30 × the Xavier initialization scale even though the standard deviation grows only 1.7 × . This creates a heterogeneous optimization problem in which many coordinates may travel little from their original positions, while some coordinates must traverse much larger distances in weight space.  \nAdaptive optimizer","cbCaiidmbH59bPxR","https://ap.wps.com/l/cbCaiidmbH59bPxR","pdf",1173102,1,24,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"Why do standard linear parameter updates create an optimization mismatch in neural networks?\",\"answer\":\"Because many effects in networks are naturally relative or multiplicative, while training uses additive updates in linear weight space, leading to unequal optimization distances across weight magnitudes.\"},{\"question\":\"What is SymExpLin (SEL) and how does it change the weight geometry?\",\"answer\":\"SEL reparameterizes weights using a sign-preserving symmetric-exponential pathway together with an identity-like linear pathway, controlled by learnable scale/curvature/offset parameters to balance their contributions.\"},{\"question\":\"How does SEL improve training efficiency, and what is the inference-time impact?\",\"answer\":\"Experiments on transformers trained over multiple width×depth configurations show SEL reaches matched validation loss in fewer steps, with reported step reduction translating to wall-clock speedup; after training, the reparameterization can be folded into standard linear weights, yielding no inference cost.\"}]",1784206132,60,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"learning-in-curved-weight-space-exponential-linear-weight-reparameterization-for-improved-optimization","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/learning-in-curved-weight-space-exponential-linear-weight-reparameterization-for-improved-optimization/85769/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why do standard linear parameter updates create an optimization mismatch in neural networks?","Question",{"text":74,"@type":75},"Because many effects in networks are naturally relative or multiplicative, while training uses additive updates in linear weight space, leading to unequal optimization distances across weight magnitudes.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What is SymExpLin (SEL) and how does it change the weight geometry?",{"text":79,"@type":75},"SEL reparameterizes weights using a sign-preserving symmetric-exponential pathway together with an identity-like linear pathway, controlled by learnable scale/curvature/offset parameters to balance their contributions.",{"name":81,"@type":72,"acceptedAnswer":82},"How does SEL improve training efficiency, and what is the inference-time impact?",{"text":83,"@type":75},"Experiments on transformers trained over multiple width×depth configurations show SEL reaches matched validation loss in fewer steps, with reported step reduction translating to wall-clock speedup; after training, the reparameterization can be folded into standard linear weights, yielding no inference cost.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,108,113,118,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":28,"slug":107},5,"Comic","comic",{"id":109,"doc_module":4,"doc_module_name":45,"category_name":110,"show_sort_weight":111,"slug":112},6,"Technology",50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":119,"slug":120},30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]