[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85180-en":3,"doc-seo-85180-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85180,962075114765,"Quinn","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","LeRoPE Learnable RoPE Frequencies Improve Language Modeling","Rotary Positional Encodings (RoPE) rotate query–key vector pairs by angles determined by relative token offsets. In RoPE, per-frequency rotation rates follow a geometric sequence set by a fixed base-frequency hyperparameter, and prior improvements adjust either the base or the set of rotated dimensions. Learned RoPE (LeRoPE) replaces these hyperparameters with learnable scalars per frequency band. Training language models from 52M to 2.5B parameters shows LeRoPE consistently outperforming RoPE and partial RoPE, with RoPE requiring 3.4% more FLOPs at the largest scale.","LeRoPE: Learnable RoPE Frequencies Improve Language Modeling  \nPetros Karypis∗ UC San Diego  \nSean O’Brien∗ UC San Diego  \nShreyas Kadekodi  \nUC San Diego  \nRui Zhu  \nIndependent Researcher  \narXiv :2607 . 10 134v 1 [ cs .LG] 11 Jul 2026  \nJulian McAuley  \nUC San Diego  \nAbstract  \nRotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset.  \nThe position-wise rates of rotation in RoPE typically follow a geometric sequence specified by a fixed base-frequency hyperparameter. Prior work has improved performance by either increasing this parameter to slow rotation or by applying RoPE to only a subset of QK dimensions. In this work we modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters. We validate Learned RoPE by training a ladder of language models from scratch, ranging from 52M to 2.5B parameters. We observe and analyze the emergence of a high-norm, positional LeRoPE band. LeRoPE consistently outperforms RoPE and partial RoPE across all scales, with RoPE requiring 3.4% more compute (FLOPs) to match LeRoPE at the largest scale.  \n1 Introduction  \nPositional encodings are typically included in Transformer-based models [Vaswani et al., 2017] because the self-attention operation on its own does not directly carry positional signal. Rotary Positional Encodings (RoPE) [Su et al., 2021] are used by a number of popular language models [Touvron et al., 2023, Chowdhery et al., 2022, Jiang et al., 2023, Kamath et al., 2025] . RoPE splits the query and key vectors into two-dimensional pairs, or frequency bands, and rotates each pair by an angle proportional to token position. The effect of these rotations on the query–key inner product depends only on the relative offset between tokens. Each band rotates at its own fixed rate, following a geometric sequence set by a base hyperparameter: high-frequency bands complete many rotations within a context, while the lowest-frequency bands rotate minimally.  \nThe choice of these frequencies matters. Recent work finds that the slowest bands carry semantic rather than positional information [Barbero et al., 2024] . This positional invariance allows content matches to hold at any separation, making it beneficial to remove rotation from a subset of bands entirely, or to adjust the base, which sets how many bands rotate appreciably within the context window [Barbero et al., 2024, Men et al., 2024, Oka et al., 2026b] .  \nMotivated by the above, and noting that the RoPE operation is differentiable with respect to its frequency, we propose learning the frequencies. Learned RoPE (LeRoPE) adds one learned scalar per frequency band, 32 parameters in total for each of our models. Training on a ladder of models from 52M to 2.5B parameters, LeRoPE outperforms RoPE at every scale. At the largest scale, RoPE requires 3 .4% more compute to match the performance of LeRoPE (Figure 1) .  \n∗ Primary authors. Correspondence: [pkarypis@ucsd.edu](pkarypis@ucsd.edu)  \nPreprint.  \nCompute Multiplier Across Scales In-Distribution Loss Gap  \nModel Size (Total Params)  \n52M 217M 608M 1 . 3B 2. 5B  \nCompute multiplier  \n1 .08 ×  \n1 .06 ×  \n1 .04 ×  \n1 .02 ×  \n1 .00 ×  \nLoss vs . RoPE (Nats)  \n0.000  \n-0.005  \n-0.010  \n-0.015  \n-0.020  \np-RoPE RoPE  \nLeRoPE  \n0 1024 2048 Token Position  \nFigure 1: a: Compute multipliers vs. RoPE. The compute multiplier is the factor by which the baseline’s training compute must increase to match the variant’s loss. The multiplier declines from ∼ 1.07 at the smallest scale and plateaus near 1 .03–1.04, a small but persistent advantage. This means RoPE needs 3.4% more compute to reach the loss LeRoPE achieves at 2 .5B. All runs use a single seed; learning rates are swept per method for models below 1.34B (§4.1) . b: Average loss gap per token position","cbCaieseD0IEnSyL","https://ap.wps.com/l/cbCaieseD0IEnSyL","pdf",6032297,1,27,"English","en",105,"# Introduction\n## Positional encodings and RoPE motivation\n# Preliminaries\n## Positional embeddings\n## RoPE","[{\"question\":\"What problem does LeRoPE address in Rotary Positional Encodings (RoPE)?\",\"answer\":\"RoPE uses fixed frequency rotation rates derived from a base-frequency hyperparameter. LeRoPE targets this limitation by learning frequencies rather than treating them as fixed design choices.\"},{\"question\":\"How does Learned RoPE (LeRoPE) modify RoPE?\",\"answer\":\"LeRoPE learns a scalar per frequency band, turning the rotation frequencies from hyperparameters into trainable parameters. This allows the model to adapt the rotation schedule during training.\"},{\"question\":\"What experimental results show LeRoPE’s effectiveness?\",\"answer\":\"Across a training ladder of language models from 52M to 2.5B parameters, LeRoPE consistently outperforms RoPE and partial RoPE. At the largest scale, RoPE needs about 3.4% more compute (FLOPs) to match LeRoPE’s performance.\"}]",1784201583,68,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"lerope-learnable-rope-frequencies-improve-language-modeling","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/lerope-learnable-rope-frequencies-improve-language-modeling/85180/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does LeRoPE address in Rotary Positional Encodings (RoPE)?","Question",{"text":75,"@type":76},"RoPE uses fixed frequency rotation rates derived from a base-frequency hyperparameter. LeRoPE targets this limitation by learning frequencies rather than treating them as fixed design choices.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does Learned RoPE (LeRoPE) modify RoPE?",{"text":80,"@type":76},"LeRoPE learns a scalar per frequency band, turning the rotation frequencies from hyperparameters into trainable parameters. This allows the model to adapt the rotation schedule during training.",{"name":82,"@type":73,"acceptedAnswer":83},"What experimental results show LeRoPE’s effectiveness?",{"text":84,"@type":76},"Across a training ladder of language models from 52M to 2.5B parameters, LeRoPE consistently outperforms RoPE and partial RoPE. At the largest scale, RoPE needs about 3.4% more compute (FLOPs) to match LeRoPE’s performance.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]