How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

Jul 16, 2026

Rotary Position Embeddings (RoPE) give transformers a fixed grid of positional frequencies, but trained models concentrate usage unevenly. The work studies what drives this non-uniform frequency selection and offers a data-centered explanation: frequencies are chosen to match the relative-distance structure of training data. Each frequency is treated as a positional lens, yielding a field-resolution tradeoff where the optimal frequency scales as 1/W for a dependency width W. The principle links RoPE frequency scaling to position-interpolation length generalization. It shows long-context generalization depends on scale matching between learned frequencies and training-time dependencies, and on how those dependencies extend to longer contexts, supported by self-similarity in natural language.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Abstract

Introduction

Inductive biases and positional preferences in attention

Motivation from non-uniform RoPE frequency usage

Data-centered theory: dependency profiles as positional lenses

Frequency scaling and the tradeoff between field and resolution

Position interpolation and conditions for length generalization

arXiv :2607 .07678v 1 [ cs .LG] 8 Jul 2026 How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization Xinyi Wu∗1 Siyuan Liu∗†2 Ali Jadbabaie 1 1MIT IDSS 2IIIS, Tsinghua University {xinyiwu,[jadbabai}@mit.edu](jadbabai}@mit.edu) [liusiyua23@mails.tsinghua.edu.cn](liusiyua23@mails.tsinghua.edu.cn) Abstract Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose a data-centered explanation: RoPE frequencies are selected to match the relative-distance structure of the training data. Viewing each frequency as a positional lens, we formalize a field-resolution tradeoff and show that, for a data-induced dependency profile of width W , the optimal frequency scales as 1/W. This frequency-matching principle explains controlled observations on synthetic and text-based data, and suggests that the mid-low frequency bands observed in language models arise from the multi-scale dependency structure of natural language. We further connect frequency selection to position-interpolation-based length generalization: scaling frequencies down expands the effective field while reducing resolution. This helps when longer-context dependencies are approximate dilations of those seen during training, but can fail when relevant dependencies do not scale with context length. Empirically, we show that natural language exhibits approximate self-similarity across positional scales, explaining why test-time frequency scaling can support long-context generalization. Overall, our results identify a data-driven mechanism behind emergent RoPE frequency usage and show that long-context generalization depends on two forms of scale matching: between learned frequencies and trainingtime dependencies, and between frequency scaling and how those dependencies extend to longer contexts. 1 Introduction A longstanding view in machine learning is that generalization depends on inductive biases, or preferences that prioritize certain solutions over others when data is finite [4, 28] . One of the central questions in modern machine learning is how such biases arise in models designed to be broadly flexible [34, 43] . Attention is a striking example [2, 20, 39]; although every token can in principle attend to every other token, trained attention models are far from structureless. They exhibit strong positional preferences [14, 22, 43], attention sinks [13, 44], head specialization [30], and highly non-uniform use of embedding dimensions [3, 17, 29] . These phenomena suggest that training does more than fit a predictor: it induces systematic preferences in how a flexible architecture uses its internal degrees of freedom. Rotary Position Embeddings (RoPE) provide a particularly clean instance of this phenomenon [33] . RoPE gives a model access to a fixed grid of positional frequencies, yet trained models do not use these frequencies uniformly. Instead, query and key representations often concentrate their norm in restricted frequency bands [3, 17, 29] . This raises a fundamental question: What determines which RoPE frequencies a model learns to use? One possible explanation is that high RoPE frequencies encode position while low frequencies encode semantics [3] . This intuition is partly motivated by the fact that very low RoPE frequencies *Equal contribution. † Work done during visit at MIT. Preprint. vary slowly over the training context and are therefore nearly invariant to small changes in position. However, low sensitivity to position should not be mistaken for non-positional information. A low frequency in RoPE still modulates attention scores as a function of relative distance; it simply varies more gradually across the context. Moreover, prior work shows that changing the training sequence length alone can shift RoPE frequency usage [29], suggest

FAQs

What determines which RoPE frequencies a model uses after training?

The study argues that RoPE frequencies are selected to match the relative-distance structure of the training data, expressed through a data-induced positional dependency profile.

How is RoPE frequency usage formalized in the proposed theory?

Each RoPE frequency is treated as a positional lens, leading to a field-resolution tradeoff: higher frequencies give sharper local resolution but wrap sooner, while lower frequencies cover broader ranges more coarsely.

Why does position interpolation help with long-context length generalization, and when can it fail?

It rescales RoPE frequencies so the effective field grows while local resolution decreases, helping when test-time dependencies are approximately stretched versions of training-time ones; it can fail when relevant dependencies do not scale with context length.

Jul 16, 2026

Upload by Clementine