FourierQK Spectral Preprocessing of Query Key Projections Improves Transformer Attention

Jul 16, 2026

FFT-based frequency-domain preprocessing applied to learned query and key projections substantially improves transformer attention for character-level language modelling. On TinyShakespeare, fixed random spectral filtering yields val=1.031 (Δ +0.443), a single learned frequency yields val=0.608 (Δ +0.867), and a four-frequency learned multi-scale scheme yields val=0.309 with a 79% validation-loss reduction versus standard dot-product attention. Results are reproducible across seeds and supported by a shuffled validation diagnostic that reduces positional leakage concerns.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Spectral Attention

Architecture

Filter variants

arXiv :2607 .07478v 1 [ cs .LG] 8 Jul 2026 FourierQK: Spectral Preprocessing of Query–Key Projections Improves Transformer Attention Athanasios Zeris∗ [https://orcid.org/0009-0002-6907-2400](https://orcid.org/0009-0002-6907-2400) Abstract FFT-based frequency-domain preprocessing of learned query and key projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a ﬁxed random spectral ﬁlter achieves val=1.031 (∆ = +0 .443); a single learned frequency initialised at paragraph scale achieves val=0.608 (∆ = +0 .867); and multi-frequency spectral attention with four learned frequencies spanning paragraph to word scale achieves val=0.309 (∆ =+1 . 166)—a 79% reduction in validation loss over standard dot-product attention. The single-frequency result is conﬁrmed across three independent random seeds (mean val=0.236, std=0.019), establishing reproducibility. The four learned frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens per cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales in dramatic text. The improvement appears to be speciﬁc to spectral preprocessing: neither random orthogonal rotations nor random non-orthogonal projections of Q/K produce measurable gains over standard attention, suggesting the beneﬁt comes from global sequence mixing in the frequency domain before score computation rather than from metric distortion or representation remapping. All results are veriﬁed using a shufﬂed validation diagnostic that provides evidence against positional leakage. Causal time-domain ﬁlters (Gaussian, Mexican Hat, causal Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT reconstruction kernel κ(−τ ) = κ (τ ) is structurally non-causal, coupling every position to future tokens regardless of boundary handling. This identiﬁes a precise architectural boundary between globally-mixing spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation [Zeris, 2026f] . This work is architecturally distinct from Lee-Thorp et al. [2021] (FNET), which replaces attention with Fourier mixing of token embeddings and has no Q/K projections or attention score matrix. Here, spectral preprocessing is applied only to the learned Q/K projections while the full attention score structure is preserved, enabling the frequency hierarchy to emerge from the attention mechanism itself. 1 Introduction Standard transformer attention computes pairwise scores as dot products of learned query and key projections: eij = qi√·~~ ~~kdj~~ ~~ q = WQ x, k = WK x (1) ∗Independent Researcher, Athens, Greece. Correspondence: [athzeris@gmail.com](athzeris@gmail.com). ORCID: [https://orcid.org/0009-0002-6907-2400](https://orcid.org/0009-0002-6907-2400) . Code: [https://github.com/AthanasiosZeris/energy-gated-attention](https://github.com/AthanasiosZeris/energy-gated-attention). Part of a seven-paper series on spectral methods in transformer attention. Preprint. Under review. This computes similarity in the embedding space learned by WQ and WK . A natural question is: does transforming Q and K into a different representation space before computing similarity improve attention? This paper investigates spectral preprocessing—applying frequency-domain ﬁlters to Q and K before the score computation. The motivation comes from prior work in this series: Papers 1–4 [Zeris, 2026a,b,c,d] established that spectral energy and phase structure in transformer representations are informative signals. Main contribution. We show that FFT-based bilateral spectral preprocessing of Q/K projections genuinely improves language modelling, even with random (unlearned) ﬁlters. The improvement is veriﬁed using a shufﬂed validation diagnostic: models trained on reordered validation sequences achieve much higher loss, conﬁrming the gain comes from genuine sequence learning (ev

FAQs

How do the authors verify that the gains come from genuine sequence learning rather than artifacts?

They use a shuffled validation diagnostic where models trained on reordered validation sequences show a much higher loss, indicating the improvement is not due to positional artifacts or leakage.

Jul 16, 2026

Upload by Seraphina