[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84232-en":3,"doc-seo-84232-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84232,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","FourierQK Spectral Preprocessing of Query Key Projections Improves Transformer Attention","FFT-based frequency-domain preprocessing applied to learned query and key projections substantially improves transformer attention for character-level language modelling. On TinyShakespeare, fixed random spectral filtering yields val=1.031 (Δ +0.443), a single learned frequency yields val=0.608 (Δ +0.867), and a four-frequency learned multi-scale scheme yields val=0.309 with a 79% validation-loss reduction versus standard dot-product attention. Results are reproducible across seeds and supported by a shuffled validation diagnostic that reduces positional leakage concerns.","arXiv :2607 .07478v 1 [ cs .LG] 8 Jul 2026  \nFourierQK: Spectral Preprocessing of Query–Key Projections Improves Transformer Attention  \nAthanasios Zeris∗  \n[https://orcid.org/0009-0002-6907-2400](https://orcid.org/0009-0002-6907-2400)  \nAbstract  \nFFT-based frequency-domain preprocessing of learned query and key projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (∆ = +0 .443); a single learned frequency initialised at paragraph scale achieves val=0.608 (∆ = +0 .867); and multi-frequency spectral attention with four learned frequencies spanning paragraph to word scale achieves val=0.309 (∆ =+1 . 166)—a 79% reduction in validation loss over standard dot-product attention.  \nThe single-frequency result is confirmed across three independent random seeds (mean val=0.236, std=0.019), establishing reproducibility. The four learned frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens per cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales in dramatic text.  \nThe improvement appears to be specific to spectral preprocessing: neither random orthogonal rotations nor random non-orthogonal projections of Q/K produce measurable gains over standard attention, suggesting the benefit comes from global sequence mixing in the frequency domain before score computation rather than from metric distortion or representation remapping. All results are verified using a shuffled validation diagnostic that provides evidence against positional leakage.  \nCausal time-domain filters (Gaussian, Mexican Hat, causal Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT reconstruction kernel κ(−τ ) = κ (τ ) is structurally non-causal, coupling every position to future tokens regardless of boundary handling. This identifies a precise architectural boundary between globally-mixing spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation [Zeris, 2026f] .  \nThis work is architecturally distinct from Lee-Thorp et al. [2021] (FNET), which replaces attention with Fourier mixing of token embeddings and has no Q/K projections or attention score matrix. Here, spectral preprocessing is applied only to the learned Q/K projections while the full attention score structure is preserved, enabling the frequency hierarchy to emerge from the attention mechanism itself.  \n1 Introduction  \nStandard transformer attention computes pairwise scores as dot products of learned query and key projections:  \n eij = qi√·~~ ~~kdj~~ ~~ q = WQ x, k = WK x (1)  \n∗Independent Researcher, Athens, Greece.  \nCorrespondence: [athzeris@gmail.com](athzeris@gmail.com).  \nORCID: [https://orcid.org/0009-0002-6907-2400](https://orcid.org/0009-0002-6907-2400) .  \nCode: [https://github.com/AthanasiosZeris/energy-gated-attention](https://github.com/AthanasiosZeris/energy-gated-attention).  \nPart of a seven-paper series on spectral methods in transformer attention.  \nPreprint. Under review.  \nThis computes similarity in the embedding space learned by WQ and WK . A natural question is: does transforming Q and K into a different representation space before computing similarity improve attention?  \nThis paper investigates spectral preprocessing—applying frequency-domain filters to Q and K before the score computation. The motivation comes from prior work in this series: Papers 1–4 [Zeris, 2026a,b,c,d] established that spectral energy and phase structure in transformer representations are informative signals.  \nMain contribution. We show that FFT-based bilateral spectral preprocessing of Q/K projections genuinely improves language modelling, even with random (unlearned) filters. The improvement is verified using a shuffled validation diagnostic: models trained on reordered validation sequences achieve much higher loss, confirming the gain comes from genuine sequence learning (ev","cbCainxq3mHQSqWF","https://ap.wps.com/l/cbCainxq3mHQSqWF","pdf",506906,1,16,"English","en",105,"# Abstract\n# Introduction\n# Spectral Attention\n## Architecture\n## Filter variants","[{\"question\":\"How do the authors verify that the gains come from genuine sequence learning rather than artifacts?\",\"answer\":\"They use a shuffled validation diagnostic where models trained on reordered validation sequences show a much higher loss, indicating the improvement is not due to positional artifacts or leakage.\"}]",1784194215,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"fourierqk-spectral-preprocessing-of-query-key-projections-improves-transformer-attention","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/fourierqk-spectral-preprocessing-of-query-key-projections-improves-transformer-attention/84232/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"How do the authors verify that the gains come from genuine sequence learning rather than artifacts?","Question",{"text":75,"@type":76},"They use a shuffled validation diagnostic where models trained on reordered validation sequences show a much higher loss, indicating the improvement is not due to positional artifacts or leakage.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,111,114,119,122,126],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":45,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":45,"category_name":109,"show_sort_weight":28,"slug":110},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":112,"slug":113},30,"research-report",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":98,"slug":129},19,"General","general"]