FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

Jul 16, 2026

Upload by Jiven

Parameter-efficient fine-tuning (PEFT) uses small trainable updates, but existing approaches effectively hard-code a single representation domain: LoRA constrains updates in the spatial domain, while spectral adapters constrain updates in the Fourier domain. This work treats the domain choice as a learnable design variable. FRAME (Fractional-Fourier Mixture of Experts) routes tokens to fractional-Fourier experts with trainable orders that interpolate between spatial and Fourier endpoints, improving task and layer specialization with minimal added cost.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Abstract

Introduction

Parameter-efficient fine-tuning and LoRA

Spatial vs. Fourier adapters

Fractional-Fourier as a continuous domain

FRAME method overview

Results and analysis

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts Tom Saliencro1 , Maya Lindqvist1 , Rohan Desai2 , Priya Nair1 , Daniel Whitmore2 1University of California, Irvine 2University of Washington [saliencro@gmail.com](saliencro@gmail.com) arXiv :2607 .00162v1 [ cs .LG] 30 Jun 2026 Abstract Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce FRAME (FractionalFourier Mixture of Experts), a mixture-ofexperts adapter in which every expert carries alearnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter) . Routing tokens through experts that occupy different points on this spatial–spectral continuum lets the model place each low-rank update in the domain where it is most compact, and—because fractional-Fourier operators of different orders are mutually incoherent—makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an O (dlog d) chirp–FFT surrogate, so FRAME adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLAMA-3 . 1-8B and QWEN2. 5-7B, FRAME improves over strong MoE-LoRA and spectral baselines—including FlyLoRA, FourierMoE, and HMoRA—while keeping the activeparameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways. 1 Introduction Parameter-efficient fine-tuning (PEFT) adapts a frozen foundation model by training a small number of additional parameters (Houlsby et al., 2019 ; Li and Liang, 2021 ; Lester et al., 2021) . Lowrank adaptation (LoRA) is the dominant instance: it writes the weight update of a linear layer as a product of two low-rank matrices and has become a default for instruction tuning and domain specialization (Hu et al., 2022 ; Dettmers et al., 2023) . A natural way to increase the capacity of a single adapter without inflating its active parameter count is to turn it into a mixture of experts (MoE), routing each token to a few specialized low-rank modules (Dou et al., 2023 ; Li et al., 2024 ; Wu et al., 2024 ; Tian et al., 2024) . Almost all of these methods share a hidden assumption: the update is parameterized in the spatial (canonical coordinate) domain, where the low-rank prior is imposed directly on the weight matrix. A separate and growing line of work instead parameterizes the update in the Fourier domain, learning a sparse or low-rank set of spectral coefficients and mapping them back with an inverse transform (Gao et al., 2024 ; Borse et al., 2024 ; Bilican et al., 2025 ; Zhang et al., 2025) . Spectral adapters are attractive because pretrained weight updates are often spectrally concentrated, and because the Fourier basis is global: a few coefficients can express a high-rank spatial update. The most recent spectral method even builds a mixture of frequency-band experts with a frequency-aware router (Jiang et al., 2026) . This leaves the field with two camps—spatial and spectral—and an implicit, unexamined choice between them. We make that choice explicit and ask a different question: in which domain should an adapter be low-rank? Our answer is that the domain should not be fixed at all. The spatial and Fourier bases are merely the two endpoints of a one-parameter family of unitary transforms, the fractional-Fourier transform (FrFT), whose order a rotates a signal continuously in the time–frequency plane: a=0 is the identi

FAQs

What problem does FRAME address in PEFT methods like LoRA and spectral adapters?

FRAME addresses the implicit assumption that the adaptation domain should be fixed. It argues that neither the spatial domain nor the Fourier domain is universally optimal across tasks, layers, or tokens.

How does FRAME combine spatial and Fourier representations?

FRAME uses a mixture of experts where each expert applies a low-rank update in a fractional-Fourier domain with a learnable scalar order. Setting the order to 0 recovers LoRA (spatial), and setting it to 1 recovers a Fourier-domain spectral adapter.

Why does FRAME claim its experts are less interfering during multi-task composition?

Experts with different fractional-Fourier orders are described as mutually incoherent, which naturally decorrelates them. This reduces interference and improves the composition of multiple tasks.

Jul 16, 2026

Upload by Jiven