DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

Jul 16, 2026

State Space Models (SSMs) enable efficient long-sequence learning but must compress an unbounded history into a fixed-size state, making precise retrieval over long ranges inherently difficult. Delay State Space Models (DSSMs) extend diagonal SSM recurrences with explicit delayed-state feedback inspired by delay differential equations. The approach introduces new stability parameterization, history management, and discretization, while deriving frequency-domain transfer functions to enable accurate FFT training. Experiments show strong gains on delayed-retrieval tasks and competitive results on standard sequence metrics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Introduction

Background

Continuous State Space Models

Discretization and Convolutional View

arXiv :2607 . 10244v 1 [ cs .LG] 11 Jul 2026 DSSMs: State Space Models with Explicit Memory via Delay Differential Equations Yixiao Qian 1 Song Chen2 ,∗ Jiaxu Liu3 Shengze Cai 1 Chao Xu 1 ,∗ 1 College of Control Science and Engineering, Zhejiang University 2Department of Mathematics, National University of Singapore 3 School of Mathematical Sciences, Zhejiang University [song.chen@nus.edu.sg](song.chen@nus.edu.sg) , [cxu@zju.edu.cn](cxu@zju.edu.cn) ∗ Corresponding authors. Abstract State Space Models (SSMs) have emerged as a powerful paradigm for efficient long-sequence modeling, offering parallel training and fast linear-time recurrent inference. However, like other recurrent architectures, SSMs must compress an unbounded history into a fixed-size state, which limits context retention and makes precise retrieval over long-range context inherently difficult. To overcome this limitation, we propose Delay State Space Models (DSSMs), a delay differential equation (DDE)-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback. Making explicit delayed feedback practical requires new stability parameterization, history management, and FFTtraining tools. We address these challenges with a practical discretization and parameterization grounded in a simple delay-independent stability condition. To bypass direct time-domain kernel construction, we derive the DSSM transfer function and compute kernels in the frequency domain, using a kernel contour shift to suppress aliasing and recover accurate FFT training. Empirically, DSSMs substantially improve targeted delayed-retrieval tasks while outperforming S4D on most standard sequence metrics and remaining close on the others. 1 Introduction Sequence modeling is a core problem in modern deep learning. Recently, State Space Models (SSMs) have drawn increasing attention as an efficient family of sequence models because they enable parallel training and fast linear-time inference while showing strong performance on long-sequence tasks [Guet al., 2021, 2022, Gu and Dao, 2024] . However, like other recurrent architectures, SSMs must compress past information into a finite recurrent state, where it is represented through exponentially decaying modes [Wang and Xue, 2023, Wang and Li, 2023], making exact copying and precise retrieval harder than in attention-based models [Jelassi et al., 2024] . This limitation has motivated several complementary directions, including recall-oriented SSM layers such as H3 [Fu et al., 2022], more expressive state-space designs [Lahoti et al., 2026], and hybrid architectures that combine attention with SSMs, such as Jamba, Zamba, and Samba [Lieber et al., 2024, Glorioso et al., 2024, Ren et al., 2024] . This leaves open how to strengthen precise long-range memory while retaining efficient parallel training and fast inference: Transformers offer strong long-range memory and parallel training but incur expensive inference [Shazeer, 2019, Pope et al., 2023], whereas standard SSMs achieve parallel training and efficient inference yet still struggle with precise long-range retrieval. To overcome this bottleneck, we revisit the underlying dynamics and introduce time-delays into the state evolution. Unlike standard ordinary differential equations (ODEs), delay differential equations Preprint. | | <br> | | --- | --- | Figure 1: Overview of the discrete DSSM update. The state combines the input term˜, the main-branch term from ht−1, and the delayed-branch term from the buffered delayed state ht−τ . The formal discrete recurrence is developed in Sec. 3.2. (DDEs) explicitly incorporate historical states through delayed terms, thereby providing a natural mechanism for improving precise long-range memory. Related work in scientific machine learning likewise suggests that introducing delay-aware continuous dynamics can improve expressive power and long-horizon prediction [Zhu et al., 2021, Gup

FAQs

Why do standard state space models struggle with long-range precise retrieval?

They must compress an unbounded history into a fixed-size state represented by exponentially decaying modes, which makes exact copying and precise long-range retrieval harder than attention-based models.

What is a Delay State Space Model (DSSM)?

A DSSM is a delay differential equation-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback to improve long-range memory.

How does DSSM enable efficient FFT training despite delayed feedback?

It derives the DSSM transfer function and computes kernels in the frequency domain, using a kernel contour shift to suppress time-domain aliasing and recover accurate FFT training.

Jul 16, 2026

Upload by Seraphina