Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion

Jul 16, 2026

Upload by Patrick

Continuous diffusion language models report record-low generative perplexity (Gen-PPL), but the metric can be misleading: these models repeat substantially more than human text, and Gen-PPL rewards that repetition. Removing the repeated component increases Gen-PPL markedly and reveals systematic repetition loops. The cause is traced to a one-dimensional contractive attractor in self-conditioning feedback, where a single direction steers samples into a repetition basin. ACE (Attractor-Contrast-Escape) subtracts this direction, reducing repetition near human level while keeping quality competitive and transferring across model sizes and samplers.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Abstract

Introduction

Defect: Gen-PPL rewards repetition

Mechanism: one-dimensional self-conditioning attractor

Fix: ACE (Attractor-Contrast-Escape)

Evaluation considerations

arXiv :2607 .00588v 1 [ cs .CL] 1 Jul 2026 LOW PERPLEXITY IS REPETITION: A ONEDIMENSIONAL SELF-CONDITIONING ATTRACTOR IN CONTINUOUS DIFFUSION LMS Shuai Zhang 1 ,2 , Zijie Chen2 , Hongliang He2 , Lun Du3 ,†, Zhenzhong Lan2 ,† 1Zhejiang University 2Westlake University 3Ant Group [zhangshuai@westlake.edu.cn](zhangshuai@westlake.edu.cn) [lanzhenzhong@westlake.edu.cn](lanzhenzhong@westlake.edu.cn)[ ](lanzhenzhong@westlake.edu.cn)†Corresponding authors ABSTRACT Continuous diffusion language models such as ELF report record-low generative perplexity (Gen-PPL) . We find a catch: these models repeat far more than human text, and Gen-PPL rewards rather than penalizes that repetition, so its low scores overstate quality. Strip the repetition and ELF-B’s Gen-PPL rises from 19.5 to 27.7; the smallest model even posts the best Gen-PPL because it repeats most. Wetrace the repetition to its source: a contractive attractor along a single direction in the self-conditioning feedback loop, the loop that feeds each step’s clean estimate into the next. Because the failure is one-dimensional, a one-dimensional fix suffices, and we propose one. ACE (Attractor-Contrast-Escape) subtracts that single, label-free direction from the feedback at each step. Estimated once on the 105M model, the direction cuts repetition to near the human level while keeping quality competitive, and transfers near-unchanged to the 342M and 652M models and across samplers; the same recipe recovers useful directions on other architectures. Since Gen-PPL itself rewards repetition, we instead measure the compute each fix needs to produce human-clean text, where ACE is 1.5–5 × cheaper. 1 INTRODUCTION Continuous diffusion language models (DLMs) are a promising non-autoregressive route to text generation: they denoise a whole sequence in parallel within a differentiable embedding space, steerable by gradients and guidance. Self-conditioning (Chen et al., 2022) improves their sample quality by feeding the model’s own clean estimate back into each step to refine the next. Recent models such as ELF (Hu et al., 2026) report low generative perplexity (Gen-PPL), the number the field reads as generation quality. We find that this headline hides a defect: ELF’s samples repeat far more than human text, and Gen-PPL rewards the repetition instead of penalizing it. Strip therepetition and ELF-B’s Gen-PPL rises from 19.5 to 27.7, enough for the larger ELF-M to overtake it; the smallest model posts the best Gen-PPL only because it repeats the most (Table 1) . The defect is heavy and systematic. A large share of ELF samples lock onto a few repeated 4-grams and loop them for hundreds of words (Table 17), which human text essentially never does. The link is not only across models but within one: at a fixed setting, sample-level repetition correlates with the GPT-2 (Radford et al., 2019) PPL the samples are scored by (Table 16) . The defect stays hidden because the certifying metric is blind to it: repeated text is highly probable under the scorer, so it earns a flatteringly low Gen-PPL, analogous to likelihood-based degeneration in autoregressive generation (Holtzman et al., 2020; Welleck et al., 2020) . We trace the defect to its mechanism rather than stop at the symptom. Like audio feedback, this self-conditioning loop settles on whatever is most self-predictable, which is repeated content. Two probes pin this down. Turning the feedback strength up, with nothing else changed, drives repetition up and Gen-PPL down together (§3): the loop creates the repetition the metric then rewards. And linearizing the loop, its Jacobian has a single slowest-contracting mode, so the repeated state is a one-dimensional contractive attractor along one direction d (Fig. 1; §4), a basin ~~ ~~ 2 ~~ ~~ 1 0 1 2 3 4 u projection onto d Figure 1: Repetition is a basin; ACE escapes it. Even as ELF denoises toward a clean sample, self-conditioning drags its representation u along one direction d

FAQs

Why do continuous diffusion language models show very low Gen-PPL despite poor human-likeness?

Gen-PPL rewards the repetition produced by the models. Because repeated text is highly probable under the scorer, the metric decreases even as human-like quality worsens.

What mechanism causes the excessive repetition in self-conditioned continuous diffusion models?

The self-conditioning feedback loop contains a one-dimensional contractive attractor. As denoising proceeds, representations drift along a single direction into a repetition basin.

How does ACE reduce repetition while maintaining competitive generation quality?

ACE subtracts the single high-minus-low repetition direction from the self-conditioning feedback at every step. The direction is recovered label-free and no retraining is required, leading to near-human repetition rates with competitive quality.

Jul 16, 2026

Upload by Patrick