LOST IN SAMPLING - ASSESSING LEXICAL REACHABILITY IN LLMS VIA THE WORD COVERAGE SCORE (WCS) - Preprint

Sep 15, 2026

Uploaded by Patrick

Modern Large Language Models (LLMs) are often criticized for repetitive, homogeneous text despite large latent vocabularies. This work shifts focus from training data and model knowledge to decoding mechanics, showing how standard sampling filters prune contextually appropriate human words. It introduces the Word Coverage Score (WCS), measuring lexical survival of low-frequency, high-information words as a function of sampling parameters. Auditing open-weight models on human-authored corpus fragments identifies reachable probability mass that still fails to realize key lexical choices, supporting a framework for trading coherence against lexical richness.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

Homogeneity and lexical diversity in LLM outputs

Zipfian and Mandelbrot-like vocabulary behavior

Word Coverage Score (WCS) and the four-stage methodology

LOST IN SAMPLING: ASSESSING LEXICAL REACHABILITY IN LLMS VIA THE WORD COVERAGE SCORE (WCS) arXiv : 2605 .27268v2 [cs .CL] 2 1 Aug 2026 Samer Awad, Javier Conde, Carlos Arriaga Information and Processing Telecommunications Center Universidad Politécnica de Madrid Madrid, Spain Tairan Fu Politecnico di Milano Milano, Italy Javier Coronado-Blázquez Banco de España Madrid, Spain Pedro Reviriego Information and Processing Telecommunications Center Universidad Politécnica de Madrid Madrid, Spain August 24, 2026 ABSTRACT Modern Large Language Models (LLMs) are often criticized for producing repetitive and homogeneous text, despite possessing vast latent vocabularies. While previous research has focused on model knowledge and training data, we investigate the role of decoding mechanics in suppressing linguistic diversity. We introduce the Word Coverage Score (WCS), a metric that quantifies the extent to which contextually appropriate human vocabulary is mathematically pruned by standard sampling filters (e.g., Top-p, Top-k, and Min-p) . Rather than assessing static knowledge, the WCS measures the lexical survival rate of low-frequency, high-information human words as a function of sampling parameters. By auditing open-weight models on human-authored corpus fragments, we identify which logical lexical choices are rendered unreachable by the decoder, even when they reside within the probability space. Our results provide quantitative evidence that industry-standard sampling defaults act as unintended censorship mechanisms, smoothing the unique textures of human expression into a homogenized discourse. The WCS offers a rigorous framework for optimizing the trade-off between text coherence and lexical richness, providing a diagnostic tool for preserving the diversity of human language in generative models. 1 Introduction The texts generated by Large Language Models (LLMs) tend to be homogeneous, lacking the richness of human discourse and often converging toward a narrow set of typical phrasing and structural patterns [1] . Recent large-scale studies indicate that modern LLMs produce outputs that are more similar to one another across different model families than human-authored texts [2] . This homogenization is characterized by a structural collapse of the probability distribution, often termed mode collapse, where models consistently favor safe, high-probability sequences at the expense of lexical variety [3] . One possible cause of this phenomenon is model alignment or instruction tuning [4], implemented using techniques such as Reinforcement Learning with Human Feedback (RLHF) or Direct Preference Optimization (DPO), in which models learn to prioritize familiar-sounding responses that annotators prefer, thereby amplifying distribution collapse [5, 6, 7] . A relevant aspect of this homogeneity is how LLMs use vocabulary. Recent studies have shown that text produced by some models has lower lexical diversity [8] . A comprehensive evaluation of these features in conversational models reveals that lexical richness is highly sensitive to model parameters, such as presence penalties and temperature, as well as the specific roles assigned to the model [9] . This reduction in variety is particularly evident in the suppression of rare, low-frequency words that exist in the training data but are rarely selected during inference. Interestingly, while A PREPRINT-AUGUST 24, 2026 human language follows the classical Zipf’s Law [10], frontier LLM outputs across various vendors have been recently found to converge toward a two-parameter Mandelbrot ranking distribution [11] which reveals that LLMs suffer from an artificial steepness in the Zipfian tail, indicating that the probability of selecting rare tokens decays significantly faster than in natural human corpora. This suggests that aligned models are constrained in the depth of the vocabulary they can use. Previous research has focused on model knowledge an

FAQs

What problem does the paper address about LLM-generated text?

The paper addresses the tendency of LLM outputs to become repetitive and homogeneous, showing reduced lexical diversity compared with human-authored text.

What is the Word Coverage Score (WCS)?

WCS is a metric that quantifies how much contextually appropriate human vocabulary is pruned by standard sampling filters during generation, focusing on the survival of low-frequency informative words.

How does the Forced-Path Audit evaluate lexical reachability?

It tests whether standard samplers (e.g., Top-p, Top-k, Min-p) mathematically censor specific target words when placed in human-authored passages, even if those words lie within the model’s probability space.

Sep 15, 2026

Uploaded by Patrick