TokEval - A Tokenizer Evaluation Suite - arXiv 2608.18062v3

Sep 15, 2026

Uploaded by Aria

Language model tokenizers are often chosen using limited evaluation, although tokenizer design strongly influences downstream model capabilities. TokEval is introduced as a framework of intrinsic tokenizer evaluation metrics extending beyond traditional measures like fertility and compression rate. It targets linguistically and structurally meaningful properties such as UTF-8 boundary integrity and digit place-value alignment. Controlled pretraining isolates tokenizer training mixture, pretokenization strategy, and training algorithm, and evaluates language modeling, mathematical reasoning, and code generation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages39

Reading Time1 h 38 min

AccessFree to download

Introduction

TokEval Contributions

Background and Related Work

Tokenization Algorithms

arXiv : 2608 . 18062v 3 [cs .CL] 9 Sep 2026 TokEval: A Tokenizer Evaluation Suite Clara Meister EPFL [clara.meister@epfl.ch](clara.meister@epfl.ch) Abstract Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers’ training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting modelson bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman |ρ| up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree. § cimeister/tokenizer-intrinsic-evals [huggingface.co/cmeister/tokenizer-lm-ablations](huggingface.co/cmeister/tokenizer-lm-ablations) 1 Introduction Subword tokenization is a ubiquitous component of modern language model pipelines. Nearly all contemporary large language models (LLMs) rely on some variant of subword segmentation (e.g., Byte Pair Encoding (BPE; Gage, 1994; Sennrich et al., 2016) or the UnigramLM algorithm (Kudo, 2018)) to map raw text to discrete token sequences, the format on which models can be efficiently trained and evaluated. Despite this central role, tokenizer design often receives limited attention. In contrast to the substantial efforts devoted to optimizing model architectures, training data, and learning algorithms, the tokenizeris typically selected using a small set of heuristics, such as vocabulary size, compression rate, or informal inspection of segmentation quality. The choice is then rarely revisited over the course of model development. This neglect persists despite growing evidence that tokenization choices shape downstream model behavior across various domains, including arithmetic accuracy, multilingual performance, and general task quality (Ali et al., 2024; Singh & Strouse, 2024; Liu et al., 2025; Rust et al., 2021; Lesci et al., 2025) . Taken together, these results show that tokenizer design is a substantive modeling choice with measurable consequences for downstream model capabilities. Perhaps a core reason for this lack of consideration is that testing out tokenizers is an expensive endeavor: the entire pipeline, from text processing to output formatting, must be reconfigured for each different design choice. Furthermore, there are two clear issues with the current intrinsic evaluation protocols:1 Firstly, the current set of intrinsic evaluation 1Intrinsic evaluation refers to evaluating a tokenizer in isolation, using only the tokenizer and a corpus, without training a downstream model. metrics is very narrow. For example, compression rate tells us only how much a tokenizer compresses, but not how well it preserves the structural properties that matter for specific downstream tasks. Second, the relationship between intrinsic metrics and downstream performance remains contested. As concrete examples, Zouhar et al. (2023) found that Rényi e

FAQs

What gap does TokEval address in tokenizer evaluation?

TokEval targets limitations of existing intrinsic evaluation by introducing richer tokenizer metrics beyond compression rate and by using controlled experiments to clarify which intrinsic properties predict downstream performance.

How are downstream effects tested in the paper?

The work runs controlled language-model pretraining while varying only the tokenizer’s training data mixture, pretokenization strategy, and training algorithm, keeping other factors constant.

Which tokenizer properties does TokEval focus on?

TokEval evaluates linguistically and structurally meaningful properties, including UTF-8 character boundary integrity and digit place-value boundary alignment for mathematical content.

Sep 15, 2026

Uploaded by Aria