The Illusion of Generalization in Tabular Language Models - arXiv:2602.04031v2 - systematic reevaluation and findings

Aug 21, 2026

Tabular Language Models (TLMs) are often claimed to generalize strongly for tabular prediction, yet evidence may be distorted by evaluation design. This study systematically reevaluates Tabula-8B using 165 datasets from the UniPredict benchmark and reports three key results: weak lift for binary/categorical tasks, reliance on quartile classification for aggregate gains, widespread train-test contamination and task-level leakage, and substantial recovery under instruction-tuning without tabular exposure, narrowed by format familiarity. The results indicate generalization artifacts and motivate stronger TLM evaluation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

AccessFree to download

Abstract

Introduction

Tabular data properties and inductive biases

Prior work and the central question

Empirical verification approach

Contributions and key findings

The Illusion of Generalization in Tabular Language Models Aditya Gorla 1 Ratish Puduppully 2 arXiv :2602 .04031v2 [ cs .LG] 29 May 2026 Abstract Tabular Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic reevaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPredict benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular reasoning. We conclude with recommendations for strengthening TLM evaluation.1 1. Introduction Large Language Models (LLMs) have transformed natural language processing (Zhao et al., 2025 ; Minaee et al., 2025), and their success has inspired a new wave of methods for tabular data (Fang et al., 2024 ; Gardner et al., 2024 ; Hegselmann et al., 2023) . Beyond naive approaches such as text serialization (Dinh et al., 2022 ; Lee et al., 2025), a new class of language models specifically (pre-)trained or fine-tuned for tabular domain have emerged (Fang et al., 2024 ; Hegselmann et al., 2023 ; Gardner et al., 2024 ; Wang et al., 2023 ; Sun et al., 2024) . We refer to these as Tabular Language 1University of California, Los Angeles, USA 2IT University of Copenhagen, Denmark. Correspondence to: Aditya Gorla <[adityagorla@ucla.edu](adityagorla@ucla.edu) >, Ratish Puduppully <[rapu@itu.dk](rapu@itu.dk)>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . 1Code and artifacts are available at [https://github](https://github) . com/ratishsp/tlm-illusion. Models (TLMs) . The putative hypothesis behind TLMs mirrors that of LLMs and Vision-Language Models (VLMs): with sufficient scale, these models may learn to generalize across the structure, invariances, and patterns inherent in tabular data, enabling zero-and few-shot prediction, imputation, and synthetic generation across binary, categorical, and continuous data types (Hegselmann et al., 2023 ; Wanget al., 2023 ; Gorla et al., 2025 ; Borisov et al., 2023) . Tabular data, however, possesses three properties that jointly distinguish it from text and images (Fang et al., 2024 ; Borisov et al., 2024 ; Grinsztajn et al., 2022) . First, tabular data is row-permutation invariant: the ordering of samples carries no semantic meaning. Second, and more critically, tabular data is column-permutation invariant. Unlike text (which has sequential structure) or images (which have spatial locality), features in a table have no a priori or universal ordering2. Third, tabular data is heterogeneous, spanning binary, categorical, and continuous types. This is in stark contrast to text (discrete tokens from a fixed vocabulary) or images (bounded pixel intensities) . One can reasonably argue these properties explain why gradient-boosted trees (GBTs) and Prior-Fitted Networks (PFNs) have excelled on tabular tasks (Grinsztajn et al., 2022 ; Shwartz-Ziv & Armon, 2022 ; Hollmann et al., 2023 ; 2025) . GBTs naturally handle irregular, discontinuous patterns without assuming feature relationships (Chen & Guestrin, 2016 ; Grinsztajn et al., 2022), while PFNs encode inductive biases through synthetic (heterogeneous) data priors and leverage transformer architectures without fixed positional encodings to r

FAQs

What question does the paper investigate about tabular language models?

It asks whether TLMs truly generalize to tabular data, and if so, what mechanism enables that generalization.

What three main findings does the study report from reevaluating Tabula-8B?

The paper finds near-zero gains for binary/categorical tasks with aggregate performance driven by quartile tasks, pervasive contamination including train-test overlap and task-level leakage, and that instruction-tuning without tabular exposure recovers most standard performance while format familiarity closes much of the remaining gap.

How do the authors interpret the claimed generalization of TLMs?

They conclude that the reported generalization likely reflects evaluation artifacts rather than learned tabular reasoning, and they recommend improving TLM evaluation practices.

Aug 21, 2026

Upload by 8796093062539