TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data

Jul 16, 2026

Upload by Liam

Tabular learning still relies heavily on gradient-boosted decision trees, yet deep tabular models face major barriers on large-scale data caused by memory demands, high dimensionality, and many target classes. TabLoRA introduces a parameter-efficient trainable neural ensemble that shares a common backbone across predictors while applying predictor-specific low-rank adaptations. This enables ensemble-style prediction without duplicating full parameters, improving practicality under constrained resources and maintaining much of the benefits of full ensembles.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages42

Reading Time1 h 46 min

Introduction

Related Work

Deep Learning for Tabular Data

MLP-based Architectures

arXiv :2607 . 10077v 1 [ cs .LG] 11 Jul 2026 TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data Jiaqi Luoa , Shixin Xub,∗ a School of Mathematical Sciences, Soochow University, No.1 Shizi Street, Suzhou, 215006, Jiangsu Province, China b Digital innovation research center, Duke Kunshan University, No. 8 Duke Avenue, Kunshan, 215000, Jiangsu Province, China Abstract Tabular learning is still dominated by gradient-boosted decision trees (GBDTs), while recent deep learning approaches have become increasingly competitive. However, applying deep tabular models to large-scale datasets remains challenging, as large sample sizes, high feature dimensionality, or many target classes can introduce substantial computational cost. We propose TabLoRA, a parameter-efficient trainable neural ensemble for largescale tabular learning. Instead of using fully independent ensemble backbones, TabLoRA shares a common backbone across predictors and introduces predictor-specific low-rank adaptations, enabling ensemble-style prediction without full parameter duplication. Across benchmarks, TabLoRA achieves a favorable balance between predictive performance and practical efficiency compared with GBDT methods and recent deep learning baselines under the same resource constraints. Memory analysis and ablation stud- ∗ Corresponding author Email addresses: [jqluo@suda.edu.cn](jqluo@suda.edu.cn) (Jiaqi Luo), [sx59@duke.edu](sx59@duke.edu) (Shixin Xu ) ies further show that the proposed design improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles. Keywords: Large-scale Tabular Data, Deep Learning, Low-Rank Adaptation, Ensemble, Parameter-Efficient 1. Introduction Tabular data remains a fundamental modality in real-world machine learning applications [1] . Historically, gradient-boosted decision trees (GBDTs)[2 , 3 , 4] have long dominated this domain due to their strong empirical performance and robustness [5 , 6 , 7] . Recently, however, tabular deep learning has become increasingly competitive. In particular, in-context learning methods such as TabPFN [8 , 9 , 10] demonstrate that deep learning models can match or even outperform GBDT on small-and medium-scale datasets when properly formulated. Despite this progress, large-scale tabular learning remains computationally challenging. Here, large-scale datasets refer to settings where scale mayarise from a large number of samples, high feature dimensionality, or many target classes. In these regimes, GBDTs become bottlenecked by tree construction, split search, and class-wise modeling costs, while deep learning models suffer from prohibitive memory and computational footprints during training, making these methods costly or even infeasible. Among deep learning approaches, MLP-based models provide a simple and computationally efficient foundation, making them naturally amenable to large-scale settings. Recent work further shows that incorporating classical machine learning structures [11 , 12 , 13] can significantly enhance their performance. Compared to more complex architectures, such designs offer practical efficiency and are easier to scale to large datasets. However, even within this design space, a fundamental challenge remains. Highly expressive MLP-based models often incur substantial memory overhead that grows with feature dimensionality and dataset size, while memoryefficient designs tend to suffer from limited representation capacity. This reveals an inherent trade-off between scalability and expressivity in largescale tabular learning, which remains insufficiently addressed by existing approaches. To overcome this limitation, we propose TabLoRA, a parameter-efficient ensemble framework that combines shared backbone learning with predictorspecific low-rank adaptations. Each predictor is modeled as a low-rank perturbation of shared weights, allowing the model to approximate the b

FAQs

What problem does TabLoRA address in large-scale tabular learning?

It targets the computational difficulty of applying deep tabular models to large-scale datasets, where memory costs, feature dimensionality, and class-wise modeling can make training expensive or infeasible.

How does TabLoRA achieve an ensemble effect without full parameter duplication?

TabLoRA shares a common backbone across predictors and adds predictor-specific low-rank adaptations, allowing specialization while avoiding linear growth in memory cost.

What evidence is provided that TabLoRA improves performance and efficiency?

Benchmarks report a favorable balance between predictive performance and practical efficiency versus GBDT and recent deep baselines under the same resource constraints, with memory analysis and ablations supporting the design.

Jul 16, 2026

Upload by Liam