SQuaD-SQL Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation

Jul 16, 2026

Text-to-SQL enables natural-language access to structured databases, but deploying large language models is hindered by high compute needs in resource-constrained environments. SQuaD-SQL (Small-Qualified and Distilled for SQL) trains small language models to approach LLM-level performance using knowledge distillation and synthetic data generation. The method uses LLM-based structured data creation with task prompts, parameter-efficient fine-tuning (e.g., LoRA) on a single consumer GPU, and domain-adaptive synthetic data to improve targeted transfer. Results on WikiSQL report 86.9% execution accuracy on test, with faster inference and lower memory usage.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Introduction

Teacher–student learning framework

Efficient training and domain adaptation

Experimental evaluation

SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation Wangyu Wu 1,5 , Xiaojian Lin4 , Rong Fu2 , Zaiyang Yu3 , Xuhang Chen7 , Wenjun Yu6 , Zhenhong Chen5∗ 1The University of Liverpool 2University of Macau 3University of Chinese Academy of Sciences 4Tsinghua University 5 Microsoft 6 Shanghai University of International Business and Economics 7 Huizhou University arXiv :2607 .08 16 1v 1 [ cs .CL] 9 Jul 2026 Abstract—Text-to-SQL is a fundamental task in natural language processing that enables users to interact with structured databases using natural language. While large language models (LLMs) have demonstrated remarkable performance on this task, their substantial computational requirements hinder deployment in resource-constrained settings. In this paper, we introduce SQuaD-SQL (Small-Qualified and Distilled for SQL), a novel approach that empowers small language models (SLMs) to approach the performance of LLMs on the Text-to-SQL task while significantly improving efficiency through knowledge distillation and synthetic data generation. Our method comprises three key components: (1) LLM-based synthetic data generation, where structured knowledge is extracted from LLMs via carefully designed prompting strategies; (2) parameter-efficient fine-tuning, enabling full model training on a single consumergrade GPU; and (3) domain-adaptive fine-tuning, where domainspecific synthetic data further enhances performance in targeted domains. Experiments on the WikiSQL dataset demonstrate that SQuaD-SQL achieve execution accuracies of 86.9% in Test data, respectively, approaching the performance of LLMs, while offering faster inference and lower memory usage. These results suggest that, with proper training strategies, SLMs can serve as practical and efficient alternatives for Text-to-SQL applications in resource-limited environments. I. INTRODUCTION Text-to-SQL enables users to query structured databases using natural language, significantly lowering the barrier for database access. Early rule- or template-based systems lacked scalability, while neural models such as Seq2SQL [1] and SQLNet [2] improved performance via sketch-based decoding and reinforcement learning. Recent advances in natural language interfaces to structured data, particularly Text-to-SQL, have largely followed a scale-driven paradigm. Text-to-SQL enables users to query structured databases using natural language, significantly lowering the barrier for database access. Early rule-or template-based systems lacked scalability, while neural models such as Seq2SQL [1] and SQLNet [2] improved performance via sketch-based decoding and reinforcement learning. Subsequent work introduced more complex datasets such as Spider [3] and structure-aware architectures like RAT-SQL [4] . Pretrained tabular models, including TAPAS [5] and TaBERT [6], further advanced performance by incorporating schema and table representations. ∗ Corresponding authors: [daisychen37@foxmail.com](daisychen37@foxmail.com) Fig. 1. Overview of the proposed teacher–student learning framework. A large language model provides structured instructional signals that guide a compact student model to internalize Text-to-SQL reasoning patterns without relying on manual annotation. More recently, large language models (LLMs) [7],[8] have demonstrated strong reasoning capabilities on Text-to-SQL tasks, often achieving state-of-the-art performance. However, these gains come at the cost of substantial computational resources. LLM-based systems [9], [10], [11], [12] typically suffer from high inference latency, expensive deployment requirements involving multiple high-end GPUs, and resourceintensive training and fine-tuning procedures. Such demands diverge sharply from the efficiency observed in human learning and significantly limit the practical deployment of Textto-SQL systems in real-world, resource-constrained settings. This contrast raises a fundament

FAQs

What training strategy enables efficient learning for small language models?

SQuaD-SQL uses parameter-efficient fine-tuning such as LoRA, allowing training on a single consumer-grade GPU. It also applies domain-adaptive fine-tuning using domain-specific synthetic SQL examples to improve generalization.

Jul 16, 2026

Upload by Theodora