Punching Above Their Weight - Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks

Sep 21, 2026

Uploaded by Asher

Tiny Language Models (TLMs) are defined as models under roughly 3B parameters that can run on mainstream consumer devices. The study investigates how to adapt TLMs for verifiable multiple-choice tasks and compares three LoRA-based fine-tuning paradigms: label generation, gold only, and a discriminative classification head. Across multiple Qwen3 models (0.6B–8B) and benchmarks (HellaSwag, WinoGrande, PIQA, SciQ, ARC-C), classification-head fine-tuning improves accuracy (+2–3% at small scales) and remains competitive with large GPT-3/PaLM/GPT-4 settings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

Tiny Language Models

Memory and deployment constraints

Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks Bhavesh Sood Carnegie Mellon University [bsood@andrew.cmu.edu](bsood@andrew.cmu.edu) Jaromir Savelka Carnegie Mellon University [jsavelka@andrew.cmu.edu](jsavelka@andrew.cmu.edu) arXiv :2607 .0380 1v 1 [ cs .LG] 4 Jul 2026 Abstract We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head) on a unified setup across several Qwen3 models from 0 .6B to 8B and five benchmarks: HellaSwag, WinoGrande, PIQA, SciQ and ARC-C. Classification-head fine-tuning reliably outperforms label generation (+2–3%) at the 0 .6B and 1 .7B scales. Further, TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4 . The performance wereport for Qwen3-0.6B and Qwen3-1.7B are SOTA on HellaSwag, WinoGrande, and PIQA. 1 Introduction Running large language models (LLMs) on common consumer hardware has become increasingly important as the capabilities and ubiquity of LLMs rise at unprecedented pace. Yet, vast majority of published research focuses on LLMs that cannot be effectively run on such devices. Hence, we argue that there is a great need to coin a novel notion of a Tiny LM (TLM) that is motivated by contemporary practical constraints stemming from common consumer hardware. Because TLMs remain understudied their deployments are often sub-optimal with lower accuracy than what they can deliver. This study of fine-tuning paradigms and evaluation approaches for TLMs offers actionable guidance on how to adapt and use a TLM. The study is task-agnostic and only assumes a setting in which there is a finite set of candidate outputs for each data point with one of the candidates deemed as correct (e.g., classification, span annotation, or typical applications of LLM-as-a-judge) . We denote such tasks as verifiable multiple-choice. Our specific focus is on commonsense reasoning tasks as these would likely be the main TLMs use case on local devices. However, such tasks have been shown tobe challenging for LMs of small sizes (Li et al., 2025 ; Lu et al., 2024a) . This study makes the following contributions: 1. It defines and motivates the practical notion of Tiny Language Models (TLMs) . 2. It describes a fine-tuning approach based on discriminative classification head suitable for application of TLMs on multiple-choice tasks. 3. It is the first comparison of the three finetuning paradigms across five well-established benchmarks focused on TLMs, and the reported performance on most of them is SOTA. The main findings of this study are: a) In TLMs, the fine-tuning based on classification head offers significantly better performance than approaches based on fine-tuning for next-token (label) prediction. b) Modifying seemingly mundane details of inference can result in dramatic discrepancies in observed performance of a TLM. c) In larger models (> 3B), the gap is closed and there is no significant difference between the fine-tuning approaches. d) Additionally, the memorisation-intensive science benchmarks benefit less from the classification approach, but with no statistically significant loss either. Our findings suggest that the previously reported performance of TLMs may be simply an artifact of how they are typically fine-tuned and utilized (often sub-optimally) . This calls for careful rethinking of evaluation and progress in this space. While it Label-Gen SFT CLS-SFT UNICORN T5-11B (SFT) GPT-3 175B (few-shot) PaLM 540B (0-shot) GPT-4 (few-shot) Figure 1: Accuracy on three commonsense benchmarks for Qwen3 + LoRA models fine-tuned with Label-Gen SFT and Classification-

FAQs

What are Tiny Language Models (TLMs) in this paper?

TLMs are defined as models with parameters typically below or around 3 billion that can be effectively deployed on mainstream consumer devices, requiring less than about 8 GB of memory at BF16/FP16 precision.

Which fine-tuning paradigm performs best for verifiable multiple-choice tasks?

Fine-tuning with a discriminative classification head reliably outperforms label generation (+2–3%) at the 0.6B and 1.7B scales, and stays competitive with much larger models in evaluated setups.

What does the study compare across models and benchmarks?

It evaluates three LoRA-based fine-tuning approaches on a unified setup across several Qwen3 models (0.6B to 8B) and five multiple-choice reasoning benchmarks: HellaSwag, WinoGrande, PIQA, SciQ, and ARC-C.

Sep 21, 2026

Uploaded by Asher