When Better Teachers Don’t Make Better Students - Revisiting Knowledge Distillation for CLIP Models in VQA

Sep 15, 2026

Uploaded by Logic

Vision-language models (VLMs) deliver strong results on multimodal tasks but face high computation costs that limit practical deployment. Knowledge distillation (KD) is a common strategy for producing lightweight yet competitive models, with evidence across both language and vision. For VLMs and CLIP-style models, however, distillation research remains limited, especially in teacher scaling and in broad evaluations beyond classification or retrieval. This work performs a systematic distillation study across CLIP-style teacher models and shows that stronger teachers do not reliably produce better VQA students.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Knowledge distillation and efficiency

Gap in VLM/CLIP distillation

Goal and research question

Key findings

Contributions

When Better Teachers Don’t Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA Pume Tuchinda VISTEC Thailand Patomporn Payoungkhamdee VISTEC Thailand Parinthapat Pengpun Carnegie Mellon University United State Sarana Nutanong VISTEC Thailand Romrawin Chumpu VISTEC Thailand Peerat Limkonchotiwat✉ AI Singapore Singapore [peerat@aisingapore.org](peerat@aisingapore.org) arXiv : 25 1 1 . 17886v2 [cs .CV] 2 1 Aug 2026 Abstract Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to smallscale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models. CCS Concepts • Computing methodologies → Image representations; Computer vision tasks. Keywords Vision-language models, knowledge distillation, efficient multimodal models ACM Reference Format: Pume Tuchinda, Parinthapat Pengpun, Romrawin Chumpu, Patomporn Payoungkhamdee, Sarana Nutanong, and Peerat Limkonchotiwat. 2026. When Better Teachers Don’t Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA. In 4th International Workshop on Multimodal, Generative and Responsible Affective Computing (MRAC’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. [https://doi.org/10.1145/3840474.3840553](https://doi.org/10.1145/3840474.3840553) This work is licensed under a Creative Commons Attribution 4 .0 International License. MRAC’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2928-7/2026/11 [https://doi.org/10.1145/3840474.3840553](https://doi.org/10.1145/3840474.3840553) 1 Introduction Vision language models (VLMs) have demonstrated remarkable capabilities across diverse multimodal tasks, from image captioning to visual question answering (VQA) [1, 35, 52] . However, the substantial computational demands of these large-scale models limit their practicality for deployment and accessibility. Creating efficient VLMs that maintain competitive performance while dramatically reducing computational costs remains a critical challenge [31] . Recent advances in language, reasoning, and vision demonstrate that knowledge distillation (KD) is among the most effective approaches for producing models that are both efficient and high-performing. For instance, DistilBERT retains 97% of BERT Base’s performance while reducing the model size by 40%[30], and MiniLM showed that students distilled from stronger teachers consistently outperform those distilled from weaker ones [39] . This pattern holds across multiple works: as teacher models improve, their distilled students also benefit, often without substantial modifications to the distillation methodology [16] . Similar findings extend to vision models; for example, MobileSAM [45] matches the performance of the original Segment Anything Model (SAM) while being 60 times smaller. More recently, works such as TinyLLaMA [48] and DeepSeek-R1 [10] suggest that directly distilling from powerful teachers can even s

FAQs

How is the research conducted in this paper?

The study adopts the CLIPKD framework and evaluates distillation across multiple teacher-student configurations, analyzing student-teacher interaction dynamics to explain scalability failures.

Sep 15, 2026

Uploaded by Logic