[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273359-105":3,"detail-sidebar-cat-0-en-105":73,"doc-detail-273359-en":123},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":66,"head_meta":68,"extra_data":70,"updated_unix":72},105,"en","when-better-teachers-dont-make-better-students-revisiting-knowledge-distillation-for-clip-models-in-vqa","When Better Teachers Don’t Make Better Students - Revisiting Knowledge Distillation for CLIP Models in VQA","","Vision-language models (VLMs) deliver strong results on multimodal tasks but face high computation costs that limit practical deployment. Knowledge distillation (KD) is a common strategy for producing lightweight yet competitive models, with evidence across both language and vision. For VLMs and CLIP-style models, however, distillation research remains limited, especially in teacher scaling and in broad evaluations beyond classification or retrieval. This work performs a systematic distillation study across CLIP-style teacher models and shows that stronger teachers do not reliably produce better VQA students.",{"@graph":14,"@context":65},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/when-better-teachers-dont-make-better-students-revisiting-knowledge-distillation-for-clip-models-in-vqa/273359/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/when-better-teachers-dont-make-better-students-revisiting-knowledge-distillation-for-clip-models-in-vqa/273359.png","ImageObject",300,407,{"name":42,"@type":43},"Logic","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-23","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",14,{"@type":57,"mainEntity":58},"FAQPage",[59],{"name":60,"@type":61,"acceptedAnswer":62},"How is the research conducted in this paper?","Question",{"text":63,"@type":64},"The study adopts the CLIPKD framework and evaluates distillation across multiple teacher-student configurations, analyzing student-teacher interaction dynamics to explain scalability failures.","Answer","https://schema.org",{"og:url":32,"og:type":67,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":69,"canonical":32},"index,follow",{"doc_id":71,"site_id":7},273359,1789471049,{"code":4,"msg":74,"data":75},"success",[76,80,84,88,93,98,103,107,112,115,119],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":77,"show_sort_weight":78,"slug":79},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":81,"show_sort_weight":82,"slug":83},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Exam",70,"exam",{"id":89,"doc_module":4,"doc_module_name":25,"category_name":90,"show_sort_weight":91,"slug":92},5,"Comic",60,"comic",{"id":94,"doc_module":4,"doc_module_name":25,"category_name":95,"show_sort_weight":96,"slug":97},6,"Technology",50,"technology",{"id":99,"doc_module":4,"doc_module_name":25,"category_name":100,"show_sort_weight":101,"slug":102},7,"Healthcare",40,"healthcare",{"id":104,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":105,"slug":106},8,30,"research-report",{"id":108,"doc_module":4,"doc_module_name":25,"category_name":109,"show_sort_weight":110,"slug":111},9,"Religion & Spirituality",20,"religion-spirituality",{"id":110,"doc_module":4,"doc_module_name":25,"category_name":113,"show_sort_weight":110,"slug":114},"World Cup","world-cup",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":116,"slug":118},10,"Lifestyle","lifestyle",{"id":120,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":89,"slug":122},19,"General","general",{"code":4,"msg":74,"data":124},{"doc_id":71,"user_id":125,"nickname":42,"user_avatar":126,"doc_module":4,"category_id":104,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":127,"file_id":128,"file_url":129,"file_type":130,"file_size":131,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":116,"language":132,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":133,"faqs":134,"seo_title":135,"seo_description":12,"update_tm":72,"read_time":136,"preview_page_count":4},1099513958762,"https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1784791008015729253","When Better Teachers Don’t Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA  \nPume Tuchinda VISTEC Thailand  \nPatomporn Payoungkhamdee  \nVISTEC Thailand  \nParinthapat Pengpun  \nCarnegie Mellon University United State  \nSarana Nutanong VISTEC Thailand  \nRomrawin Chumpu VISTEC Thailand  \nPeerat Limkonchotiwat✉ AI Singapore Singapore [peerat@aisingapore.org](peerat@aisingapore.org)  \narXiv : 25 1 1 . 17886v2 [cs .CV] 2 1 Aug 2026  \nAbstract  \nVision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to smallscale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models.  \nCCS Concepts  \n• Computing methodologies → Image representations; Computer vision tasks.  \nKeywords  \nVision-language models, knowledge distillation, efficient multimodal models  \nACM Reference Format:  \nPume Tuchinda, Parinthapat Pengpun, Romrawin Chumpu, Patomporn Payoungkhamdee, Sarana Nutanong, and Peerat Limkonchotiwat. 2026. When Better Teachers Don’t Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA. In 4th International Workshop on Multimodal, Generative and Responsible Affective Computing (MRAC’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. [https://doi.org/10.1145/3840474.3840553](https://doi.org/10.1145/3840474.3840553)  \nThis work is licensed under a Creative Commons Attribution 4 .0 International License. MRAC’26, Rio de Janeiro, Brazil  \n© 2026 Copyright held by the owner/author(s) .  \nACM ISBN 979-8-4007-2928-7/2026/11  \n[https://doi.org/10.1145/3840474.3840553](https://doi.org/10.1145/3840474.3840553)  \n1 Introduction  \nVision language models (VLMs) have demonstrated remarkable capabilities across diverse multimodal tasks, from image captioning to visual question answering (VQA) [1, 35, 52] . However, the substantial computational demands of these large-scale models limit their practicality for deployment and accessibility. Creating efficient VLMs that maintain competitive performance while dramatically reducing computational costs remains a critical challenge [31] .  \nRecent advances in language, reasoning, and vision demonstrate that knowledge distillation (KD) is among the most effective approaches for producing models that are both efficient and high-performing. For instance, DistilBERT retains 97% of BERT Base’s performance while reducing the model size by 40%[30], and MiniLM showed that students distilled from stronger teachers consistently outperform those distilled from weaker ones [39] . This pattern holds across multiple works: as teacher models improve, their distilled students also benefit, often without substantial modifications to the distillation methodology [16] . Similar findings extend to vision models; for example, MobileSAM [45] matches the performance of the original Segment Anything Model (SAM) while being 60 times smaller. More recently, works such as TinyLLaMA [48] and DeepSeek-R1 [10] suggest that directly distilling from powerful teachers can even s","cbCaikGXTa8S4UP8","https://ap.wps.com/l/cbCaikGXTa8S4UP8","pdf",730603,"English","# Introduction\n## Knowledge distillation and efficiency\n## Gap in VLM/CLIP distillation\n## Goal and research question\n## Key findings\n## Contributions","[{\"question\":\"How is the research conducted in this paper?\",\"answer\":\"The study adopts the CLIPKD framework and evaluates distillation across multiple teacher-student configurations, analyzing student-teacher interaction dynamics to explain scalability failures.\"}]","When Better Teachers Don’t Make Better Students - Revisiting Knowledge Distillation for CLIP Models in VQA | PDF",25]