[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-279640-105":3,"detail-sidebar-cat-1-en-105":81,"doc-detail-279640-en":127},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","a-new-framework-of-collaborative-learning-for-adaptive-metric-distillation","A New Framework of Collaborative Learning for Adaptive Metric Distillation","","This article proposes a new adaptive metric distillation approach designed to improve student networks’ backbone features and deliver stronger classification and retrieval performance. Prior knowledge distillation methods often transfer information at logits or feature-structure levels while overlooking excessive sample relations within feature space. The proposed collaborative adaptive metric distillation (CAMD) optimizes key pair relationships via hard mining, applies teacher embedding relations as supervision for adaptive metric distillation, and uses collaborative aggregation. Experiments establish state-of-the-art results across settings.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/template/","Template",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/template/presentations/","Presentations",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/template/a-new-framework-of-collaborative-learning-for-adaptive-metric-distillation/279640/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/a-new-framework-of-collaborative-learning-for-adaptive-metric-distillation/279640.png","ImageObject",442,249,{"name":42,"@type":43},"Hazel","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-20","2026-09-16",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",6,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does CAMD address in standard knowledge distillation?","Question",{"text":63,"@type":64},"Standard KD often transfers knowledge via logits or feature structures while ignoring rich sample relations in the feature space, which can limit performance—especially for retrieval tasks.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does CAMD optimize distillation during training?",{"text":68,"@type":64},"CAMD introduces a hard mining strategy to focus optimization on relationships between key pairs.",{"name":70,"@type":61,"acceptedAnswer":71},"How is student embedding quality improved for retrieval?",{"text":72,"@type":64},"CAMD performs adaptive metric distillation by using relation patterns in teacher embeddings as explicit supervision for student feature embeddings.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},279640,1791739180,{"code":4,"msg":82,"data":83},"success",[84,88,93,98,103,108,113,118,123],{"id":85,"doc_module":22,"doc_module_name":25,"category_name":29,"show_sort_weight":86,"slug":87},11,90,"presentations",{"id":89,"doc_module":22,"doc_module_name":25,"category_name":90,"show_sort_weight":91,"slug":92},12,"Resumes",80,"resumes",{"id":94,"doc_module":22,"doc_module_name":25,"category_name":95,"show_sort_weight":96,"slug":97},14,"Invoices",70,"invoices",{"id":99,"doc_module":22,"doc_module_name":25,"category_name":100,"show_sort_weight":101,"slug":102},15,"Posters",60,"posters",{"id":104,"doc_module":22,"doc_module_name":25,"category_name":105,"show_sort_weight":106,"slug":107},16,"Social Media",50,"social-media",{"id":109,"doc_module":22,"doc_module_name":25,"category_name":110,"show_sort_weight":111,"slug":112},17,"Forms",40,"forms",{"id":114,"doc_module":22,"doc_module_name":25,"category_name":115,"show_sort_weight":116,"slug":117},18,"Letters",30,"letters",{"id":119,"doc_module":22,"doc_module_name":25,"category_name":120,"show_sort_weight":121,"slug":122},21,"Paper Templates",5,"papers-templates",{"id":124,"doc_module":22,"doc_module_name":25,"category_name":125,"show_sort_weight":4,"slug":126},158,"General","general-158",{"code":4,"msg":82,"data":128},{"doc_id":79,"user_id":129,"nickname":42,"user_avatar":130,"doc_module":22,"category_id":85,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":131,"file_id":132,"file_url":133,"file_type":134,"file_size":135,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":89,"language":136,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":12,"update_tm":140,"read_time":33},137441390410,"https://ap-avatar.wpscdn.com/avatar/2000252f4ab5702993?_k=1776741390130283984","A New Framework of Collaborative Learning for Adaptive Metric Distillation  \nHao Liu, Mang Ye , Senior Member, IEEE, Yan Wang, Sanyuan Zhao, Ping Li , Member, IEEE ,  \nand Jianbing Shen , Senior Member, IEEE  \nAbstract—This article presents a new adaptive metric distillation approach that can significantly improve the student networks’ backbone features, along with better classification results. Previous knowledge distillation (KD) methods usually focus on transferring the knowledge across the classifier logits or feature structure, ignoring the excessive sample relations in the feature space. We demonstrated that such a design greatly limits performance, especially for the retrieval task. The proposed collaborative adaptive metric distillation (CAMD) has three main advantages: 1) the optimization focuses on optimizing the relationship between key pairs by introducing the hard mining strategy into the distillation framework; 2) it provides an adaptive metric distillation that can explicitly optimize the student feature embeddings by applying the relation in the teacher embeddings as supervision; and 3) it employs a collaborative scheme for effective knowledge aggregation. Extensive experiments demonstrated that our approach sets a new state-of-the-art in both the classification and retrieval tasks, outperforming other cutting-edge distillers under various settings.  \nIndex Terms—Collaborative learning, deep neural networks, knowledge distillation (KD), model compression.  \nI. INTRODUCTION  \nDURING the last few years, deep convolution neural  \nnetworks (CNNs) have achieved many successes in a variety of applications such as computer vision, natural language processing, and reinforcement learning. Although largescale deep models have achieved overwhelming successes, they often require considerable computational and memory consumption, making it a huge challenge to deploy them on mobile devices with limited resources. There are many  \nManuscript received 20 October 2021; revised 25 April 2022 and 19 August 2022; accepted 24 October 2022 . Date of publication 14 February 2023; date of current version 4 June 2024 . This work was supported in part by the FDCT Grant SKL-IOTSC(UM)-2021-2023; in part by the MYRGCRG2022-00013-IOTSC Grant and the SRG2022-00023-IOTSC Grant; in part by the National Natural Science Foundation of China under Grant 61902027, Grant 62176188, and Grant 62066021; and in part by the Key Research and Development Program of Hubei Province under Grant 2021BAA187 .(Corresponding authors: Sanyuan Zhao; Jianbing Shen.)  \nHao Liu, Yan Wang, and Sanyuan Zhao are with the School of Computer Science, Beijing Institute of Technology, Beijing 100081, China (e-mail: [zhaosanyuan@bit.edu.cn](zhaosanyuan@bit.edu.cn)).  \nMang Ye is with the Hubei Luojia Laboratory and the School of Computer Science, Wuhan University, Wuhan 430072, China (e-mail: [mangye16@gmail.com](mangye16@gmail.com)).  \nPing Li is with the Department of Computing and the School of Design, The Hong Kong Polytechnic University, Hong Kong (e-mail: [p.li@polyu.edu.hk](p.li@polyu.edu.hk)). Jianbing Shen is with the State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of  \nMacau, Macau, China (e-mail: [shenjianbingcg@gmail.com](shenjianbingcg@gmail.com)). Digital Object Identifier 10.1109/TNNLS.2022.3226569  \nefficient deep neural networks training techniques including designing efficient building blocks for deep models [21],[25], [31], [49], network pruning [24], quantization [42], and knowledge distillation (KD) [20], [30] . The focus of this article is KD, which has become an increasingly essential topic as it is applicable to almost all network architectures and can be easily combined with other strategies.  \nKD transfers the learned knowledge from a teacher model toa student model, that is, it aligns the classification prediction distributions between the two models [20] . Compared with the one-hot label","cbCaimiisSjjwHIZ","https://ap.wps.com/l/cbCaimiisSjjwHIZ","pdf",2367960,"English","# Introduction\n## Background on deep neural networks and deployment constraints\n## Knowledge distillation basics and limitations\n## Motivation from feature-space relations and retrieval tasks\n# Proposed approach\n## CAMD advantages and optimization strategy\n## Adaptive metric distillation supervision\n## Collaborative knowledge aggregation","[{\"question\":\"What problem does CAMD address in standard knowledge distillation?\",\"answer\":\"Standard KD often transfers knowledge via logits or feature structures while ignoring rich sample relations in the feature space, which can limit performance—especially for retrieval tasks.\"},{\"question\":\"How does CAMD optimize distillation during training?\",\"answer\":\"CAMD introduces a hard mining strategy to focus optimization on relationships between key pairs.\"},{\"question\":\"How is student embedding quality improved for retrieval?\",\"answer\":\"CAMD performs adaptive metric distillation by using relation patterns in teacher embeddings as explicit supervision for student feature embeddings.\"}]","A New Framework of Collaborative Learning for Adaptive Metric Distillation | PDF",1789524749]