[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123730-en":3,"doc-seo-123730-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123730,16904993612988,"Olivia Brown","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Communication-Efficient On-Device Machine Learning - Federated Distillation and Augmentation under Non-IID Private Data","On-device machine learning leverages large volumes of user-generated private data, but inter-device communication must be minimized to make training practical. The document presents federated distillation (FD), which reduces communication payload size compared with federated learning, especially for large models. It addresses performance loss from non-IID private data by introducing federated augmentation (FAug), where devices train a generative model to synthesize local data toward an IID distribution. Experiments show FD with FAug cuts communication overhead by about 26x while retaining 95–98% test accuracy.","arXiv : 1811 . 11479v1 [ cs .LG] 28 Nov 2018  \nCommunication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data  \nEunjeong Jeong, Seungeun Oh, Hyesung Kim, Seong-Lyun Kim  \nYonsei University {ejjeong,seoh,hskim,slkim}@[ramo.yonsei.ac.kr](ramo.yonsei.ac.kr)  \nJihong Park, Mehdi Bennis  \nUniversity of Oulu {jihong.park,mehdi.bennis}@[oulu.fi](oulu.fi)  \nAbstract  \nOn-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.  \n1 Introduction  \nBig training dataset kickstarted the modern machine learning (ML) revolution. On-device ML can fuel the next evolution by allowing access to a huge volume of private data samples that are generated and owned by mobile devices [1, 2] . Preserving data privacy facilitates such access, in a way that a global model is collectively trained not by directly sharing private data but by exchanging the local model parameters of devices, as exemplified by federated learning (FL) [3–9] .  \nUnfortunately, in FL, performing the training process at each device side entails communication overhead being proportional to model sizes, forbidding the use of large-sized models. Furthermore, a user-generated training dataset is likely to be non-IID across devices. Compared to its IID-data counterpart, it decreases the prediction accuracy by up to 11% for MNIST and 51% for CIFAR-10 under FL [9] . The reduced accuracy can partly be restored by exchanging data samples, which may however induce an excessive amount of communication overhead and privacy leakage.  \nOn this account we seek for a communication-efficient on-device ML approach under non-IID private data. For communication efficiency, we propose federated distillation (FD), a distributed online knowledge distillation method whose communication payload size depends not on the model size but on the output dimension. Prior to operating federated distillation, we rectify the non-IID training dataset via federated augmentation (FAug), a data augmentation scheme using a generative adversarial network (GAN) that is collectively trained under the trade-off between privacy leakage and communication overhead. The trained GAN empowers each device to locally reproduce the data samples of all devices, so as to make the training dataset become IID.  \n32nd Conference on Neural Information Processing Systems (NIPS 2018), 2nd Workshop on Machine Learning on the Phone and other Consumer Devices (MLPCD 2), Montréal, Canada.  \n(a) FD with 2 devices and 2 labels.  \n(b) FAug with 3 target and 3 redundant MNIST labels.  \nFigure 1: Schematic overview of federated distillation (FD) and federated augmentation (FAug) .  \n2 Federated distillation  \nTraditional distributed training algorithms exchange local model parameters every epoch. It gives rise to significant communication overhead in on-device ML where mobile devices are wirelessly interconnected. FL reduces the communication cost by exchanging model parameters at intervals [3–9] . On top of such periodic communication, the proposed FD exchanges not the model parameters but the model output, allowing on-device ML to","cbCaimPRNkM9093k","https://ap.wps.com/l/cbCaimPRNkM9093k","pdf",3341104,1,6,"English","en",105,"# Abstract\n# Introduction\n# Federated distillation\n## Communication cost reduction via output exchange\n## Online knowledge distillation and co-distillation setup\n## Making distillation communication-efficient under non-IID data\n# Federated augmentation","[{\"question\":\"What problem does the document address in federated on-device machine learning?\",\"answer\":\"It targets the high communication overhead of federated training on devices and the performance degradation caused by non-IID user data across devices.\"},{\"question\":\"How does federated distillation (FD) reduce communication compared with federated learning (FL)?\",\"answer\":\"FD exchanges model outputs (logit information) rather than model parameters, so the communication payload depends on output dimension instead of model size.\"},{\"question\":\"How does federated augmentation (FAug) address non-IID private data?\",\"answer\":\"FAug trains a generative model jointly so each device can augment its local data toward an IID-like distribution, improving test accuracy under non-IID conditions.\"}]","Communication-Efficient On-Device Machine Learning - Federated Distillation and Augmentation under Non-IID Private Data | PDF",1785818226,15,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"communication-efficient-on-device-machine-learning-federated-distillation-and-augmentation-under-non-iid-private-data","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/communication-efficient-on-device-machine-learning-federated-distillation-and-augmentation-under-non-iid-private-data/123730/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the document address in federated on-device machine learning?","Question",{"text":75,"@type":76},"It targets the high communication overhead of federated training on devices and the performance degradation caused by non-IID user data across devices.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does federated distillation (FD) reduce communication compared with federated learning (FL)?",{"text":80,"@type":76},"FD exchanges model outputs (logit information) rather than model parameters, so the communication payload depends on output dimension instead of model size.",{"name":82,"@type":73,"acceptedAnswer":83},"How does federated augmentation (FAug) address non-IID private data?",{"text":84,"@type":76},"FAug trains a generative model jointly so each device can augment its local data toward an IID-like distribution, improving test accuracy under non-IID conditions.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]