[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82475-en":3,"doc-seo-82475-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82475,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","TallyTrain: Communication-Efficient Federated Distillation","Federated learning is limited by two orthogonal communication ceilings: model size, which restricts how often parameter-averaging can be performed, and class count, which makes per-probe soft-label distillation costly for large vocabularies. TallyTrain collapses the class-count cost by transmitting only each peer’s argmax class index per probe, yielding an O(log2 C)-bit signal. Under non-IID training, hard-label voting can outperform soft distillation because disagreement suppresses noise from under-trained peers. Benchmarks show matches or gains with up to three orders of magnitude less communication.","arXiv :2607 .00173v1 [ cs .LG] 30 Jun 2026  \nTallyTrain: Communication-Efficient Federated Distillation  \nRadhakrishna Achanta [rachanta@cisco.com](rachanta@cisco.com)  \nWill Reed [wilreed@cisco.com](wilreed@cisco.com)  \n[Cisco Systems Inc.](Cisco Systems Inc.)  \nAbstract  \nFederated learning is bandwidth-bound on two orthogonal axes: model size, which limitshow often parameter-averaging methods can afford to merge, and class count, which makes per-probe soft-label distillation prohibitive at large vocabularies. Both ceilings tighten as modern systems scale. We collapse the class-count axis to ⌈log2 C⌉ bits per probe by transmitting only each peer’s arg max class index, where C is the number of output classes. The resulting protocol, TallyTrain, is not merely compressed: under non-IID training it can be preferable to soft-label distillation, because under-trained peers are confidently wrong and majority voting filters this noise where soft-label averaging amplifies it. Across standard benchmarks, TallyTrain matches or beats soft-label distillation at up to three orders of magnitude less communication. We also relax the model-size axis: we compose the cheap hard-label consensus with sparse parameter merges to obtain a bandwidth-bridge variant, which Pareto-dominates every tested operating point of the standard FedAvg, FedProx and FedDF baselines.  \nKeywords: federated learning; knowledge distillation; communication efficiency; hardlabel consensus; decentralized training; non-IID  \n1. Introduction  \nFederated learning (FL) trains a model from data shards held by many peers without centralizing the raw data. The two dominant paradigms differ in what they communicate. Parameter-space methods exchange model parameters (FedAvg McMahan et al.  \n(2017)) or per-round weight deltas that play the role of an outer-loop pseudo-gradient (DiLoCo Douillard et al. (2023)) . Both have per-round bandwidth Θ(|W|), where |W| is the number of model parameters, which is impractical for billion-parameter models on edge or mobile substrates.  \nFunction-space methods (FedMD Li and Wang (2019)) exchange per-example predictions on a shared public probe set. Bandwidth scales as Θ(C·|D pub|) per peer per round, where C is the number of output classes – independent of model size, but growing linearly with C, which becomes prohibitive for large-vocabulary tasks (BPE-tokenized language models routinely use C ∈ [2 ,048 , 50 ,000]) .  \nThese paradigms share a structural assumption: bandwidth is reduced along the frequency axis, by communicating less often, while the size of each message is treated as fixed (full weights, or a full C-dimensional softmax) . This paper takes the orthogonal size axis. We keep communication frequent but make each message tiny: only the arg max class index per probe, one byte for C ≤ 256, two bytes for C ≤ 65 ,536. Argmax voting acts as a noise filter – when peers agree, the consensus is reliable, and when they disagree, voting averages out individual error rather than amplifying it (which is what the soft-label expectation  \n© 2026 R. Achanta & W. Reed.  \nAchanta Reed  \ndoes when peers are simultaneously under-trained and confident) . We call the resulting protocol TallyTrain – each round, peers tally their argmax votes on the public probe set into a single consensus histogram and then train against it – and show that, combined with sparse parameter merges, it dominates the bandwidth–accuracy Pareto frontier of federated learning across two modalities and three class counts. Our contributions are as follows:  \n(1) A hard-label communication primitive. We introduce TallyTrain and argue that the voting histogram of N peers’ argmax predictions over a public probe set is a valid – and surprisingly powerful – consensus distribution for distillation (§4.1) .  \n(2) Hard labels match or beat soft labels across modalities. The arg max channel carries essentially all the signal that soft labels do, at a per-probe bandwidth ratio that grows li","cbCailmzt8RZ4b1d","https://ap.wps.com/l/cbCailmzt8RZ4b1d","pdf",760380,1,24,"English","en",105,"# Introduction\n## Communication bottlenecks in federated learning\n## TallyTrain overview and contributions\n# Related Work","[{\"question\":\"What communication bottlenecks does the paper address in federated learning?\",\"answer\":\"It addresses two orthogonal bandwidth limits: model-size constraints for parameter averaging and class-count growth that makes soft-label distillation prohibitive for large vocabularies.\"},{\"question\":\"How does TallyTrain reduce communication cost during distillation?\",\"answer\":\"Each peer sends only the argmax class index per probe, forming a consensus histogram that is then used for training, dramatically shrinking per-message size.\"},{\"question\":\"Why can hard-label voting outperform soft-label distillation under non-IID data?\",\"answer\":\"When peers are under-trained, confident wrong predictions create noise; majority voting filters this noise, whereas soft-label expectation can amplify errors during averaging.\"}]",1784180755,60,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"tallytrain-communication-efficient-federated-distillation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/tallytrain-communication-efficient-federated-distillation/82475/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What communication bottlenecks does the paper address in federated learning?","Question",{"text":74,"@type":75},"It addresses two orthogonal bandwidth limits: model-size constraints for parameter averaging and class-count growth that makes soft-label distillation prohibitive for large vocabularies.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does TallyTrain reduce communication cost during distillation?",{"text":79,"@type":75},"Each peer sends only the argmax class index per probe, forming a consensus histogram that is then used for training, dramatically shrinking per-message size.",{"name":81,"@type":72,"acceptedAnswer":82},"Why can hard-label voting outperform soft-label distillation under non-IID data?",{"text":83,"@type":75},"When peers are under-trained, confident wrong predictions create noise; majority voting filters this noise, whereas soft-label expectation can amplify errors during averaging.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,108,113,118,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":28,"slug":107},5,"Comic","comic",{"id":109,"doc_module":4,"doc_module_name":45,"category_name":110,"show_sort_weight":111,"slug":112},6,"Technology",50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":119,"slug":120},30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]