[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-124614-en":3,"doc-seo-124614-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},124614,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697",8,"Research & Report","Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning - Research paper summary","The work presents new differentially private (DP) gradient-based machine learning mechanisms designed for multiple passes (epochs) over data. It formalizes DP mechanisms for adaptive streams under multiple participations and extends online matrix factorization DP beyond the single-epoch setting. The study develops the theory for sensitivity computation and efficient optimal matrix construction, and addresses high computational cost by proposing a Fourier-transform-based mechanism with only minor utility loss. Experiments on image classification and language modeling show improvements over prior methods, including DP-SGD.","Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning  \nChristopher A. Choquette-Choo 1 H. Brendan McMahan 1 Keith Rush 1 Abhradeep Thakurta 1  \narXiv :2211 .06530v2 [ cs .LG] 8 Jun 2023  \nAbstract  \nWe introduce new differentially private (DP) mechanisms for gradient-based machine learning (ML) with multiple passes (epochs) over adataset, substantially improving the achievable privacy-utility-computation tradeoffs. We formalize the problem of DP mechanisms for adaptive streams with multiple participations and introduce a non-trivial extension of online matrix factorization DP mechanisms to our setting.  \nThis includes establishing the necessary theory for sensitivity calculations and efficient computation of optimal matrices. For some applications like >10 , 000 SGD steps, applying these optimal techniques becomes computationally expensive.  \nWe thus design an efficient Fourier-transformbased mechanism with only a minor utility loss.  \nExtensive empirical evaluation on both examplelevel DP for image classification and user-level DP for language modeling demonstrate substantial improvements over all previous methods, including the widely-used DP-SGD. Though our primary application is to ML, our main DP results are applicable to arbitrary linear queries and hence may have much broader applicability.  \n1. Introduction  \nDifferentially private stochastic gradient descent (DPSGD) is the de facto standard algorithm for DP machine learning (ML) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016a) . However, obtaining state-of-theart privacy-utility tradeoffs critically requires use of privacy amplification techniques like shuffling (Erlingssonet al., 2019; Feldman et al., 2022) or (Poisson) subsampling (Bassily et al., 2014; Zhu & Wang, 2019; Wanget al., 2019) . These in turn require strong assumptions on  \n1 Google Research. Correspondence to:  \n\u003C{cchoquette,krush,mcmahan,[athakurta](athakurta}@google.com)[}](athakurta}@google.com)[@google.com](athakurta}@google.com)>.  \nProceedings of the 40 th International Conference on Machine Learning, Honolulu, Hawaii, USA. PMLR 202, 2023 . Copyright 2023 by the author(s) .  \nthe manner in which data is processed that often do not hold under the processing performed by centralized ML pipelines, and are particularly challenging in cross-device federated learning (Kairouz et al., 2021) .  \nKairouz et al. (2021) recently proposed the DP-FTRL framework that avoids reliance on amplification by sampling, instead leveraging DP streaming of prefix sums (Dwork et al., 2010; Chan et al., 2011; Honaker, 2015) . DP-FTRL can match (or outperform) DP-SGD in privacy-utility tradeoffs. This algorithm enabled McMahan & Thakurta (2022) to train the first known provably DP ML model on user data in a production setting.  \nSeveral works have since focused on this primitive asan instantiation of the streaming matrix mechanism (see Eq. (1)) (Henzinger et al., 2022; Fichtenberger et al., 2022; Denisov et al., 2022); in particular, Denisov et al. (2022) showed that leveraging the flexibility inherent in this formulation to design optimal matrices led to significant empirical improvements, though their work was restricted to the single-epoch setting.  \nOur Contributions This single-epoch restriction is unnatural from the perspective of modern ML, where many passes over the training data are common. We extend matrix factorization mechanisms for ML to the multi-epoch setting by tackling several intertwined problems. This enables state-of-the-art mechanisms for DP-ML, with potentially broader applicability to, e.g., online PCA, marginal estimation, and top-k selection, as discussed in Denisov et al. (2022) .  \n1) We provide a framework for computing the sensitivity of matrix mechanisms under general participation schemas with vector contributions: these are essential to ML applications where we wish to privatize high-dimensional models. However, the efficient computation of optimal matrix fac","cbCaihuxc96WEKFu","https://ap.wps.com/l/cbCaihuxc96WEKFu","pdf",6879096,1,40,"English","en",105,"# Introduction\n## Problem setting and motivation\n## Contributions\n## Sensitivity and optimal matrix computation\n## Dual formulation for multi-participation constraints\n## Computational tradeoffs and FFT mechanism\n## Empirical evaluation","[{\"question\":\"What problem does the document address in private machine learning?\",\"answer\":\"It addresses how to design differentially private, gradient-based learning mechanisms when the algorithm makes multiple passes (epochs) over the data, improving the privacy-utility-computation tradeoffs.\"},{\"question\":\"How does the approach handle multiple participations?\",\"answer\":\"It formalizes a DP setting for adaptive streams with multiple participations and establishes theory for sensitivity calculations and efficient computation of optimal matrix factorizations under those constraints.\"},{\"question\":\"Why is an efficient Fourier-transform-based mechanism introduced?\",\"answer\":\"When many optimization steps are required, computing optimal matrix factorizations becomes computationally expensive, so the document proposes a Fourier-transform-based mechanism that keeps utility loss small.\"}]","Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning - Research paper summary | PDF",1785893328,101,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"multi-epoch-matrix-factorization-mechanisms-for-private-machine-learning-research-paper-summary","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/multi-epoch-matrix-factorization-mechanisms-for-private-machine-learning-research-paper-summary/124614/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the document address in private machine learning?","Question",{"text":75,"@type":76},"It addresses how to design differentially private, gradient-based learning mechanisms when the algorithm makes multiple passes (epochs) over the data, improving the privacy-utility-computation tradeoffs.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the approach handle multiple participations?",{"text":80,"@type":76},"It formalizes a DP setting for adaptive streams with multiple participations and establishes theory for sensitivity calculations and efficient computation of optimal matrix factorizations under those constraints.",{"name":82,"@type":73,"acceptedAnswer":83},"Why is an efficient Fourier-transform-based mechanism introduced?",{"text":84,"@type":76},"When many optimization steps are required, computing optimal matrix factorizations becomes computationally expensive, so the document proposes a Fourier-transform-based mechanism that keeps utility loss small.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":21,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]