[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117338-en":3,"doc-seo-117338-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117338,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","General Tail Bounds for Non-Smooth Stochastic Mirror Descent","This paper derives novel tail bounds for the optimization error of Stochastic Mirror Descent when minimizing convex, Lipschitz, non-smooth objectives using only noisy sub-gradient oracle access. The analysis extends existing light-tailed sub-Gaussian results to heavier-tailed noise regimes, covering the last iterate and the average iterate. Results are instantiated for two noise classes: exponential-tail (sub-Weibull) and polynomial-tail distributions. The theory avoids any requirement for an upper bound on the domain diameter and is validated by experiments comparing last-iterate versus averaged solutions.","POLITECNICO DI TORINO Repository ISTITUZIONALE  \nGeneral Tail Bounds for Non-Smooth Stochastic Mirror Descent  \nOriginal  \nGeneral Tail Bounds for Non-Smooth Stochastic Mirror Descent / Eldowa, Khaled; Paudice, Andrea. -238:(2024), pp. 3205-3213. (Intervento presentato al convegno The International Conference on Artificial Intelligence and Statistics tenutosi a Valencia (ESP) nel 2-4 May 2024) .  \nAvailability:  \nThis version is available at: 11583/2990222 since: 2024-07-02T10:38:59Z  \nPublisher:  \nProceedings of Machine Learning Research  \nPublished DOI:  \nTerms of use:  \nThis article is made available under terms and conditions as specified in the corresponding bibliographic description in the repository  \nPublisher copyright  \n(Article begins on next page)  \n27 March 2025  \nGeneral Tail Bounds for Non-Smooth Stochastic Mirror Descent  \nKhaled Eldowa Andrea Paudice  \nUniversità degli Studi di Milano, Milan, Italy  \nAbstract  \nIn this paper, we provide novel tail bounds on the optimization error of Stochastic Mirror Descent for convex and Lipschitz objectives.  \nOur analysis extends the existing tail bounds from the classical light-tailed Sub-Gaussian noise case to heavier-tailed noise regimes. We study the optimization error of the last iterate as well as the average of the iterates.  \nWe instantiate our results in two important cases: a class of noise with exponential tailsand one with polynomial tails. A remarkable feature of our results is that they do not require an upper bound on the diameter of the domain. Finally, we support our theory with illustrative experiments that compare the behavior of the average of the iterates with that of the last iterate in heavy-tailed noise regimes.  \n1 INTRODUCTION  \nStochastic Mirror Descent (SMD) and its more popular Euclidean counterpart Stochastic (sub-)Gradient Descent (SGD) are at the core of modern machine learning. For example, they are widely used for performing large-scale optimization tasks, as in the case of empirical (or regularized) risk minimization, and for minimizing the statistical risk in kernel methods. In this paper, we study the performance of SMD in the general problem of minimizing a (non-smooth) convex and Lipschitz function given only noisy oracle access to its (sub-)gradients. SGD was first introduced by Ermol’ev (1969), who studied the convergence of the iterates for convex Lipschitz objectives. Subsequent studies focused on deriving in-expectation bounds on the optimization error of the average of the iterates. Denoting with T the number of iterations, these bounds  \nProceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS) 2024, Valencia, Spain. PMLR: Volume 238 . Copyright 2024 by the author(s) .  \nare of the order of 1/ √T. In their seminal work, (Nemirovski et al. , 2009) introduced SMD as a non Euclidean generalization of SGD and showed that it enjoys the same 1/ √T bound. The shortcoming of inexpectation bounds is that they do not offer guaranteeson individual runs of the algorithm. This is especially limiting when multiple runs of the algorithm are not possible, as in large scale problems, or when the data arrives in a stream. Tail bounds offer stronger guarantees that apply to individual runs of the algorithms. For a fixed confidence level δ ∈ (0 , 1), a straightforward application of Markov’s inequality, gives a bound of the order 1/(δ √T ) that holds with probability at least 1 − δ . This bound is much worse than its inexpectation counterpart, even for moderately small δ . Tighter tail bounds with only an overhead of order p log(1/δ) have been obtained under a sub-Gaussian assumption on the noise (Liu et al. , 2023) .  \nRecent works (Zhang et al. , 2020) show that in some settings, the sub-Gaussian assumption is not appropriate, and the noise is better modelled by heavier tailed distributions. Most works studying tail bounds for SMD (SGD) under heavy-tailed noise consider the extreme cases where the noise","cbCailzy1f9uGcfo","https://ap.wps.com/l/cbCailzy1f9uGcfo","pdf",943867,1,36,"English","en",105,"# Abstract\n## Optimization error and tail guarantees\n## Noise regimes: sub-Weibull and polynomial tails\n## Last iterate vs averaged iterates\n## Domain diameter independence\n## Experiments and comparisons","[{\"question\":\"What problem does the paper address?\",\"answer\":\"It studies optimization performance of Stochastic Mirror Descent for convex, Lipschitz, non-smooth objectives under noisy sub-gradient oracle access.\"},{\"question\":\"How do the derived tail bounds extend prior work?\",\"answer\":\"They extend tail bounds from classical light-tailed sub-Gaussian noise to heavier-tailed regimes, including sub-Weibull (exponentially decaying) and polynomially tailed noise.\"},{\"question\":\"Does the analysis require truncating gradients or bounding the domain diameter?\",\"answer\":\"The results apply to Stochastic Mirror Descent in its plain form without truncation, and the tail bounds do not require an upper bound on the domain diameter.\"}]","General Tail Bounds for Non-Smooth Stochastic Mirror Descent | PDF",1785675269,91,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"general-tail-bounds-for-non-smooth-stochastic-mirror-descent","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/general-tail-bounds-for-non-smooth-stochastic-mirror-descent/117338/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address?","Question",{"text":75,"@type":76},"It studies optimization performance of Stochastic Mirror Descent for convex, Lipschitz, non-smooth objectives under noisy sub-gradient oracle access.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How do the derived tail bounds extend prior work?",{"text":80,"@type":76},"They extend tail bounds from classical light-tailed sub-Gaussian noise to heavier-tailed regimes, including sub-Weibull (exponentially decaying) and polynomially tailed noise.",{"name":82,"@type":73,"acceptedAnswer":83},"Does the analysis require truncating gradients or bounding the domain diameter?",{"text":84,"@type":76},"The results apply to Stochastic Mirror Descent in its plain form without truncation, and the tail bounds do not require an upper bound on the domain diameter.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]