[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85455-en":3,"doc-seo-85455-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85455,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",8,"Research & Report","Exact Dynamics of Multi-class Stochastic Gradient Descent","A framework is developed to analyze learning dynamics of high-dimensional problems trained with one-pass stochastic gradient descent on data drawn from multiple anisotropic classes. The main theorem gives exact expressions for key performance quantities—risk and overlap with the true signal—through a deterministic ODE system in the high-dimensional limit, covering broad optimization settings and cases where the number of classes scales with dimension. The framework is applied to binary logistic regression and least-squares under several covariance structures, revealing a structural phase transition and asymptotic loss behavior driven by anisotropy.","arXiv :2510 . 14074v2 [ stat .ML] 13 Jul 2026  \nExact Dynamics of Multi-class Stochastic Gradient Descent  \nElizabeth Collins-Woodfin and Inbar Seroussi  \nJuly 14, 2026  \nAbstract  \nWe develop a framework for analyzing the learning dynamics of high-dimensional problems trained using one-pass stochastic gradient descent (SGD) with data from multiple anisotropic classes. Our main theorem provides exact expressions for quantities of interest, including the risk and the overlap with the true signal, in terms of a deterministic system of ODEs, valid in the high-dimensional limit. The theorem holds for a broad class of optimization problems and extends to settings where the number of classes grows with dimension. To illustrate its utility, we investigate in detail the effect of the data’s anisotropic structure on the problems of binary logistic regression and least-squares (LS) loss. We study the LS in alinear multiclass setup and derive a learning-rate threshold that depends on the average eigenvalue of the covariance matrices. In the binary logistic regression, we study three cases: isotropic covariances, data covariance matrices with a large fraction of zero eigenvalues (denoted as the zero-one model ), and covariance matrices with power-law spectra. We show that a structural phase transition occurs. In particular, for the zero-one model and the power-law model with sufficiently large power, SGD aligns more closely with values of the class mean that are projected onto the “clean directions” (i.e., directions of smaller variance) . This is supported by analytical studies and numerical simulations, which show the exact asymptotic behavior of the loss in the high-dimensional limit. The effects of data anisotropy that we demonstrate are likely to hold beyond these examples and illustrate one application of the broader theorem that we prove.  \n1 Introduction  \nStochastic optimization algorithms are applied to high-dimensional data and parameter spaces. Understanding the dynamics and generalization of these algorithms remains a central challenge in modern machine learning. While significant progress has been made on isotropic Gaussian data distributions, real-world datasets exhibit substantially more structure: they are comprised of multiple classes, each with anisotropic noise. This anisotropy and non-Gaussianity can significantly affect algorithm convergence and stability, yet most theoretical analyses either assume spherical Gaussians or consider only a small number of classes with isotropic covariance structure.  \nIn this work, we study the dynamics of stochastic gradient descent (SGD) on risk minimization problems over Gaussian mixture models (GMMs) with anisotropic class-wise covariance structures. Our framework handles multiple classes, including a number of classes that can grow with input dimension, and data with non-zero mean, addressing more realistic datasets.  \nGMMs are particularly well-suited for high-dimensional analysis due to their mathematical tractability and surprising universality. GMM universality in high-dimension means that the performance of many algorithms (like generalized linear models) depends asymptotically only on the first-and second-order moments of the data; see for example [19] . Recent work has shown that the Gaussian mixture assumption captures key behaviors across diverse machine learning tasks. [33, 19] demonstrated that empirical learning curves follow GMM behavior even for non-Gaussian data. [44] showed that deep learning representations in generative adversarial nets (GANs) can be characterized by their first two moments for a wide range of classifiers in high dimensions. Furthermore, GMMs serve as effective toy models for understanding the training dynamics of neural network final layers through the neural collapse phenomenon [39], where representations from the penultimate layer often behave like linearly separable Gaussian mixtures.  \nDespite their widespread use, most theoretical studies o","cbCaitfdtOXTLBTc","https://ap.wps.com/l/cbCaitfdtOXTLBTc","pdf",2440154,2,1,53,"English","en",105,"# Introduction\n## Main contributions","[{\"question\":\"What does the paper’s main theorem provide for multi-class SGD?\",\"answer\":\"It provides exact expressions for quantities of interest, including risk and overlap with the true signal, via a deterministic ODE system valid in the high-dimensional limit.\"},{\"question\":\"Which models and loss functions are used to illustrate the theorem?\",\"answer\":\"The paper studies binary logistic regression and least-squares loss, including an aligned linear multiclass least-squares setup and several covariance-structure variants for logistic regression.\"},{\"question\":\"What effect does anisotropic covariance have on SGD behavior?\",\"answer\":\"For certain power-law and zero-one covariance models, the paper shows a structural phase transition and that SGD aligns more closely with class-mean directions projected onto lower-variance “clean directions,” supported by analytical and numerical results.\"}]",1784203679,134,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"exact-dynamics-of-multi-class-stochastic-gradient-descent","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/exact-dynamics-of-multi-class-stochastic-gradient-descent/85455/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What does the paper’s main theorem provide for multi-class SGD?","Question",{"text":75,"@type":76},"It provides exact expressions for quantities of interest, including risk and overlap with the true signal, via a deterministic ODE system valid in the high-dimensional limit.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which models and loss functions are used to illustrate the theorem?",{"text":80,"@type":76},"The paper studies binary logistic regression and least-squares loss, including an aligned linear multiclass least-squares setup and several covariance-structure variants for logistic regression.",{"name":82,"@type":73,"acceptedAnswer":83},"What effect does anisotropic covariance have on SGD behavior?",{"text":84,"@type":76},"For certain power-law and zero-one covariance models, the paper shows a structural phase transition and that SGD aligns more closely with class-mean directions projected onto lower-variance “clean directions,” supported by analytical and numerical results.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]