[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121991-en":3,"doc-seo-121991-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121991,5909877438554,"Maeve","https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272",8,"Research & Report","Mini-Workshop: Interpolation and Over-parameterization in Statistics and Machine Learning","The report documents the Oberwolfach Mini-Workshop on interpolation and over-parameterization, focusing on the theoretical and statistical foundations of “benign overfitting” or “harmless interpolation.” It summarizes how exact interpolation of noisy training data can remain statistically optimal and closely matches behaviors of modern deep learning. Contributions cover optimization guarantees, statistical theory for interpolating over-parameterized models, and analysis of expressivity and learnability for transformer architectures, alongside discussions extending results to non-Gaussian settings and nontrivial data regimes.","Mathematisches Forschungsinstitut Oberwolfach  \nReport No. 41/2023  \nDOI: 10.4171/OWR/2023/41  \nMini-Workshop: Interpolation and Over-parameterizationin Statistics and Machine Learning  \nOrganized by  \nMikhail Belkin, San Diego  \nAlexandre Tsybakov, Paris  \nFanny Yang, Zurich  \n17 September – 22 September 2023  \nAbstract. In recent years it has become clear that, contrary to traditional statistical beliefs, methods that interpolate (􀀌t exactly) the noisy training data, can still be statistically optimal. In particular, this phenomenon of “benign over􀀌tting” or “harmless interpolation” seems to be close to the practical regimes of modern deep learning systems, and, arguably, underlies many of their behaviors. This workshop brought together experts on the emerging theory of interpolation in statistical methods, its theoretical foundations and  \napplications to machine learning and deep learning.  \nMathematics Subject Classi􀀌cation (2020): 62-xx.  \nIntroduction by the Organizers  \nThis mini-workshop was attended by a group of researchers who work on several topics related to benign over􀀌tting such as harmless interpolation for linear models and neural networks, implicit bias of 􀀌rst-order algorithms on shallow neural networks and transformers, and other topics. The participants presented novel optimization guarantees and statistical theory for interpolating solutions when training overparameterized models, as well as recent advancements in analyzing the expressivity and learnability of di􀀋erent problem classes by transformer architectures. As a result of the talks, several technical discussions ensued between researchers who had not collaborated before, for example, on extending tight benign over􀀌tting results towards non-Gaussian distributions and proving implicit biases for di􀀋erent architectures and imbalanced data. We expect that many of these discussions will lead to future publications. In total, the workshop program  \n2360 Oberwolfach Report 41/2023  \nincluded 12 talks and a longer discussion session led by Misha Belkin. The lively discussion was centered around the key ingredient behind and the relevance of the existing theory for understanding deep learning. The discussion revealed di􀀋erent opinions, one being that 􀀌nding the appropriate low-dimensional structure in language would be the key to understanding Large Language Models vs. other alternative concepts not based on linearity and low dimensionality. The debate led to follow-up interactions that are on-going after the workshop. A discussion group that arose from the debate is working on rede􀀌ning the theory to certify the reliability of neural networks.  \nInterpolation and Over-parameterization in Statistics and Machine Learning 2361  \nMini-Workshop: Interpolation and Over-parameterization in Statistics and Machine Learning  \nTable of Contents  \nGuillaume Lecu􀀓e (joint with Zong Shang) A geometrical viewpoint on the benign over􀀌tting phenomenon ........ 2363  \nEnno Mammen (joint with Munir Hiabu, Joseph Meyer) Planted regression forests ......................................... 2363  \nKonstantin Donhauser (joint with Fanny Yang, Guillaume Wang, Michael Aerni, Marco Milanta, Stefan Stojanovic and Nicolo Ruggeri)  \nSurprising behaviors of sparse min-norm and max-margin interpolators . 2364  \nNathan Srebro Interpolation Learning with Short Programs and Shallow Neural Networks ....................................................... 2365  \nOhad Shamir (joint with Guy Kornowski, Gilad Yehudai, Daniel Barzilai) Tempered and benign over􀀌tting in neural networks and kernels ....... 2365  \nDaniel Hsu (joint with Clayton Sanford and Matus Telgarsky) Representational strengths and limitations of transformers ............ 2366  \nAlberto Bietti  \nTransformers and Associative Memories ............................ 2366  \nMatus Telgarsky  \nBenign Calibration .............................................. 2367  \nVidya Muthukumar (joint with Mikhail Belkin, Daniel Hsu, Adhyyan Narang","cbCaibXFW5aHRzhb","https://ap.wps.com/l/cbCaibXFW5aHRzhb","pdf",177464,1,18,"English","en",105,"# Introduction by the Organizers\n## Attendance and workshop scope\n## Summary of discussions\n# Table of Contents\n## A geometrical viewpoint on the benign overfitting phenomenon\n## Planted regression forests\n## Surprising behaviors of sparse min-norm and max-margin interpolators\n## Interpolation Learning with Short Programs and Shallow Neural Networks\n## Tempered and benign overfitting in neural networks and kernels\n## Representational strengths and limitations of transformers\n## Transformers and Associative Memories\n## Benign Calibration\n## Classification versus regression with ℓ2-minimizing solutions\n## Implicit Geometries through the Imbalance Lens\n## Some statistical insights into PINNs\n## In-context learning linear models with transformers","[{\"question\":\"What is “benign overfitting” in the context of this workshop?\",\"answer\":\"It refers to regimes where methods that interpolate noisy training data exactly can still be statistically optimal. The report highlights this as a key departure from traditional statistical beliefs.\"},{\"question\":\"Which themes were discussed regarding interpolation and over-parameterized models?\",\"answer\":\"The workshop addressed harmless interpolation for linear models and neural networks, implicit bias of first-order algorithms on shallow networks and transformers, and optimization/statistical guarantees for interpolating over-parameterized models.\"},{\"question\":\"How did the workshop’s discussions aim to extend existing theory?\",\"answer\":\"Participants discussed extending tight benign overfitting results beyond Gaussian distributions, deriving implicit biases for different architectures and imbalanced data, and developing theory to certify the reliability of neural networks.\"}]","Mini-Workshop: Interpolation and Over-parameterization in Statistics and Machine Learning | PDF",1785808183,45,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"mini-workshop-interpolation-and-over-parameterization-in-statistics-and-machine-learning","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/mini-workshop-interpolation-and-over-parameterization-in-statistics-and-machine-learning/121991/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is “benign overfitting” in the context of this workshop?","Question",{"text":75,"@type":76},"It refers to regimes where methods that interpolate noisy training data exactly can still be statistically optimal. The report highlights this as a key departure from traditional statistical beliefs.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which themes were discussed regarding interpolation and over-parameterized models?",{"text":80,"@type":76},"The workshop addressed harmless interpolation for linear models and neural networks, implicit bias of first-order algorithms on shallow networks and transformers, and optimization/statistical guarantees for interpolating over-parameterized models.",{"name":82,"@type":73,"acceptedAnswer":83},"How did the workshop’s discussions aim to extend existing theory?",{"text":84,"@type":76},"Participants discussed extending tight benign overfitting results beyond Gaussian distributions, deriving implicit biases for different architectures and imbalanced data, and developing theory to certify the reliability of neural networks.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]