[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121259-en":3,"doc-seo-121259-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121259,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","TWO-LEVEL OVERLAPPING ADDITIVE SCHWARZ PRECONDITIONER FOR TRAINING SCIENTIFIC MACHINE LEARNING APPLICATIONS","We introduce a two-level overlapping additive Schwarz preconditioner that accelerates training for scientific machine learning applications. The method is motivated by nonlinear two-level overlapping additive Schwarz preconditioning, and decomposes neural network parameters into overlapping subdomains. A novel subdomain-wise synchronization strategy and a coarse-level training step implicitly impose the network feed-forward structure. Numerical experiments on physics-informed neural networks and operator learning show faster convergence of the standard LBFGS optimizer, improved model accuracy, and support for model-parallel computation to further reduce training time.","arXiv :2406 . 10997v1 [math .NA] 16 Jun 2024  \nTWO-LEVEL OVERLAPPING ADDITIVE SCHWARZ PRECONDITIONER FOR TRAINING SCIENTIFIC MACHINE  \nLEARNING APPLICATIONS ∗  \nYOUNGKYU LEE†, ALENA KOPANIČÁKOVÁ† ‡, AND GEORGE EM KARNIADAKIS† §  \nAbstract. We introduce a novel two-level overlapping additive Schwarz preconditioner for accelerating the training of scientific machine learning applications. The design of the proposed preconditioner is motivated by the nonlinear two-level overlapping additive Schwarz preconditioner. The neural network parameters are decomposed into groups (subdomains) with overlapping regions. In addition, the network’s feed-forward structure is indirectly imposed through a novel subdomain-wise synchronization strategy and a coarse-level training step. Through a series of numerical experiments, which consider physics-informed neural networks and operator learning approaches, we demonstrate that the proposed two-level preconditioner significantly speeds up the convergence of the standard (LBFGS) optimizer while also yielding more accurate machine learning models. Moreover, the devised preconditioner is designed to take advantage of model-parallel computations, which can further reduce the training time.  \nKey words. scientific machine learning, nonlinear preconditioning, Schwarz methods, domaindecomposition  \nMSC codes. 90C30, 90C26, 90C06, 65M55, 68T07  \n1. Introduction. Deep neural networks (DNNs) are universal approximators, capable of approximating any continuous function defined on a compact domain to arbitrary accuracy [4, 11 , 62] . Therefore, they have been recently widely used to obtain ansatz spaces for the solutions of partial differential equations (PDEs) fora wide range of scientific and engineering problems; see [35] for an overview. The popularity of these DNN approaches can be largely attributed to their simplicity, mesh-free nature, and ability to incorporate data [60, 63], enabling them to effectively tackle complex, high-dimensional, forward and inverse problems in single and multiquery scenarios [56] .  \nTwo prominent approaches for solving parametric PDEs using DNNs appear in the literature. The first approach approximates a solution of a given PDE for a particular choice of parameters using a DNN [60, 63] . Popular representatives of this approach are physics informed neural networks (PINNs) [60], which train the DNNs by minimizing the mean square error that incorporates the PDE residual, boundary/initial conditions, and/or observed data. The second approach considers learning the differential operator [52, 67] from the parametrized source terms, boundary/initial conditions, or material properties. The operator learning DNNs, such as DeepONet (DON) [52] or FNO [49], are typically trained by minimizing the mean square error  \n∗ Submitted to the editors 30-June-2024 .  \nFunding: Y.L. was supported in part by Basic Science Research Program through NRF funded by the Ministry of Education (No. RS2023-00247199) . A.K. was supported by the Swiss National Science Foundation (SNSF) under the projects “Multilevel training of DeepONets – multiscale and multiphysics applications” and “ML2 - Multilevel and Domain Decomposition Methods for Machine Learning”, as well as by the Platform for Advanced Scientific Computing (PASC) under the project [Exa Train](Exa Train. G.E.K. is)[. G.E.K. is](Exa Train. G.E.K. is) supported by the ONR Vannevar Bush Faculty Fellowship. We also acknowledge support from the DOE-MMICS SEA-CROGS DE-SC0023191 award and Ansys Inc.  \n†Division of Applied Mathematics, Brown University, Providence, USA ([youngkyu_lee@brown.edu](youngkyu_lee@brown.edu), [alena_kopanicakova@brown.edu](alena_kopanicakova@brown.edu), [george_karniadakis@brown.edu](george_karniadakis@brown.edu)).  \n‡Euler Institute, Università della Svizzera italiana, Lugano, Switzerland ([alena.kopanicakova@usi.ch](alena.kopanicakova@usi.ch)).  \n§ School of Engineering, Brown University, Providence, USA  \n2  \nbetween the netw","cbCaipgIRNHm4LCZ","https://ap.wps.com/l/cbCaipgIRNHm4LCZ","pdf",2774190,1,24,"English","en",105,"# Introduction\n## Neural network approaches for parametric PDEs\n## Parallel optimization paradigms (data-parallel and model-parallel)\n## Domain decomposition and additive Schwarz methods\n## Scalability challenges and motivation for two-level preconditioning","[{\"question\":\"What is the main contribution of the proposed method?\",\"answer\":\"It introduces a two-level overlapping additive Schwarz preconditioner designed to accelerate training in scientific machine learning applications while improving convergence and accuracy.\"},{\"question\":\"How does the preconditioner interact with the neural network structure?\",\"answer\":\"Network parameters are decomposed into overlapping subdomains, and a novel subdomain-wise synchronization strategy plus a coarse-level training step indirectly enforces the feed-forward structure.\"},{\"question\":\"What training setups and experiments are used to validate performance?\",\"answer\":\"Experiments consider physics-informed neural networks and operator learning approaches, demonstrating faster convergence with the LBFGS optimizer and more accurate trained models.\"}]","TWO-LEVEL OVERLAPPING ADDITIVE SCHWARZ PRECONDITIONER FOR TRAINING SCIENTIFIC MACHINE LEARNING APPLICATIONS | PDF",1785734703,60,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"two-level-overlapping-additive-schwarz-preconditioner-for-training-scientific-machine-learning-applications","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/two-level-overlapping-additive-schwarz-preconditioner-for-training-scientific-machine-learning-applications/121259/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main contribution of the proposed method?","Question",{"text":75,"@type":76},"It introduces a two-level overlapping additive Schwarz preconditioner designed to accelerate training in scientific machine learning applications while improving convergence and accuracy.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the preconditioner interact with the neural network structure?",{"text":80,"@type":76},"Network parameters are decomposed into overlapping subdomains, and a novel subdomain-wise synchronization strategy plus a coarse-level training step indirectly enforces the feed-forward structure.",{"name":82,"@type":73,"acceptedAnswer":83},"What training setups and experiments are used to validate performance?",{"text":84,"@type":76},"Experiments consider physics-informed neural networks and operator learning approaches, demonstrating faster convergence with the LBFGS optimizer and more accurate trained models.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":29,"slug":108},5,"Comic","comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]