[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-120039-en":3,"doc-seo-120039-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},120039,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","PETScML - Second-order solvers for training regression problems in Scientific Machine Learning","Scientific machine learning has emerged as a data-driven approach that uses deep-learning techniques to analyze results from computational science and engineering. Training neural networks is a highly non-convex optimization task, and although stochastic first-order methods dominate, SciML regression settings often provide smoother data and more informative empirical risk characterizations. A lightweight PETScML framework bridges deep-learning software and conventional unconstrained minimization solvers, enabling second-order trust-region optimization. Empirical results show Gauss-Newton-based trust-region methods reduce generalization errors, outperforming or matching adaptive first-order baselines in cost and accuracy.","arXiv :2403 . 12188v1 [ cs .LG] 18 Mar 2024  \nPETScML: Second-order solvers for training regression problems in Scientific Machine Learning  \nStefano Zampini 1 , Umberto Zerbinati2 , George Turkiyyah 1 , David Keyes 1  \n1 Extreme Computing Research Center, King Abdullah University of Science and Technology  \n2 Mathematical Institute, University of Oxford  \nMarch 20, 2024  \nAbstract  \nIn recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis, by means of deep-learning techniques, of data produced by computational science and engineering applications.  \nAt the core of these methods is the supervised training algorithm to learn the neural network realization, a highly non-convex optimization problem that is usually solved using stochastic gradient methods. However, distinct from deep-learning practice, scientific machine-learning training problems feature a much larger volume of smooth data and better characterizations of the empirical risk functions, which make them suited for conventional solvers for unconstrained optimization.  \nWe introduce a lightweight software framework built on top of the Portable and Extensible Toolkit for Scientific computation to bridge the gap between deep-learning software and conventional solvers for unconstrained minimization.  \nWe empirically demonstrate the superior efficacy of a trust region method based on the Gauss-Newton approximation of the Hessian in improving the generalization errors arising from regression tasks when learning surrogate models for a wide range of scientific machine-learning techniques and test cases. All the conventional secondorder solvers tested, including L-BFGS and inexact Newton with line-search, compare favorably, either in terms of cost or accuracy, with the adaptive first-order methods used to validate the surrogate models.  \n1 Introduction  \nIn recent years, there has been a growing interest in incorporating data-driven approaches into computational science and engineering, inspired by the advancements in deep-learning  \nmethods [36, 62] . This trend has given rise to the emerging discipline of scientific machine learning (SciML), which aims to tackle domain-specific data challenges by harnessing the predictive capabilities, interpretability, and domain knowledge offered by physics-based models. One of the attractions of neural network models lies in their ability to handle high-dimensional function approximations [16, 28]; this has led to the design of numerous techniques as scientific tools for uncovering the underlying physical laws hidden within experimental data. Examples include the discovery of partial differential equations (PDEs)  \n[11], the learning of PDEs [58, 7], and PDE solvers [59, 75] . Noteworthily, deep-learning techniques have successfully pushed the boundaries of molecular dynamics simulations, enabling accurate simulations with hundreds of millions of atoms through ab initio methods [31], and they have also been applied to solving the Hamilton–Jacobi–Bellman equations encountered in deterministic control problems [50] .  \n1.1 Background  \nNon-convex minimization problems are a fundamental challenge when training deep-learning models. The loss landscape can be very complicated, given the presence of multiple minima and saddle points, each possessing distinct generalization properties. Second-order optimization methods applied to over-parametrized deep-learning training problems have demonstrated overfitting tendencies, which hinder their generalization capabilities. This has led to the widespread preference for stochastic first-order methods, as they naturally introduce regularization in the stochastic regime, which helps to mitigate overfitting [64] . Nevertheless, the landscape of deep learning is ever-evolving, and in the era of vast datasets, we find ourselves in a highly informative data regime for training these models. The sheer volume of data can potentially shift the paradig","cbCaikZxgjpwEcEc","https://ap.wps.com/l/cbCaikZxgjpwEcEc","pdf",1274782,1,30,"English","en",105,"# Abstract\n# Introduction\n## Background\n## Related work","[{\"question\":\"What problem does PETScML address in scientific machine learning training?\",\"answer\":\"PETScML addresses the gap between deep-learning training workflows and conventional unconstrained optimization solvers for highly non-convex neural-network training, especially in SciML regression tasks.\"},{\"question\":\"Why are second-order methods a good fit for SciML regression problems?\",\"answer\":\"SciML training problems can feature larger volumes of smooth data and better characterizations of empirical risk functions, making them suitable for conventional unconstrained optimization solvers.\"},{\"question\":\"What optimization strategy does the paper emphasize for improving generalization errors?\",\"answer\":\"The paper emphasizes a trust region method using a Gauss-Newton approximation of the Hessian to improve generalization errors for regression surrogate models.\"}]","PETScML - Second-order solvers for training regression problems in Scientific Machine Learning | PDF",1785727846,76,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"petscml-second-order-solvers-for-training-regression-problems-in-scientific-machine-learning","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/petscml-second-order-solvers-for-training-regression-problems-in-scientific-machine-learning/120039/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does PETScML address in scientific machine learning training?","Question",{"text":75,"@type":76},"PETScML addresses the gap between deep-learning training workflows and conventional unconstrained optimization solvers for highly non-convex neural-network training, especially in SciML regression tasks.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why are second-order methods a good fit for SciML regression problems?",{"text":80,"@type":76},"SciML training problems can feature larger volumes of smooth data and better characterizations of empirical risk functions, making them suitable for conventional unconstrained optimization solvers.",{"name":82,"@type":73,"acceptedAnswer":83},"What optimization strategy does the paper emphasize for improving generalization errors?",{"text":84,"@type":76},"The paper emphasizes a trust region method using a Gauss-Newton approximation of the Hessian to improve generalization errors for regression surrogate models.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":21,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]