[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-290940-105":59,"doc-detail-290940-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","towards-foundation-models-for-scientific-machine-learning-characterizing-scaling-and-transfer-behavior","Towards Foundation Models for Scientific Machine Learning - Characterizing Scaling and Transfer Behavior","","Pre-trained machine learning models are leveraged for scientific machine learning through transfer learning. The study characterizes how transfer behavior changes when pre-trained model size scales, downstream dataset size scales, physics parameters move beyond the training distribution, and a single model trained on mixed physics problems is adapted to multiple downstream tasks. Results show fine-tuned transfer achieves target accuracy with orders-of-magnitude fewer downstream examples than training from scratch, with consistent behavior and increasing gains versus scratch scaling across PDE learning tasks.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/towards-foundation-models-for-scientific-machine-learning-characterizing-scaling-and-transfer-behavior/290940/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/towards-foundation-models-for-scientific-machine-learning-characterizing-scaling-and-transfer-behavior/290940.png","ImageObject",300,407,{"name":92,"@type":93},"Himbo","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-18","2026-09-17",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",12,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What is the main goal of the paper on scientific machine learning foundation models?","Question",{"text":113,"@type":114},"To determine whether the pre-train-and-fine-tune paradigm used in NLP/CV can work for SciML, by analyzing transfer learning behavior for PDE learning tasks.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"Which factors are studied to understand transfer behavior?",{"text":118,"@type":114},"Pre-trained model size scaling, downstream dataset size scaling, physics parameters pushed out of distribution, and adaptation of a single model trained on mixtures to different downstream applications.",{"name":120,"@type":111,"acceptedAnswer":121},"How does fine-tuned transfer learning compare with training from scratch?",{"text":122,"@type":114},"With appropriate fine-tuning, transfer learning reaches desired accuracy levels using orders of magnitude fewer downstream examples and shows consistent behavior across tasks, including out-of-distribution cases.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},290940,1789643001,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":36,"preview_page_count":4},687197100911,"https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697","Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior  \nShashank Subramanian[shashanksubramanian@lbl.gov](shashanksubramanian@lbl.gov)[ ](shashanksubramanian@lbl.gov)Lawrence Berkeley National Lab  \nWahid Bhimji  \n[wbhimji@lbl.gov](wbhimji@lbl.gov)[ ](wbhimji@lbl.gov)Lawrence Berkeley National Lab  \nPeter Harrington  \n[pharrington@lbl.gov](pharrington@lbl.gov)[ ](pharrington@lbl.gov)Lawrence Berkeley National Lab  \nDmitriy Morozov  \n[dmorozov@lbl.gov](dmorozov@lbl.gov)[ ](dmorozov@lbl.gov)Lawrence Berkeley National Lab  \nKurt Keutzer  \n[keutzer@eecs.berkeley.edu](keutzer@eecs.berkeley.edu)[ ](keutzer@eecs.berkeley.edu)UC Berkeley  \nMichael W. Mahoney [mmahoney@stat.berkeley.edu](mmahoney@stat.berkeley.edu)[ ](mmahoney@stat.berkeley.edu)LBNL, ICSI, and UC Berkeley  \narXiv :2306 .00258v 1 [ cs .LG] 1 Jun 2023  \nAmir Gholami  \n[amirgh@berkeley.edu](amirgh@berkeley.edu)[ ](amirgh@berkeley.edu)ICSI, UC Berkeley  \nABSTRACT  \nPre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV) . Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre-trained model size is scaled,(ii) the downstream training dataset size is scaled,(iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that—when fine-tuned appropriately—transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behaviour across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential ofthe “pre-train and fine-tune” paradigm for SciML problems, demonstrating a path towards building SciML foundation models. We open-source our code at [1] .  \n1 INTRODUCTION  \nFoundation models have received considerable interest recently [3] . This terminology refers to certain models that are trained on extremely large and diverse quantities of data and applied to a wide range of tasks. Rather than being designed for any single task, a foundation model serves as a “prior” or “foundation” upon which other models can be built. It does so by using transfer learning (TL) methods to fine-tune or adapt the foundation model to a wide range of downstream tasks, using minimal additional data for each additional task. Perhaps the most well-known foundation models are pre-trained large-language models (LLMs) such as BERT [11] and the GPT models [5, 39, 40] . The scaling with respect to the amount of data, the size of the model, and the amount of compute [20, 23] is key to the training of these models. An important aspect of a  \ntrained foundation model is the notion of emergence—by leveraging shared features across the training tasks, the model is able to perform tasks seemingly different than those for which it was trained. This approach to model development is quite different than the traditional approach of training a one-off model from scratch for each specific problem and each specific dataset. Naturally, it is of interest how broadly this methodological approach can be applied.  \nScientific machine learning (SciML) [46] is an area that combines tools from ML and scientific computing to address domain-specific scientific and engineering challenges. It holds promise to drive the next wave of data-driven discovery in the physical ","cbCaiktYNj6SOxZa","https://ap.wps.com/l/cbCaiktYNj6SOxZa","pdf",12316968,16,"English","# Abstract\n# Introduction\n## Foundation models and emergence\n## SciML and neural PDE solvers","[{\"question\":\"What is the main goal of the paper on scientific machine learning foundation models?\",\"answer\":\"To determine whether the pre-train-and-fine-tune paradigm used in NLP/CV can work for SciML, by analyzing transfer learning behavior for PDE learning tasks.\"},{\"question\":\"Which factors are studied to understand transfer behavior?\",\"answer\":\"Pre-trained model size scaling, downstream dataset size scaling, physics parameters pushed out of distribution, and adaptation of a single model trained on mixtures to different downstream applications.\"},{\"question\":\"How does fine-tuned transfer learning compare with training from scratch?\",\"answer\":\"With appropriate fine-tuning, transfer learning reaches desired accuracy levels using orders of magnitude fewer downstream examples and shows consistent behavior across tasks, including out-of-distribution cases.\"}]","Towards Foundation Models for Scientific Machine Learning - Characterizing Scaling and Transfer Behavior | PDF"]