Towards Foundation Models for Scientific Machine Learning - Characterizing Scaling and Transfer Behavior

Sep 17, 2026

Uploaded by Himbo

Pre-trained machine learning models are leveraged for scientific machine learning through transfer learning. The study characterizes how transfer behavior changes when pre-trained model size scales, downstream dataset size scales, physics parameters move beyond the training distribution, and a single model trained on mixed physics problems is adapted to multiple downstream tasks. Results show fine-tuned transfer achieves target accuracy with orders-of-magnitude fewer downstream examples than training from scratch, with consistent behavior and increasing gains versus scratch scaling across PDE learning tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Abstract

Introduction

Foundation models and emergence

SciML and neural PDE solvers

Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior Shashank Subramanian[shashanksubramanian@lbl.gov](shashanksubramanian@lbl.gov)[ ](shashanksubramanian@lbl.gov)Lawrence Berkeley National Lab Wahid Bhimji [wbhimji@lbl.gov](wbhimji@lbl.gov)[ ](wbhimji@lbl.gov)Lawrence Berkeley National Lab Peter Harrington [pharrington@lbl.gov](pharrington@lbl.gov)[ ](pharrington@lbl.gov)Lawrence Berkeley National Lab Dmitriy Morozov [dmorozov@lbl.gov](dmorozov@lbl.gov)[ ](dmorozov@lbl.gov)Lawrence Berkeley National Lab Kurt Keutzer [keutzer@eecs.berkeley.edu](keutzer@eecs.berkeley.edu)[ ](keutzer@eecs.berkeley.edu)UC Berkeley Michael W. Mahoney [mmahoney@stat.berkeley.edu](mmahoney@stat.berkeley.edu)[ ](mmahoney@stat.berkeley.edu)LBNL, ICSI, and UC Berkeley arXiv :2306 .00258v 1 [ cs .LG] 1 Jun 2023 Amir Gholami [amirgh@berkeley.edu](amirgh@berkeley.edu)[ ](amirgh@berkeley.edu)ICSI, UC Berkeley ABSTRACT Pre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV) . Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre-trained model size is scaled,(ii) the downstream training dataset size is scaled,(iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that—when fine-tuned appropriately—transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behaviour across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential ofthe “pre-train and fine-tune” paradigm for SciML problems, demonstrating a path towards building SciML foundation models. We open-source our code at [1] . 1 INTRODUCTION Foundation models have received considerable interest recently [3] . This terminology refers to certain models that are trained on extremely large and diverse quantities of data and applied to a wide range of tasks. Rather than being designed for any single task, a foundation model serves as a “prior” or “foundation” upon which other models can be built. It does so by using transfer learning (TL) methods to fine-tune or adapt the foundation model to a wide range of downstream tasks, using minimal additional data for each additional task. Perhaps the most well-known foundation models are pre-trained large-language models (LLMs) such as BERT [11] and the GPT models [5, 39, 40] . The scaling with respect to the amount of data, the size of the model, and the amount of compute [20, 23] is key to the training of these models. An important aspect of a trained foundation model is the notion of emergence—by leveraging shared features across the training tasks, the model is able to perform tasks seemingly different than those for which it was trained. This approach to model development is quite different than the traditional approach of training a one-off model from scratch for each specific problem and each specific dataset. Naturally, it is of interest how broadly this methodological approach can be applied. Scientific machine learning (SciML) [46] is an area that combines tools from ML and scientific computing to address domain-specific scientific and engineering challenges. It holds promise to drive the next wave of data-driven discovery in the physical

FAQs

What is the main goal of the paper on scientific machine learning foundation models?

To determine whether the pre-train-and-fine-tune paradigm used in NLP/CV can work for SciML, by analyzing transfer learning behavior for PDE learning tasks.

Which factors are studied to understand transfer behavior?

Pre-trained model size scaling, downstream dataset size scaling, physics parameters pushed out of distribution, and adaptation of a single model trained on mixtures to different downstream applications.

How does fine-tuned transfer learning compare with training from scratch?

With appropriate fine-tuning, transfer learning reaches desired accuracy levels using orders of magnitude fewer downstream examples and shows consistent behavior across tasks, including out-of-distribution cases.

Sep 17, 2026

Uploaded by Himbo