A Rigorous Uncertainty-Aware Quantification Framework Is Essential for Reproducible and Replicable Machine Learning Workflows

Aug 5, 2026

Upload by Anda

Replicable machine learning and AI predictions underpin scientific workflows, yet results can vary widely due to training/testing data availability, model architecture and parameters, random initialization, and differences across software platforms or library versions. Published claims often emphasize tuned accuracy without disclosing input ranges that generate it, weakening reproducibility. The text argues for rigorous, uncertainty-aware quantification to support transparency, measure reproducibility and replicability, and establish trust when ML models replace costly simulations or predict without ground truth.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Motivation: Why ML/AI results fail to reproduce

Sources of variability (data, initialization, platforms)

Limits of current reporting and transparency

Accuracy-first publications and missing input ranges

Integration of ML/AI into scientific workflows

HPC workflows and ML module replacement

Toward trust: transparency, FAIR, containers, and trustworthy computing

Reproducibility taxonomies and implementation requirements

Trustworthy AI and uncertainty-aware quantification

A Rigorous Uncertainty-Aware Quantification Framework Is Essential for Reproducible and Replicable Machine Learning Workflows Line Pouchard1, Kristofer G. Reyes1,2, Francis J. Alexander3, Byung-Jun Yoon1,4 1Brookhaven National Laboratory, Upton, NY 11973, USA 2 University at Buffalo, Buffalo, NY 14260, USA 3Argonne National Laboratory, IL 60439, USA 4 Texas A&M University, TX 77843, USA The ability to successfully replicate predictions by machine learning (ML) or artificial intelligence (AI) models and results in scientific workflows that incorporate such ML/AI predictions is driven by numerous factors, such as the availability of training/testing datasets, the choice of model architectures and parameters, and initial conditions [1,2] . In applications relying on deep learning models, e.g., in image recognition, reproducibility depends upon initializing random seeds that can be silently set by underlying libraries, among other factors [69] . Even when the same data input and initial scripts are re-used, predictions by ML/AI models can exhibit large variability, including outliers that make these results appear unreliable [3,4] . Changing the underlying ML platforms, even new versions of the same, can alter results in a significant way [5,6] . For example, a recent reproducibility study [70] reported that a simple transcription of the same model that was originally implemented in the Java-based Magpie/Weka framework [71] to the Python-based Matminer/scikit-learn framework resulted in a significant unexpected discrepancy in the predictions made by the two platforms. Published results for ML experiments often privilege accuracy obtained with much tuning, and the publications reporting these results may not necessarily provide the ranges of input conditions that produce the reported accuracy, resulting in irreproducible results [7,8] . Varying input ranges for key physical variables in physical experiments and computational studies can be crucial to the applicability of ML algorithms to various classes of experiments. The systems-level view that encourages users to ignore low-level details and focuses instead on modeling the aggregate input-output of a particular process has generated progress in automating experimental and computational scientific workflows. In this perspective, complicated sub-systems are replaced by black-box ML/AI built from data. However, the probabilistic viewpoint that makes ML powerful at general-purpose modeling can also make its calculations opaque [9,10,11,12,42] . This is particularly important when ML models behave in unpredictable ways or when models are used to predict quantities for which there is no ground truth, as in the cases of models developed for scientific discovery. Instead of a verified result based on trusted calculations, scientists may be faced with varying predictions and no rationale to determine the best course of action. One of the major challenges scientists will face in the coming years is the integration of ML/AI models and predictions in scientific computational and experimental workflows, whether these predictions replace expensive computational calculations, aid in predicting calculation results, help search through high-dimensional spaces to obtain preliminary candidates for analysis, and numerous new, emerging or yet unforeseen applications. We consider ML/AI predictions for scientific experiments that include numerical simulation campaigns and machine learning tasks, typically orchestrated in computational workflows. Traditionally, scientific workflows rely on building blocks carefully composed with high quality, curated data and first-principles scientific calculations often executed on High Performance Computing (HPC) systems [13] . Examples of promising use of ML/AI in scientific workflows include replacing some computationally expensive modules with cheaper ML-based ones, mitigating challenges that arise from limited and possibly noisy observational data,

FAQs

Why can machine learning predictions become non-replicable across workflows?

Variability arises from differences in datasets, model architectures and parameters, random seed initialization hidden in libraries, and changes across ML platforms or software versions.

How does insufficient reporting reduce reproducibility in ML experiments?

Many publications highlight tuned accuracy without providing the ranges of input conditions that produce those results, making it difficult to reproduce the reported performance.

What role does uncertainty-aware quantification play for trustworthy ML in science?

A rigorous uncertainty-aware approach enables measurement of reproducibility and replicability, clarifies the reliability of model outputs, and supports decision-making when ground truth is unavailable.

Aug 5, 2026

Upload by Anda