COST-EFFICIENT MLOPS ARCHITECTURE FOR SCIENTIFIC MACHINE LEARNING - A PRACTICAL DESIGN FOR DIFFUSION-BASED GNSS DATA ASSIMILATION ON SELF-MANAGED INFRASTRUCTURE

Sep 17, 2026

Uploaded by Skyler

High-resolution weather forecasting increasingly relies on machine learning models, including diffusion-based methods, for tasks such as data assimilation. Training these models requires substantial computational power, often pushing research teams toward expensive managed MLOps platforms or complex self-hosted solutions. This thesis builds a scalable, reproducible, cost-efficient MLOps pipeline for scientific machine learning on self-managed GPU infrastructure. The work integrates ClearML with a custom Go-based GPU resource manager to automate training lifecycle while maintaining simplicity and cost control.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages80

Reading Time3 h 22 min

AccessFree to download

Introduction

Background and related works

Machine learning

Machine learning operations (MLOps)

MLOps maturity model

MLOps architecture comparison

Methodology

Design science research (DSR)

Problem identification

Objectives

Evaluation

Design and development

Technology stack

Architectural overview

Training orchestration

Compute resource management

Data processing

Experiment tracking

Performance monitoring

Evaluation & results

Easy adoption (O1)

Cost effective (O2)

Extendable (O3)

Ease of use (O4)

Discussion

Interpretation of results

Comparison with state of the art

Practical implications

Limitations

Future work

Conclusion

Nico Renaldo COST-EFFICIENT MLOPS ARCHITECTURE FOR SCIENTIFIC MACHINE LEARNING A PRACTICAL DESIGN FOR DIFFUSION-BASED GNSS DATA ASSIMILATION ON SELF-MANAGED INFRASTRUCTURE Master’s thesis Faculty of Information Technology and Communication Sciences December 2025 i ABSTRACT Nico Renaldo Cost-Efficient MLOps Architecture for Scientific Machine Learning: A Practical Design for Diffusion-Based GNSS Data Assimilation on Self-Managed Infrastructure Master’s thesis Tampere University Master’s Programme in Signal Processing and Machine Learning December 2025 High-resolution weather forecasting increasingly relies on machine learning models, including diffusion-based methods, for tasks such as data assimilation. Training these models requires substantial computational power, often pushing research teams toward expensive managed MLOps platforms or complex self-hosted solutions. This thesis addresses the challenge of building a scalable, reproducible, and cost-efficient MLOps pipeline for scientific machine learning on self-managed GPU infrastructure. Conducted in collaboration with Skyfora and leveraging Scaleway’s GPU resources, this research presents a hybrid MLOps architecture that integrates ClearML for experiment orchestration and tracking with a custom Go-based resource manager for dynamic GPU provisioning. The system occupies a middle ground between enterprise-level orchestration tools architecture and manually managed research infrastructure, automating the full training job lifecycle while maintaining simplicity and cost efficiency. Evaluation results demonstrate a 76.9% reduction in operational costs compared to static provisioning through dynamic resource management. The system consolidates the developer interface into three tools (Git, Python, and ClearML), reducing typical training workflows from 20–30 minutes to 4 minutes from code changes to result retrieval. Using predominantly off-the-shelf components (83%) and only 17% of custom development (approximately 2,548 lines of Go code), the architecture achieves MLOps Maturity Level 2 (Automated Training) with a foundation for Level 3 capabilities. These findings demonstrate that resource-constrained research teams can build robust, reproducible MLOps pipelines without the complexity of enterprise orchestration tools or the costs of managed platforms, ultimately enabling researchers to focus on scientific discovery rather than infrastructure management. Keywords: MLOps, Scientific Machine Learning, Diffusion Models, Cloud Computing, Resource Management, ClearML, Scaleway, Reproducibility The originality of this thesis has been checked using the Turnitin Originality service. ii PREFACE This thesis was conducted within the Faculty of Information Technology and Communication Sciences at Tampere University, in collaboration with Skyfora Oy. I want to express my deepest gratitude to my supervisor, Professor David Hästbacka, for his expert guidance, insightful feedback, and continuous support throughout this research. I am also sincerely thankful to Sergio Moreschini for his valuable contribution in formulating and refining the research question, and to Alex Matakos from Skyfora for his significant support in shaping the topic and for providing the opportunity to carry out this work in partnership with the company. I am endlessly grateful to my partner, Valerie. Her steady support, patience, and optimism carried me through every high and low of this journey. Her constant encouragement and belief in my work have been indispensable to the completion of this thesis. Tampere, 16th December 2025 Nico Renaldo iii CONTENTS 1 INTRODUCTION ................................ 1 2 BACKGROUND AND RELATED WORKS .................... 3 2.1 MACHINE LEARNING ............................ 3 2.2 MACHINE LEARNING OPERATIONS (MLOPS) .............. 4 2.3 MLOPS MATURITY MODEL ........................ 6 2.4 MLOPS ARCHITECTURE COMPARISON .........

FAQs

What problem does the thesis address in diffusion-based GNSS data assimilation?

It addresses how to build a scalable, reproducible, and cost-efficient MLOps pipeline for scientific machine learning when using self-managed GPU infrastructure instead of expensive managed platforms or complex self-hosted setups.

How does the proposed MLOps architecture manage GPU resources and orchestrate experiments?

It uses ClearML for experiment orchestration and tracking, combined with a custom Go-based resource manager that dynamically provisions GPUs and automates the full training job lifecycle.

What were the evaluation outcomes of the cost-efficient approach?

The evaluation reports a 76.9% reduction in operational costs versus static provisioning, and it streamlines typical training workflows by consolidating the developer interface into Git, Python, and ClearML, reducing the workflow time from 20–30 minutes to 4 minutes.

Sep 17, 2026

Uploaded by Skyler