A Cloud-Agnostic Serverless Architecture for Distributed Machine Learning - research paper

Aug 2, 2026

Upload by Levi

Serverless computing shows strong promise for big data analytics, particularly with machine learning. Prior research rarely addresses cloud-agnostic serverless architectures that reuse existing parallel ML implementations. This work proposes a multicloud serverless architecture enabling ML engineers, regardless of cloud expertise, to port parallel stateful ML algorithms while avoiding vendor lock-in. Two algorithms—k-means clustering and logistic regression—are ported to serverless and evaluated.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

Background and problem statement

Proposed cloud-agnostic serverless architecture

Experimental evaluation

Performance and scalability results

Shared state storage designs (locked vs lock-free)

This is a repository copy of A Cloud-Agnostic Serverless Architecture for Distributed Machine Learning. White Rose Research Online URL for this paper: [https://eprints.whiterose.ac.uk/id/eprint/226782/](https://eprints.whiterose.ac.uk/id/eprint/226782/) [Version: Accepted Version](Version: Accepted Version) Proceedings Paper: Predoaia, [Ionut orcid.org/0000-0002-2009-4054 and Garc](Ionut orcid.org/0000-0002-2009-4054 and Garc) ía-López, Pedro (2025) A CloudAgnostic Serverless Architecture for Distributed Machine Learning. In: Proceedings-2024 IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, BDCAT 2024. 11th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, BDCAT 2024, 16-19 Dec 2024 IEEE , ARE , pp. 131-140. [https://doi.org/10.1109/BDCAT63179.2024.00032](https://doi.org/10.1109/BDCAT63179.2024.00032) Reuse This article is distributed under the terms of the Creative Commons Attribution (CC BY) licence. This licence allows you to distribute, remix, tweak, and build upon the work, even commercially, as long as you credit the authors for the original work. More information and the full terms of the licence here: [https://creativecommons.org/licenses/](https://creativecommons.org/licenses/) Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk) including the URL of the record and the reason for the withdrawal request. [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk)[ ](eprints@whiterose.ac.uk)[https://eprints.whiterose.ac.uk/](https://eprints.whiterose.ac.uk/) A Cloud-Agnostic Serverless Architecture for Distributed Machine Learning Ionut Predoaia University of York United Kingdom [ionut.predoaia@york.ac.uk](ionut.predoaia@york.ac.uk) Pedro GarcÂıa-LÂopez Universitat Rovira i Virgili Spain [pedro.garcia@urv.cat](pedro.garcia@urv.cat) AbstractÐServerless computing has shown vast potential for big data analytics applications, especially involving machine learning algorithms. Nevertheless, little consideration has been given in the literature to cloud-agnostic serverless architectures that leverage existing parallel implementations of machine learning algorithms. This work bridges this gap by proposing a multicloud serverless architecture for distributed machine learning, that enables machine learning engineers without cloud computing expertise to effortlessly port already implemented parallel machine learning algorithms to serverless, whilst overcoming vendor lock-in. In this work, two stateful machine learning algorithms have been ported to serverless, k-means clustering and logistic regression. The serverless implementation ofk-means provided superior performance and scalability compared to aserverful implementation when using a number of workers that is equal to or slightly lower than the total number of vCPUs available on the VM running the serverful implementation. Additionally, it achieved an 87-fold speedup compared to a sequential implementation. Moreover, two storage designs of the shared state will be proposed for the serverless implementations, one that requires locks for updating the shared state, and another that is lock-free. Our experimental evaluation demonstrates that the performance of the lock-free serverless implementation ofk-means declines with the increase in the number of clusters. Index TermsÐDistributed Machine Learning, Big Data, Serverless Architectures, Cloud Agnostic, Multicloud, Lithops I. INTRODUCTION The rise of serverless computing has widely propagated the democratization of massive-scale data parallelism. By leveraging serverless computing, cloud users can today launch thousands of concurrent stateless functions to run big data analytics workloads, without the need of complex cluster management and the burden of managing and provisioning cloud resources. One can sea

FAQs

What problem does the proposed architecture address?

It targets the lack of cloud-agnostic serverless architectures that can reuse existing parallel machine learning implementations without vendor lock-in.

Which machine learning algorithms are ported and evaluated?

The paper ports and evaluates k-means clustering and logistic regression in a serverless setting.

How does the serverless k-means performance compare to a serverful implementation?

The serverless k-means implementation shows superior performance and scalability for worker counts around or slightly below the serverful VM’s available vCPUs, achieving an 87-fold speedup over a sequential baseline.

Aug 2, 2026

Upload by Levi