Comparison of Autoscaling Frameworks for Containerised Machine-Learning Applications in a Local and Cloud Environment - read online free

Aug 5, 2026

Upload by Violet

Automated allocation of computing resources, or autoscaling, is essential for deploying machine learning (ML) inference workloads while keeping stable response times under fluctuating demand. The study evaluates deployment techniques across application-level scaling (TorchServe, RayServe) and container-level scaling (K3s) in a local environment, then compares container and machine-level scaling in AWS using ECS and EKS. Performance is assessed via mean and standard deviation of inference time for a multi-client scenario and by upscaling response times, leading to a recommended strategy for local and cloud deployments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

AccessFree to download

Introduction

Deployment Methods

Multi-model application scenario

Scaling levels and frameworks

Method Comparison

Multi-client evaluation

RayServe Performance Analysis

Resource usage and suitability

Summary and Next Steps

Comparison of Autoscaling Frameworks for Containerised Machine-Learning-Applications in a Local and Cloud Environment 1st Christian Schrder Manufacturing Technologies Vitesco Technologies GmbH Limbach-Oberfrohna, Germany [cschroeder.research@gmail.com](cschroeder.research@gmail.com) 2nd Ren Bhm Manufacturing Technologies Vitesco Technologies GmbH Limbach-Oberfrohna, Germany [rene.boehm@vitesco.com](rene.boehm@vitesco.com) 3rd Alexander Lampe Dept. Engineering University of Applied Sciences Mittweida, Germany [lampe@hs-mittweida.de](lampe@hs-mittweida.de) arXiv :2311 . 18659v2 [ cs .DC] 25 Feb 2024 Abstract—When deploying machine learning (ML) applications, the automated allocation of computing resources - commonly referred to as autoscaling - is crucial for maintaining a consistent inference time under fluctuating workloads. The objective is to maximize the Quality of Service metrics, emphasizing performance and availability, while minimizing resource costs. In this paper, we compare scalable deployment techniques across three levels of scaling: at the application level (TorchServe, RayServe) and the container level (K3s) in a local environment (production server), as well as at the container and machine levels in a cloud environment (Amazon Web Services Elastic Container Service and Elastic Kubernetes Service). The comparison is conducted through the study of mean and standard deviation of inference time in a multi-client scenario, along with upscaling response times. Based on this analysis, we propose a deployment strategy for both local and cloud-based environments. Index Terms—Autoscaling, Cloud computing, Kubernetes, Amazon Web Services, Scalability, Container, Machine Learning Inference I. INTRODUCTION ML models can be utilized for anomaly detection in automated optical quality controls or time series data analysis for manufacturing processes such as screwing, welding, or mounting. The deployment of these models must adhere to Quality of Service requirements, encompassing performance metrics like inference times and availability metrics related to upscaling response times. Additional challenges [1] arise during the transition of software from the development environment to the production environment. The IT department’s support for software solutions in a production setting necessitates standardized, production-tested frameworks. With 1,416 logos featured on the 2023 Machine Learning, Artificial Intelligence, and Data Landscape [2], establishing a standard proves to bea significant challenge. Autoscaling has become a highly researched field, with a predominant focus on the development of autoscaling rules. Common techniques involve the use of machine learning algorithms such as reinforcement learning [3] [4] [5] [6], decision trees [7] [8] or long short term models [9] . Other methods are based on multi-level metric monitoring [10], fuzzy logic [11], second order autoregressive moving average (ARMA) [12] and simple moving average (SMA) [13] . Additionally, combinations of local predictors, including support vector machines and ARMA [14] or long short term memory neural networks and autoregressive integrated moving average [15] are employed. While most autoscaling techniques focus on horizontal scaling, vertical scaling [16] and combinations of both [17] [18] are also of great interest. Reference [19] compares the autoscaling services of three cloud provider (Amazon Web Services, Microsoft Azure, Google Cloud Platform) on virtual machine (VM) and container level. To our best knowledge, there has been no research conducted to compare container and application-level autoscaling in an ML-specific task in both local and cloud environments. This paper focuses on the use of well-known server and container orchestration frameworks. These frameworks include ML-specific webservers equipped with autoscaling capabilities, such as TorchServe and RayServe. Additionally, scalable container frameworks like Kubern

FAQs

What problem does the paper address when deploying ML applications?

It addresses how to use autoscaling to keep inference time consistent under fluctuating workloads, while balancing Quality of Service with resource cost.

Which frameworks are compared across local and cloud environments?

The local comparison covers TorchServe and RayServe at the application level and K3s at the container level, while the cloud comparison uses AWS ECS and EKS across container and machine levels.

How is performance evaluated in the comparison?

Inference time stability is measured using the mean and standard deviation across a multi-client scenario, and scaling behavior is assessed using upscaling response times.

Aug 5, 2026

Upload by Violet