Using machine learning for intelligent shard sizing on the cloud

Aug 4, 2026

Upload by Aurelia

Cloud sharding deployments often rely on conservative approximations to choose the number of cloud instances and shard sizes, which can mismatch real demand and create overloaded deployments. Traditional reactive refinement occurs after load spikes, forcing additional work on an already stressed system. The paper introduces an application-specific machine learning approach using multiple linear regression to predict request latency and determine whether cloud capacity satisfies the service level agreement, enabling accurate shard and server sizing and reducing reactive refinement needs. Experiments on a popular database schema report highly accurate predictions and detailed validation results.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Background and Applicability

Reactive refinement and its drawbacks

Predictive sharding scheme

Quantitative evaluation

Using machine learning for intelligent shard sizing on the cloud Narayanan Venkateswaran1, Anurag Shekhar2, Suvamoy Changder3 and Narayan C. Debnath4 1Department of Computer Science and Engineering, National Institute of Technology Durgapur, India 2MySQL Oracle India Pvt Ltd., India 3Department of Computer Science and Engineering, National Institute of Technology Durgapur, India 4Department of Software Engineering, Eastern International University, Vietnam | Received Dec 31st, 2018 | | --- | | Keyword:
Machine Learning Sharding
Horizontal Partitioning Cloud
Server Sizing Deployment Planning Resource Allocation Data Sizing | Corresponding Author: Sharding implementations use conservative approximations for determining the number of cloud instances required and the size of the shards to be storedon each of them. Conservative approximations are often inaccurate and result in overloaded deployments, which need reactive refinement. Reactive refinement results in demand for additional resources from an already overloaded system and is counterproductive. This paper proposes an algorithm that eliminates the need for conservative approximations and reduces the need for reactive refinement. A multiple linear regression based machine learning algorithm is used to predict the latency of requests for a given application deployed on a cloud machine. The predicted latency helps to decide accurately and with certainty if the capacity of the cloud machine will satisfy the service level agreement for effective operation of the application. Application of the proposed methods on a popular database schema on the cloud resulted in highly accurate predictions. The results of the deployment and the tests performed to establish the accuracy have been presented in detail and are shown to establish the authenticity of the claims. Suvamoy Changder, Department of Computer Science and Engineering, National Institute of Technology Durgapur, Mahatma Gandhi Avenue, Durgapur 713209, West Bengal, INDIA Email: [suvamoy.nitdgp@gmail.com](suvamoy.nitdgp@gmail.com) Article Info ABSTRACT 1. Introduction Any increase in the number of users of an application causes the data stored and used by the application to increase. This raises the demand for storage and processing resources. Since the cloud allows for such ondemand scaling of resources, it is a popular choice for similar applications requiring elastic backends [16] . Sharding topologies split the database into multiple shards and store one or more shards in each cloud instance [11] [18] . Resharding operations, shard splits and merges, are done on demand by leveraging the elasticity provided by the cloud instances. Several datastores with successful and popular sharding implementations have added elastic extensibility on the cloud [6][26] . Sharding implementations distribute the data uniformly across the set of available servers [23] [27] . If the uniform distribution causes workload in excess of server capacity, the amount of data stored is refined reactively. Reactive refinement is often performed after the load spike is detected on a shard. Refining shards imposes an additional workload to the system, and can prove to be inefficient. This paper proposes an application specific, machine learning based partitioning scheme as an alternative for the uniform distribution of data across servers. The methods proposed in this paper allow for accurate server sizing and reduce the need for reactive refinement in an application specific sharding solution. Unlike the existing solutions, the proposed solution attempts to create accurate partitions from the beginning by learning from empirical and existing data. section 2 talks about existing sharding solutions, their use of reactive refinement and the associated problems. section 3 on page 4 proposes the predictive sharding scheme. The proposed method is tested quantitatively in section 4 on page 7 and it is shown that the

FAQs

Why do conservative approximations in cloud sharding lead to overloaded deployments?

They can be inaccurate when mapping the required number of instances and shard sizes to real demand, so the deployment may not match actual workload needs and becomes overloaded.

What problem does reactive refinement introduce after a load spike?

Reactive refinement increases workload on the system already experiencing high load, and it can be inefficient while it requests more resources.

How does the proposed machine learning approach improve shard sizing accuracy?

It uses multiple linear regression to predict request latency for an application on a given cloud machine, helping decide accurately whether capacity will satisfy the service level agreement and reducing the need for reactive refinement.

Aug 4, 2026

Upload by Aurelia