Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources - A Hierarchical Agent-based Approach

Aug 2, 2026

Algorithm selection and hyperparameter tuning are essential to academic and applied machine learning, yet growing numbers of diverse, distributed resources make these steps increasingly complex. The paper presents a fully automatic, collaborative agent-based mechanism that selects distributedly organized machine learning algorithms and tunes their hyperparameters simultaneously. It extends an existing hierarchical agent-based platform by augmenting its query structure to support selection and tuning without restricting the underlying mechanisms. The study combines theoretical assessment, formal verification, and analysis, showing correctness, efficient resource utilization, and linear time/space complexity relative to available resources, validated through experiments on 24 algorithms and 9 datasets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages32

Reading Time1 h 21 min

AccessFree to download

Abstract

1 Introduction

Background and motivation

Related work (algorithm selection, tuning, CASH)

2 Multi-Agent Systems (MAS)

arXiv :2309 .06604v2 [ cs .LG] 14 Sep 2023 Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources: A Hierarchical Agent-based Approach Ahmad Esmaeili*, Julia T. Rayz and Eric T. Matson Department of Computer and Information Technology, Purdue University, 401 N. Grant St., West Lafayette, IN, 47907, USA. *Corresponding author(s). E-mail(s): [aesmaei@purdue.edu](aesmaei@purdue.edu) ; Contributing [authors: jtaylor1@purdue.edu](authors: jtaylor1@purdue.edu); [ematson@purdue.edu](ematson@purdue.edu) ; Abstract Algorithm selection and hyperparameter tuning are critical steps in both academic and applied machine learning. On the other hand, these steps are becoming ever increasingly delicate due to the extensive rise in the number, diversity, and distributedness of machine learning resources. Multi-agent systems, when applied to the design of machine learning platforms, bring about several distinctive characteristics such as scalability, flexibility, and robustness, just to name a few. This paper proposes a fully automatic and collaborative agent-based mechanism for selecting distributedly organized machine learning algorithms and simultaneously tuning their hyperparameters. Our method builds upon an existing agent-based hierarchical machine-learning platform and augments its query structure to support the aforementioned functionalities without being limited to specific learning, selection, and tuning mechanisms. We have conducted theoretical assessments, formal verification, and analytical study to demonstrate the correctness, resource utilization, and computational efficiency of our technique. According to the results, our solution is totally correct and exhibits linear time and space complexity in relation to the size of available resources. To provide concrete examples of how the proposed methodologies can effectively adapt and perform across a range of algorithmic options and datasets, we have also conducted a series of experiments using a system comprised of 24 algorithms and 9 datasets. Keywords: Multi-agent Systems, Distributed Machine Learning, Hyperparameter Tuning, Algorithm Selection 1 1 Introduction The last decades have witnessed a significant surge in the volume and diversity of the Machine Learning (ML) algorithms and datasets provided by multi-disciplinary research communities. Fueled by both the abundance of inexpensive yet powerful computational units and the increasing necessity of using ML-based solutions in every-day applications, such a fast-paced growth has introduced new challenges with respect to the organization, sharing, and selection of ML resources on one hand and automatically optimizing the generalization capability of the learning models on the other. These challenges have become even more compounded by the requirement for efficient parallel approaches to handle locally and geographically distributed ML algorithm and dataset portfolios [1] . The ML literature contains numerous research studies focusing on efficient and effective methods for algorithm selection and hyperparameter optimization. The modelselection approaches reported in [2–7] and the hyperparameter tuning methods proposed in [8–14] are some noteworthy examples that address these problems separately. Additionally, there are studies that treat algorithm selection and hyperparameter optimization as a combined problem known as Combined Algorithm Selection and Hyperparameter (CASH) optimization problem. Coining the term “CASH”, Thornton et al. reported the first prominent research in this area and contributed an automatic tool called Auto-WEKA [15] . Utilizing the full range of classification algorithms provided by WEKA [16], the proposed tool employs the Bayesian Optimization (BO) techniques such as Sequential Model-based Algorithm Configuration (SMAC) [17] and Tree-structured Parzen Estimator (TPE) [10] to automatically explore the search space comprising the algorithms and their

FAQs

What problem does the paper address?

It addresses jointly selecting distributedly organized machine learning algorithms and tuning their hyperparameters under the growing complexity of distributed ML resources.

How does the proposed approach work at a high level?

It uses a hierarchical agent-based multi-agent mechanism that extends an existing agent-based machine-learning platform by augmenting its query structure to support both selection and hyperparameter tuning collaboratively.

How are the results evaluated?

The paper uses theoretical assessments, formal verification, analytical study, and experiments with a system comprising 24 algorithms and 9 datasets.

Aug 2, 2026

Upload by Lucas Martin