Comparison of Bayesian and particle swarm algorithms for hyperparameter optimisation in machine learning applications in high energy physics

Aug 6, 2026

Upload by Himbo

Machine learning workflows require selecting numerous hyperparameters that strongly influence model performance. This paper compares particle swarm optimisation (PSO) and Bayesian optimisation (BO) for autonomous hyperparameter determination across machine learning tasks typical of high energy physics. The evaluation focuses on whether each method can effectively leverage highly parallel computing resources common in contemporary HEP experiments. Results assess performance relative to sequential execution on a single machine.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Introduction

Hyperparameters in ML for HEP

Formulation as a function maximisation problem

Benchmark tasks and parallel evaluation

Bayesian Optimisation

Surrogate model and Gaussian process

Acquisition function and iterative optimisation

Comparison of Bayesian and particle swarm algorithms for hyperparameter optimisation in machine learning applications in high energy physics Tani, Laurits 1 [laurits. tani@cern. ch](laurits. tani@cern. ch) Veelken, Christian 1 [Christian. Veelken@cern. ch](Christian. Veelken@cern. ch) arXiv:2201.06809v2 [[physics.data-an](physics.data-an)] 13 Oct 2023 1 National Institute Of Chemical Physics And Biophysics (NICPB), Akadeemia tee 23, 12618 Tallinn, Estonia Abstract When using machine learning (ML) techniques, users typically need to choose a plethora of algorithmspecific parameters, referred to as hyperparameters. In this paper, we compare the performance of two algorithms, particle swarm optimisation (PSO) and Bayesian optimisation (BO), for the autonomous determination of these hyperparameters in applications to different ML tasks typical for the field of high energy physics (HEP) . Our evaluation of the performance includes a comparison of the capability of the PSO and BO algorithms to make efficient use of the highly parallel computing resources that are characteristic of contemporary HEP experiments. Keywords: hyperparameter optimization; high energy physics; evolutionary algorithms; machine learning 1 Introduction Machine learning (ML) methods often aid the analysis of the vast amounts of data that are produced by contemporary high energy physics (HEP) experiments. The ML algorithms often feature tunable parameters, referred to as hyperparameters, which need to be chosen by the user and often have a significant effect on the algorithm’s performance. In a previous publication [1] we presented two different algorithms, particle swarm optimisation (PSO) and the genetic algorithm, for the autonomous determination of these hyperparameters. In the present paper we compare the performance of the PSO algorithm, the more promising of the two algorithms studied in our previous publication, to the performance of the Bayesian optimisation (BO) [2, 3, 4, 5, 6] algorithm. The latter is widely used for the task of finding optimal hyperparameter values in the context of ML applications since the pioneering work of Refs. [7, 8, 9] . The “asynchronous successive halving algorithm”(ASHA) [10] is an alternative algorithm for optimising the values of hyperparameters, which is popular in the ML community outside the field of HEP. As in our previous publication, we formulate the task of determining the set of optimal hyperparam- eter values as a function maximisation problem. More specifically, given an ML algorithm A, we seek to find a point h in the space H of hyperparameters, such that the performance of the ML algorithm A attains its maximum at this point: ˆh = argmax s (h) , h ∈ H where the objective (or “score”) function (OF) s (h) quantifies the performance of the ML algorithm A, and the point h at which s (h) attains its maximum is denoted by the symbol ˆh . We compare the performance of the PSO and BO algorithms on two benchmark tasks, the task of finding the minimum of the Rosenbrock function [11], and on a typical data analysis task in the field of HEP, the “ATLAS Higgs boson machine learning challenge” [12] . An important aspect in applications of ML algorithms in the field of HEP is an algorithm’s capability to make efficient use of massively parallel computing facilities. A single training of an ML algorithm on a single machine may take several hours, days, or, in extreme cases, even weeks. In the context of the hyperparameter optimisation task, such a single training corresponds to a single evaluation of the OF s(h) . In order for the hyperparameter optimisation task to finish within an acceptable time, different evaluations of the OF, i. e. different ML trainings, need to be executed in parallel. The computing facilities of contemporary HEP experiments typically allow users concurrent access to hundreds, sometimes even thousands, of machines. It is there- fore of high practical relevance whether t

FAQs

What problem does the paper address in machine learning for high energy physics?

It addresses how to autonomously determine ML hyperparameters, which otherwise must be chosen by the user and can significantly affect performance.

Which two optimisation algorithms are compared for hyperparameter tuning?

The paper compares particle swarm optimisation (PSO) and Bayesian optimisation (BO) for hyperparameter optimisation.

How does the evaluation account for the computing environment of HEP experiments?

It compares how efficiently PSO and BO use the massively parallel computing resources typical of contemporary high energy physics experiments, evaluating performance versus fully sequential execution.

Aug 6, 2026

Upload by Himbo