A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning - Research paper

Aug 3, 2026

Upload by Rowan

The paper formulates machine-learning hyperparameter tuning as a bilevel optimization problem, where the upper level seeks optimal hyperparameters and the lower level optimizes model parameters via the training objective. A micro genetic algorithm explores discrete hyperparameters, while a linear-programming enhancement enables efficient hyper-local search over continuous hyperparameters. The work contributes a linear program formulation that can be integrated into general hyperparameter search techniques and applied for fine-tuning. Experiments on MNIST and CIFAR-10 show clear performance promise when the linear enhancement augments population-based methods.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Abstract

Introduction

Hyperparameter optimization as bilevel learning

Bilevel optimization structure and challenges

Problem formulation and hyperparameter sets

Upper-level and lower-level objectives

Discrete and continuous hyperparameters

Proposed method overview

Micro genetic algorithm for discrete search

Linear programming for hyper-local continuous search

A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning Ankur Sinha, IEEE Senior Member Brij Disa Centre for Data Science and Artificial Intelligence Indian Institute of Management Ahmedabad Gujarat, India 380015 [asinha@iima.ac.in](asinha@iima.ac.in) Paritosh Pankaj Department of Statistics and Data Science Indian Institute of Technology Kanpur Uttar Pradesh, India 208016 [ppankaj21@iitk.ac.in](ppankaj21@iitk.ac.in) arXiv :2407 .00613v1 [ cs .LG] 30 Jun 2024 Abstract—In this paper, we formulate the hyperparameter tuning problem in machine learning as a bilevel program. The bilevel program is solved using a micro genetic algorithm that is enhanced with a linear program. While the genetic algorithm searches over discrete hyperparameters, the linear program enhancement allows hyper local search over continuous hyperparameters. The major contribution in this paper is the formulation of a linear program that supports fast search over continuous hyperparameters, and can be integrated with any hyperparameter search technique. It can also be applied directly on any trained machine learning or deep learning model for the purpose of fine-tuning. We test the performance of the proposed approach on two datasets, MNIST and CIFAR-10. Our results clearly demonstrate that using the linear program enhancement offers significant promise when incorporated with any populationbased approach for hyperparameter tuning. Index Terms—Bilevel optimization, genetic algorithms, machine learning, hyperparameter tuning, linear program. I. INTRODUCTION Hyperparameter optimization is an incredibly challenging task in machine learning, as hyperparameters are external to the model and can’t be determined based on the training data alone. These common hyperparameters include, network architecture (for example, number of layers and number of neurons per layer), optimization parameters (for example, learning rate and momentum), and regularization parameters (for example, weight decay and dropout) . The most common approach to identify the right set of hyperparameters involves training models with different hyperparameters on the training dataset and then evaluating the models on the validation dataset. The best performing hyperparameters are chosen. The hyperparameter optimization problem is intrinsically a bilevel optimization task where the upper level problem searches for the optimal hyperparameters and the lower level problem searches for the optimal model parameters for the corresponding hyperparameters. In the context of evolutionary algorithms as well a number of algorithm hyperparameters have to be tuned and their optimal choice can be made using abilevel optimization approach. Formulating the hyperparameter optimization problem as a bilevel optimization task is a familiar approach in machine learning [6] and also in evolutionary computation [32] . A bilevel optimization problem involves two levels of optimization with each level having its own objective function, set of variables, and set of constraints. A large body of literature exists on bilevel optimization for which the readers may refer to [4], [10],[30] . A bilevel optimization problem is challenging because the upper level variables appear as parameters in the lower level optimization problem, while the lower level problem has to be optimized with respect to the lower level variables. Solving the lower level optimization problem for a given set of upper level variables and ensuring that the upper level constraints are satisfied lead to a feasible solution to the bilevel optimization problem. Linear bilevel programs [13], [37], [38] and quadratic bilevel problems [1], [11] are widely solved using Karush-Kuhn-Tucker based approach. Researchers have looked at other approaches based on gradients [26], penalty [17], [34], [39], trust-region [8], [24], among others, to solve bilevel optimization problems. In the domain of evolutionary algorith

FAQs

How does the paper model hyperparameter tuning in machine learning?

It formulates hyperparameter tuning as a bilevel program: the upper level minimizes validation loss over hyperparameters, while the lower level minimizes training loss over model parameters given those hyperparameters.

What role does the enhanced genetic algorithm play?

The micro genetic algorithm performs the search over discrete hyperparameters, using evolutionary exploration to propose candidate architectures and related discrete settings.

Why is linear programming added to the genetic algorithm approach?

The linear-programming enhancement enables fast hyper-local search over continuous hyperparameters, providing efficient refinement beyond discrete evolutionary sampling.

Aug 3, 2026

Upload by Rowan