[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121298-en":3,"doc-seo-121298-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":11,"language":21,"language_code":22,"site_id":23,"html_lang":22,"table_of_contents":24,"faqs":25,"seo_title":26,"seo_description":14,"update_tm":27,"read_time":28},121298,1099514067415,"Rowan","https://ap-avatar.wpscdn.com/avatar/100002539d78ffe74a7?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779092875211072502",8,"Research & Report","A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning - Research paper","The paper formulates machine-learning hyperparameter tuning as a bilevel optimization problem, where the upper level seeks optimal hyperparameters and the lower level optimizes model parameters via the training objective. A micro genetic algorithm explores discrete hyperparameters, while a linear-programming enhancement enables efficient hyper-local search over continuous hyperparameters. The work contributes a linear program formulation that can be integrated into general hyperparameter search techniques and applied for fine-tuning. Experiments on MNIST and CIFAR-10 show clear performance promise when the linear enhancement augments population-based methods.","A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning  \nAnkur Sinha, IEEE Senior Member  \nBrij Disa Centre for Data Science and Artificial Intelligence Indian Institute of Management Ahmedabad Gujarat, India 380015 [asinha@iima.ac.in](asinha@iima.ac.in)  \nParitosh Pankaj  \nDepartment of Statistics and Data Science Indian Institute of Technology Kanpur Uttar Pradesh, India 208016 [ppankaj21@iitk.ac.in](ppankaj21@iitk.ac.in)  \narXiv :2407 .00613v1 [ cs .LG] 30 Jun 2024  \nAbstract—In this paper, we formulate the hyperparameter tuning problem in machine learning as a bilevel program. The bilevel program is solved using a micro genetic algorithm that is enhanced with a linear program. While the genetic algorithm searches over discrete hyperparameters, the linear program enhancement allows hyper local search over continuous hyperparameters. The major contribution in this paper is the formulation of a linear program that supports fast search over continuous hyperparameters, and can be integrated with any hyperparameter search technique. It can also be applied directly on any trained machine learning or deep learning model for the purpose of fine-tuning. We test the performance of the proposed approach on two datasets, MNIST and CIFAR-10. Our results clearly demonstrate that using the linear program enhancement offers significant promise when incorporated with any populationbased approach for hyperparameter tuning.  \nIndex Terms—Bilevel optimization, genetic algorithms, machine learning, hyperparameter tuning, linear program.  \nI. INTRODUCTION  \nHyperparameter optimization is an incredibly challenging task in machine learning, as hyperparameters are external to the model and can’t be determined based on the training data alone. These common hyperparameters include, network architecture (for example, number of layers and number of neurons per layer), optimization parameters (for example, learning rate and momentum), and regularization parameters (for example, weight decay and dropout) . The most common approach to identify the right set of hyperparameters involves training models with different hyperparameters on the training dataset and then evaluating the models on the validation dataset. The best performing hyperparameters are chosen.  \nThe hyperparameter optimization problem is intrinsically a bilevel optimization task where the upper level problem searches for the optimal hyperparameters and the lower level problem searches for the optimal model parameters for the corresponding hyperparameters. In the context of evolutionary algorithms as well a number of algorithm hyperparameters have to be tuned and their optimal choice can be made using abilevel optimization approach. Formulating the hyperparameter optimization problem as a bilevel optimization task is a familiar approach in machine learning [6] and also in evolutionary computation [32] .  \nA bilevel optimization problem involves two levels of optimization with each level having its own objective function, set of variables, and set of constraints. A large body of literature exists on bilevel optimization for which the readers may refer to [4], [10],[30] . A bilevel optimization problem is challenging because the upper level variables appear as parameters in the lower level optimization problem, while the lower level problem has to be optimized with respect to the lower level variables. Solving the lower level optimization problem for a given set of upper level variables and ensuring that the upper level constraints are satisfied lead to a feasible solution to the bilevel optimization problem. Linear bilevel programs [13], [37], [38] and quadratic bilevel problems [1], [11] are widely solved using Karush-Kuhn-Tucker based approach. Researchers have looked at other approaches based on gradients [26], penalty [17], [34], [39], trust-region [8], [24], among others, to solve bilevel optimization problems. In the domain of evolutionary algorith","cbCaim571rqUCkJe","https://ap.wps.com/l/cbCaim571rqUCkJe","pdf",518178,1,"English","en",105,"# Abstract\n# Introduction\n## Hyperparameter optimization as bilevel learning\n## Bilevel optimization structure and challenges\n# Problem formulation and hyperparameter sets\n## Upper-level and lower-level objectives\n## Discrete and continuous hyperparameters\n# Proposed method overview\n## Micro genetic algorithm for discrete search\n## Linear programming for hyper-local continuous search","[{\"question\":\"How does the paper model hyperparameter tuning in machine learning?\",\"answer\":\"It formulates hyperparameter tuning as a bilevel program: the upper level minimizes validation loss over hyperparameters, while the lower level minimizes training loss over model parameters given those hyperparameters.\"},{\"question\":\"What role does the enhanced genetic algorithm play?\",\"answer\":\"The micro genetic algorithm performs the search over discrete hyperparameters, using evolutionary exploration to propose candidate architectures and related discrete settings.\"},{\"question\":\"Why is linear programming added to the genetic algorithm approach?\",\"answer\":\"The linear-programming enhancement enables fast hyper-local search over continuous hyperparameters, providing efficient refinement beyond discrete evolutionary sampling.\"}]","A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning - Research paper | PDF",1785734965,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":23,"language":22,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"a-linear-programming-enhanced-genetic-algorithm-for-hyperparameter-tuning-in-machine-learning-research-paper","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/a-linear-programming-enhanced-genetic-algorithm-for-hyperparameter-tuning-in-machine-learning-research-paper/121298/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":22,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"How does the paper model hyperparameter tuning in machine learning?","Question",{"text":74,"@type":75},"It formulates hyperparameter tuning as a bilevel program: the upper level minimizes validation loss over hyperparameters, while the lower level minimizes training loss over model parameters given those hyperparameters.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What role does the enhanced genetic algorithm play?",{"text":79,"@type":75},"The micro genetic algorithm performs the search over discrete hyperparameters, using evolutionary exploration to propose candidate architectures and related discrete settings.",{"name":81,"@type":72,"acceptedAnswer":82},"Why is linear programming added to the genetic algorithm approach?",{"text":83,"@type":75},"The linear-programming enhancement enables fast hyper-local search over continuous hyperparameters, providing efficient refinement beyond discrete evolutionary sampling.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":23},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":28,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":28,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]