[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-504981-105":59,"doc-detail-504981-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","a-framework-for-generating-informative-benchmark-instances","A Framework for Generating Informative Benchmark Instances","","Benchmarking evaluates the relative performance of alternative solving approaches, yet its usefulness is constrained by the limited quantity and quality of available problem instances. Modern constraint programming allows parameterized class-level models over instance data, enabling automated instance generation. This paper introduces a framework that produces many graded and discriminative benchmark instances for informative evaluation, demonstrated on five MiniZinc competition problems and used to analyze solver behavior across the instance space, including performance variance subsets.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/a-framework-for-generating-informative-benchmark-instances/504981/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/a-framework-for-generating-informative-benchmark-instances/504981.png","ImageObject",300,407,{"name":92,"@type":93},"xXChrisXx","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-20","2026-10-04",true,{"@type":102,"interactionType":103,"userInteractionCount":34},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"Why does benchmarking require more than a fixed set of instances?","Question",{"text":112,"@type":113},"Benchmarking can be limited by both the quantity and quality of available instances, which may bias conclusions. Informative benchmarking also needs mechanisms to explore instance subsets where solver performance differs.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"What problem does the AutoIG framework address?",{"text":117,"@type":113},"AutoIG generates benchmark instances that are graded by difficulty and can discriminate between solving approaches. It supports informative evaluation beyond a simple overall ranking.",{"name":119,"@type":110,"acceptedAnswer":120},"How is solver performance analyzed in the proposed approach?",{"text":121,"@type":113},"In addition to ranking solvers, the framework examines solver behavior across the whole instance space by identifying subsets where performance significantly varies from average results.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},504981,1791275850,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":34,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},8796112263567,"https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2","arXiv :2205 . 14753v1 [ cs .AI] 29 May 2022  \nA Framework for Generating Informative Benchmark Instances  \nNguyen Dang !   \nSchool of Computer Science, University of St Andrews, United Kingdom Özgür Akgün !   \nSchool of Computer Science, University of St Andrews, United Kingdom Joan Espasa !   \nSchool of Computer Science, University of St Andrews, United Kingdom Ian Miguel !   \nSchool of Computer Science, University of St Andrews, United Kingdom Peter Nightingale !   \nDepartment of Computer Science, University of York, United Kingdom  \n~~ Abstract ~~  \nBenchmarking is an important tool for assessing the relative performance of alternative solving approaches. However, the utility of benchmarking is limited by the quantity and quality of the available problem instances. Modern constraint programming languages typically allow the specification of a class-level model that is parameterised over instance data. This separation presents an opportunity for automated approaches to generate instance data that define instances that are graded (solvable at a certain difficulty level for a solver) or can discriminate between two solving approaches. In this paper, we introduce a framework that combines these two properties to generate a large number of benchmark instances, purposely generated for effective and informative benchmarking. We use five problems that were used in the MiniZinc competition to demonstrate the usage of our framework. In addition to producing a ranking among solvers, our framework gives a broader understanding of the behaviour of each solver for the whole instance space; for example by finding subsets of instances where the solver performance significantly varies from its average performance.  \n2012 ACM Subject Classification Theory of computation → Constraint and logic programming Keywords and phrases Instance generation, Benchmarking, Constraint Programming Supplementary Material Code: [https://github.com/stacs-cp/AutoIG](https://github.com/stacs-cp/AutoIG)  \nFunding Nguyen Dang: is a Leverhulme Early Career Fellow  \nIan Miguel: supported by EPSRC EP/V027182/1  \nAcknowledgements This work uses the Cirrus UK National Tier-2 HPC Service at EPCC ([http:](http:)//[www.cirrus.ac.uk](www.cirrus.ac.uk) funded by the University of Edinburgh and EPSRC (EP/P020267/1).)[) funded by the University of Edinburgh and EPSRC (EP/P020267/1).](www.cirrus.ac.uk) funded by the University of Edinburgh and EPSRC (EP/P020267/1).)  \n 1  Introduction  \nA practitioner faced with solving a new problem has a difficult choice among many solving algorithms, whose performance on the new problem is unknown and is likely to be variable. One approach is to draw instances from the problem to benchmark the various solvers under consideration, [i.e. an](i.e. an) empirical study of relative performance. This approach is favoured for computationally challenging tasks since the performance behaviour of a non-trivial algorithm is difficult to predict and is unlikely to be susceptible to a purely theoretical analysis [10] . As Beiranvand et al. [11] argue, care must be taken to select an instance set with a variety of difficulty for benchmarking in order to obtain the best insight into solver performance.  \nConstraint programming (CP) approaches particularly benefit from empirical analysis, since modern tool chains like MiniZinc [34] and savilerow [35] support targeting multiple  \n2 A Framework for Generating Informative Benchmark Instances  \nFigure 1 Flowchart of the whole AutoIG application process  \nsolvers from a solver-independent constraint model. These may be entirely different paradigms, such as SAT [13], SMT [9] or indeed CP, and so can vary in performance significantly.  \nThe need for empirical benchmarking is further supported by competitions run by several research communities, like the MiniZinc challenge [44] in the CP community, the SAT competition [18] and the AI planning competition [47] . Solver developers enter a competition by providing a default configuratio","cbCaiivqkrHAki0z","https://ap.wps.com/l/cbCaiivqkrHAki0z","pdf",933067,18,"English","# Introduction\n## Informative benchmarking and instance discrimination\n## AutoIG framework overview","[{\"question\":\"Why does benchmarking require more than a fixed set of instances?\",\"answer\":\"Benchmarking can be limited by both the quantity and quality of available instances, which may bias conclusions. Informative benchmarking also needs mechanisms to explore instance subsets where solver performance differs.\"},{\"question\":\"What problem does the AutoIG framework address?\",\"answer\":\"AutoIG generates benchmark instances that are graded by difficulty and can discriminate between solving approaches. It supports informative evaluation beyond a simple overall ranking.\"},{\"question\":\"How is solver performance analyzed in the proposed approach?\",\"answer\":\"In addition to ranking solvers, the framework examines solver behavior across the whole instance space by identifying subsets where performance significantly varies from average results.\"}]","A Framework for Generating Informative Benchmark Instances | PDF",1791082706,45]