[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-504960-105":59,"doc-detail-504960-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","a-framework-for-generating-informative-benchmark-instances-article-no-18","A Framework for Generating Informative Benchmark Instances - Article No. 18","","Benchmarking enables reliable comparison of constraint-solving approaches, yet its effectiveness depends on the availability and quality of benchmark instances. Modern constraint programming supports class-level models parameterized by instance data, enabling automated generation of graded instances and instances that discriminate between solver strategies. This paper presents a unified framework that combines both properties to create a large, intentionally designed set of informative benchmarks, demonstrating it using MiniZinc competition problems to reveal ranking and deeper solver behavior across the instance space.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/a-framework-for-generating-informative-benchmark-instances-article-no-18/504960/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/a-framework-for-generating-informative-benchmark-instances-article-no-18/504960.png","ImageObject",300,407,{"name":92,"@type":93},"WPS_1790064749","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-20","2026-10-04",true,{"@type":102,"interactionType":103,"userInteractionCount":44},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What limitation of benchmarking motivates this work?","Question",{"text":112,"@type":113},"Benchmarking’s usefulness is constrained by the quantity and quality of available problem instances, which affects how well solvers can be compared.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does the framework generate benchmark instances?",{"text":117,"@type":113},"It uses parameterized, class-level models to automatically generate instance data, producing both graded (difficulty-targeted) instances and instances that discriminate between solving approaches.",{"name":119,"@type":110,"acceptedAnswer":120},"What additional insight does the framework provide beyond solver rankings?",{"text":121,"@type":113},"It helps analyze solver behavior across the whole instance space, including identifying subsets of instances where performance differs significantly from a solver’s average.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},504960,1791108118,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":44,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},3985747859154,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","A Framework for Generating Informative Benchmark Instances  \nNguyen Dang \\#   \nSchool of Computer Science, University of St Andrews, UK Özgür Akgün \\#   \nSchool of Computer Science, University of St Andrews, UK Joan Espasa \\#   \nSchool of Computer Science, University of St Andrews, UK Ian Miguel \\#   \nSchool of Computer Science, University of St Andrews, UK Peter Nightingale \\#   \nDepartment of Computer Science, University of York, UK  \n~~ Abstract ~~  \nBenchmarking is an important tool for assessing the relative performance of alternative solving approaches. However, the utility of benchmarking is limited by the quantity and quality of the available problem instances. Modern constraint programming languages typically allow the specification of a class-level model that is parameterised over instance data. This separation presents an opportunity for automated approaches to generate instance data that define instances that are graded (solvable at a certain difficulty level for a solver) or can discriminate between two solving approaches. In this paper, we introduce a framework that combines these two properties to generate a large number of benchmark instances, purposely generated for effective and informative benchmarking. We use five problems that were used in the MiniZinc competition to demonstrate the usage of our framework. In addition to producing a ranking among solvers, our framework gives a broader understanding of the behaviour of each solver for the whole instance space; for example by finding subsets of instances where the solver performance significantly varies from its average performance.  \n2012 ACM Subject Classification Theory of computation → Constraint and logic programming Keywords and phrases Instance generation, Benchmarking, Constraint Programming  \nDigital Object Identifier 10.4230/LIPIcs.CP.2022.18  \nSupplementary Material Software (Source Code): [https://github.com/stacs-cp/AutoIG](https://github.com/stacs-cp/AutoIG)  \nFunding This work uses the Cirrus UK National Tier-2 HPC Service at EPCC ([http://www.cirrus](http://www.cirrus) . [ac.uk](ac.uk)) funded by the University of Edinburgh and EPSRC (EP/P020267/1).  \nNguyen Dang: is a Leverhulme Early Career Fellow.  \nIan Miguel: supported by EPSRC EP/V027182/1 .  \n 1  Introduction  \nA practitioner faced with solving a new problem has a difficult choice among many solving algorithms, whose performance on the new problem is unknown and is likely to be variable. One approach is to draw instances from the problem to benchmark the various solvers under consideration, [i.e. an](i.e. an) empirical study of relative performance. This approach is favoured for computationally challenging tasks since the performance behaviour of a non-trivial algorithm is difficult to predict and is unlikely to be susceptible to a purely theoretical analysis [10] . As Beiranvand et al. [11] argue, care must be taken to select an instance set with a variety of difficulty for benchmarking in order to obtain the best insight into solver performance.  \n© Nguyen Dang, Özgür Akgün, Joan Espasa, Ian Miguel, and Peter Nightingale;  \nlicensed under Creative Commons License CC-BY 4.0  \n28th International Conference on Principles and Practice of Constraint Programming (CP 2022) .  \nEditor: Christine Solnon; Article No. 18; pp. 18:1–18:18  \nLeibniz International Proceedings in Informatics  \n Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Germany  \n18:2 A Framework for Generating Informative Benchmark Instances  \nFigure 1 Flowchart of the whole AutoIG application process.  \nConstraint programming (CP) approaches particularly benefit from empirical analysis, since modern tool chains like MiniZinc [34] and savilerow [35] support targeting multiple solvers from a solver-independent constraint model. These may be entirely different paradigms, such as SAT [13], SMT [9] or indeed CP, and so can vary in performance significantly.  \nThe need for empirical benchmarking is further s","cbCaimCpnK2q6SEm","https://ap.wps.com/l/cbCaimCpnK2q6SEm","pdf",1119910,18,"English","# Introduction\n## Benchmarking and the need for informative instances\n## Competition setup and solver ranking\n## Motivation for identifying performance-varying instance subsets","[{\"question\":\"What limitation of benchmarking motivates this work?\",\"answer\":\"Benchmarking’s usefulness is constrained by the quantity and quality of available problem instances, which affects how well solvers can be compared.\"},{\"question\":\"How does the framework generate benchmark instances?\",\"answer\":\"It uses parameterized, class-level models to automatically generate instance data, producing both graded (difficulty-targeted) instances and instances that discriminate between solving approaches.\"},{\"question\":\"What additional insight does the framework provide beyond solver rankings?\",\"answer\":\"It helps analyze solver behavior across the whole instance space, including identifying subsets of instances where performance differs significantly from a solver’s average.\"}]","A Framework for Generating Informative Benchmark Instances - Article No. 18 | PDF",1791081499,45]