[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-505059-105":59,"doc-detail-505059-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","comprehensive-algorithm-portfolio-evaluation-using-item-response-theory","Comprehensive Algorithm Portfolio Evaluation using Item Response Theory","","Item Response Theory (IRT), originally developed in educational psychometrics to estimate student ability and test-item difficulty and discrimination, is adapted for machine learning algorithm assessment. The paper introduces a modified IRT-based framework to evaluate a portfolio of algorithms over repositories of datasets, while also eliciting additional algorithm-performance characteristics such as consistency and anomalousness. The method derives these insights via an inversion and reinterpretation of the traditional IRT model without extra dataset feature computation, and demonstrates broad applicability.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/comprehensive-algorithm-portfolio-evaluation-using-item-response-theory/505059/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/comprehensive-algorithm-portfolio-evaluation-using-item-response-theory/505059.png","ImageObject",300,407,{"name":92,"@type":93},"Aldword","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-16","2026-10-04",true,{"@type":102,"interactionType":103,"userInteractionCount":34},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What problem does this paper address in algorithm evaluation?","Question",{"text":112,"@type":113},"It addresses the limitations of evaluating only a small number of algorithms on a limited set of test problems, which does not reveal where algorithms fit within a modern portfolio’s capabilities. The work aims to better understand strengths, weaknesses, and performance drivers across diverse datasets.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How is Item Response Theory adapted for machine learning algorithms?",{"text":117,"@type":113},"Algorithms are treated like “students,” while individual observations to be classified are treated like “test questions.” A modified IRT-based framework fits parameters to infer discrimination and difficulty effects and to characterize algorithm performance.",{"name":119,"@type":110,"acceptedAnswer":120},"What additional characteristics does the framework extract beyond standard IRT parameters?",{"text":121,"@type":113},"It elicits a richer suite of characteristics describing important aspects of algorithm performance, including algorithm consistency and anomalousness. These arise from a novel inversion and reinterpretation of the traditional IRT model without requiring extra dataset feature computations.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},505059,1791259066,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":34,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},2336478940917,"https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8","arXiv :2307 . 15850v1 [ stat .ML] 29 Jul 2023  \nComprehensive Algorithm Portfolio Evaluation using Item Response Theory  \nSevvandi Kandanaarachchi 1 , Kate Smith-Miles2  \n1 CSIRO’s Data61, Clayton, VIC 3168, Australia  \n2 School of Mathematics and Statistics, University of Melbourne, Parkville, VIC 3010, Australia.  \nAugust 1, 2023  \nAbstract  \nItem Response Theory (IRT) has been proposed within the field of Educational Psychometrics to assess student ability as well as test question difficulty and discrimination power. More recently, IRT has been applied to evaluate machine learning algorithm performance on a single classification dataset, where the student is now an algorithm, and the test question is an observation to be classified by the algorithm. In this paper we present a modified IRT-based framework for evaluating a portfolio of algorithms across a repository of datasets, while simultaneously eliciting a richer suite of characteristics - such as algorithm consistency and anomalousness -that describe important aspects of algorithm performance. These characteristics arise from a novel inversion andreinterpretation of the traditional IRT model without requiring additional dataset feature computations. We test this framework on algorithm portfolios for a wide range of applications, demonstrating the broad applicability of this method as an insightful algorithm evaluation tool. Furthermore, the explainable nature of IRT parameters yield an increased understanding of algorithm portfolios.  \nKey words—Item Response Theory, algorithm evaluation, algorithm portfolios, classification, machine learning, algorithm selection, instance space analysis, explainable algorithm evaluation.  \n1 Introduction  \nEvaluating a diverse set algorithms across a comprehensive set of test problems contributes to an increased understanding of the interplay between test problem characteristics, algorithm mechanisms and algorithm performance. Such an evaluation helps determine an algorithm’s strengths and weaknesses, and provides a broad overview of the collective capabilities of an algorithm portfolio. The drawback of many studies that evaluate only a small number of algorithms on a limited set of test problems is that they fail to reveal where any algorithm belongs within a state-of-the-art algorithm portfolio’s capabilities, or where the unique strengthsand weaknesses of algorithms lie considering a diverse range of test problem difficulties and  \nchallenges. After several decades of calls for a more ”empirical science” of algorithm testing (Hooker 1994, 1995), research communities in many fields are now pulling together the components needed for rigorous evaluation of algorithms-open source algorithms and shared test problem repositories-that provide the foundation for new methodologies for empirical evaluations (McGeoch 1996, Hall & Posner 2010, Smith-Miles et al. 2014, Casalicchio et al. 2019, Bischl et al. 2016) .  \nIn this paper we present a framework that evaluates a portfolio of algorithms based on a novel adaptation of Item Response Theory (IRT) . The general premise of IRT is that thereis a hidden “quality” or a trait, such as verbal or mathematical ability, that cannot be directly measured (Hambleton & Swaminathan 2013) but can be inferred from responses to well-designed test questions that are suitably difficult and discriminating. A test instrument such as a questionnaire or an exam containing test items is used to capture participant responses. Using the participant responses to the test items, an IRT model is fitted to estimate the discrimination and difficulty of test items and the ability of participants. In an educational setting, the ability relates to the knowledge of the subject matter tested on the exam; the discrimination of test items inform us which items are better at discriminating between strong and weak students; and the difficulty parameters indicate the difficulty of each test item given the response profile from the p","cbCaiu1OOVa5o3QK","https://ap.wps.com/l/cbCaiu1OOVa5o3QK","pdf",1429568,55,"English","# Abstract\n# Key words\n# 1 Introduction","[{\"question\":\"What problem does this paper address in algorithm evaluation?\",\"answer\":\"It addresses the limitations of evaluating only a small number of algorithms on a limited set of test problems, which does not reveal where algorithms fit within a modern portfolio’s capabilities. The work aims to better understand strengths, weaknesses, and performance drivers across diverse datasets.\"},{\"question\":\"How is Item Response Theory adapted for machine learning algorithms?\",\"answer\":\"Algorithms are treated like “students,” while individual observations to be classified are treated like “test questions.” A modified IRT-based framework fits parameters to infer discrimination and difficulty effects and to characterize algorithm performance.\"},{\"question\":\"What additional characteristics does the framework extract beyond standard IRT parameters?\",\"answer\":\"It elicits a richer suite of characteristics describing important aspects of algorithm performance, including algorithm consistency and anomalousness. These arise from a novel inversion and reinterpretation of the traditional IRT model without requiring extra dataset feature computations.\"}]","Comprehensive Algorithm Portfolio Evaluation using Item Response Theory | PDF",1791087810,139]