[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117737-en":3,"doc-seo-117737-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":11,"language":21,"language_code":22,"site_id":23,"html_lang":22,"table_of_contents":24,"faqs":25,"seo_title":26,"seo_description":14,"update_tm":27,"read_time":28},117737,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Sources of Irreproducibility in Machine Learning - A Review","Benchmark studies in several machine learning subfields have shown that reported progress may not translate into reproducible advances. The central cause is often irreproducibility in model comparison studies, where known sources of variability are not adequately controlled. This review synthesizes the literature to identify and categorize reported sources of irreproducibility, discusses how they affect verification by independent researchers, and outlines three directions for further inquiry.","Sources of Irreproducibility in Machine Learning: A Review  \nOdd Erik Gundersen 1 􀀃 , Kevin Coakley 1 ;2 and Christine Kirkpatrick2  \n1Norwegian University of Science and Technology, Trondheim, Norway  \n2 San Diego Supercomputer Center, California, USA  \n[odderik@ntnu.no](odderik@ntnu.no), fkcoakley, [christine](christineg@sdsc.edu)[g](christineg@sdsc.edu)[@sdsc.edu](christineg@sdsc.edu)  \narXiv :2204 .07610v1 [ cs .LG] 15 Apr 2022  \nAbstract  \nLately, several benchmark studies have shown that the state of the art in some of the sub-fields of machine learning actually has not progressed despite progress being reported in the literature. The lack of progress is partly caused by the irreproducibility of many model comparison studies. Model comparison studies are conducted that do not control for many known sources of irreproducibility. This leads to results that cannot be verified by third parties. Our objective is to provide an overview of the sources of irreproducibility that are reported in the literature. We review the literature to provide an overview and a taxonomy in addition to a discussion on the identified sources of irreproducibility.  \nFinally, we identify three lines of further inquiry.  \n1 Introduction  \nProgress in machine learning is to a large degree driven by empirical evidence. Model comparison experiments are the standard method to identify the best performing machine learning model for a given task [Melis et al., 2018; Sculley et al., 2018; Bouthillier et al., 2019; Dacrema et al., 2021], and clear wins are typically required for a research paper to be published at a top venue [Sculley et al., 2018] . Wagstaff [2012] proposed to focus research on real world impact challenges such as improve ELO rating of chess engines and discovering a new physical law using machine learning. Although several such impact challenges have been achieved lately, such as in game play, protein folding and software programming, model comparison studies are still the standard. Lately, however, several model comparison studies that benchmark state of the art results have shown that progress isnot as steady as one could get the impression of when reading the scientific literature. Topics covered by such studies include forecasting [Makridakis et al., 2018], natural language processing [Belz et al., 2021a], generative adversarial networks [Lucic et al., 2018], deep reinforcement learning [Henderson et al., 2018], recommender systems [Dacrema et al., 2019], and image recognition [Bouthillier et al., 2019] . The  \n􀀃 Contact Author  \nfindings establish that many state of the art results published at top venues in machine learning are not reproducible.  \nProper methodology requires a good understanding of what can cause irreproducible results. An experiment conducted by Pham et al. [2020] where 16 identical training runs of a deep learning model resulted in test accuracy varying from 8.9% to 99% . According to a survey also done by Pham et al. [2020], 83 . 8% out of 901 participants were unsure or unaware about variance caused by how an experiment is implemented while most, although not all, participants knew about the algorithmic-level influences on variation in performance. This poor understanding of what can cause conclusion to be false in machine learning method development trickles down to the application areas that are downstream and use the methods developed by machine learning researchers, such as medicine [Roberts et al., 2021] .  \nIn the application areas, the stakes can be high and lead to unfortunate situations if relying on unsound evidence. While false findings and irreproducible results are expected products of the scientific method, the method is self-correcting if done correctly. However, methodological care has to be taken to reach sound conclusions when interpreting machine learning results. All too often, the conclusions reached are applied toa scope beyond what the original experiment was poised to suggest. While there are cha","cbCaiv6Dt3UiVrmY","https://ap.wps.com/l/cbCaiv6Dt3UiVrmY","pdf",294801,1,"English","en",105,"# Introduction\n## Model comparison as an empirical driver\n## Why irreproducibility matters\n# Reproducibility\n## Definition and documentation\n## Components of an experiment workflow","[{\"question\":\"What problem does the review address in machine learning benchmarks?\",\"answer\":\"Reported state-of-the-art progress is often not reproducible, especially in model comparison studies, leading to results that third parties cannot verify.\"},{\"question\":\"How does inadequate methodology contribute to irreproducible conclusions?\",\"answer\":\" Model comparison studies may fail to control for many known sources of irreproducibility, so experimental outcomes can vary in ways that are not captured or controlled.\"},{\"question\":\"What does the paper provide beyond identifying causes of irreproducibility?\",\"answer\":\"It proposes a taxonomy and discusses the identified sources of irreproducibility, and it also outlines three lines of further inquiry.\"}]","Sources of Irreproducibility in Machine Learning - A Review | PDF",1785679284,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":23,"language":22,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"sources-of-irreproducibility-in-machine-learning-a-review","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/sources-of-irreproducibility-in-machine-learning-a-review/117737/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":22,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does the review address in machine learning benchmarks?","Question",{"text":74,"@type":75},"Reported state-of-the-art progress is often not reproducible, especially in model comparison studies, leading to results that third parties cannot verify.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does inadequate methodology contribute to irreproducible conclusions?",{"text":79,"@type":75},"Model comparison studies may fail to control for many known sources of irreproducibility, so experimental outcomes can vary in ways that are not captured or controlled.",{"name":81,"@type":72,"acceptedAnswer":82},"What does the paper provide beyond identifying causes of irreproducibility?",{"text":83,"@type":75},"It proposes a taxonomy and discusses the identified sources of irreproducibility, and it also outlines three lines of further inquiry.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":23},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":28,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":28,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]