[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121682-en":3,"doc-seo-121682-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121682,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697",8,"Research & Report","Reproducibility in Machine Learning-Driven Research","Research faces a reproducibility crisis where results and findings from many studies are difficult or impossible to reproduce, especially in machine learning and AI. Common obstacles include unavailable data and source code, and strong sensitivity to ML training conditions. Although various solutions such as ML platforms are discussed, reproducibility levels in ML-driven research are not improving sufficiently. This mini survey reviews literature on ML reproducibility with aims to assess the current landscape, identify barriers and issues, and point to drivers like tools, practices, and interventions to support reproducibility.","arXiv :2307 . 10320v 1 [ cs .LG] 19 Jul 2023  \nReproducibility in Machine Learning-Driven  \nResearch  \nHarald Semmelrock2 , Simone Kopeinik 1 , Dieter Theiler 1 , Tony  \nRoss-Hellauer 1 ;2 , and Dominik Kowald (B) 1 ;2  \n1 Know-Center GmbH, Graz, Austria  \nskopeinik,dtheiler,tross,[dkowald@know-center.at](dkowald@know-center.at)  \n2 Graz University of Technology, Graz, Austria  \n[harald.semmelrock@student.tugraz.at](harald.semmelrock@student.tugraz.at)  \nAbstract. Research is facing a reproducibility crisis, in which the results and findings of many studies are difficult or even impossible to reproduce.  \nThis is also the case in machine learning (ML) and artificial intelligence (AI) research. Often, this is the case due to unpublished data and/or source-code, and due to sensitivity to ML training conditions. Although different solutions to address this issue are discussed in the research community such as using ML platforms, the level of reproducibility in MLdriven research is not increasing substantially. Therefore, in this mini survey, we review the literature on reproducibility in ML-driven research with three main aims: (i) reflect on the current situation of ML reproducibility in various research fields,(ii) identify reproducibility issues and barriers that exist in these research fields applying ML, and (iii) identify potential drivers such as tools, practices, and interventions that support ML reproducibility. With this, we hope to contribute to decisions on the viability of different solutions for supporting ML reproducibility.  \nKeywords: Machine Learning · Artificial Intelligence · Reproducibility  \n· Replicability  \n1 Introduction  \nSimilar to other scientific fields [4] [41], research in artificial intelligence (AI) in general, and machine learning (ML) in particular, is facing a reproducibility crisis [24] . Here, especially unpublished source-code and sensitivity to ML training conditions make it nearly impossible to reproduce existing ML publications, which also makes it very hard to verify the claims and findings stated in the publications.  \nOne potential solution for enhancing reproducibility in ML is the use of ML platforms such as OpenML, Google Cloud ML, Microsoft Azure ML or Kaggle. However, in a recent study [21] found that the same experiment executed on different platforms leads to different results. This suggests that still a lot of research is needed until out-of-the-box reproducibility can be provided. However,  \n2 Semmelrock, Kopeinik, Theiler, Ross-Hellauer, and Kowald  \na systematic overview of the literature on ML reproducibility is still missing, especially with respect to the barriers and drivers of reproducibility that can be found in the literature. An example of a driver could be code sharing or hosting reproducibility tracks/challenges at scientific conferences [16] . One example for this is the reproducibility track at the European Conference on Information Retrieval (ECIR) [28,31]  \nWith respect to potential barriers, it is still not clear to what extent the use of ML could even fuel reproducibility issues [17], e.g., via bad ML practices such as data leakage [26] .  \nThis work aims to provide an overview of the situation and identify the different drivers and barriers present. This should allow for a better understanding of the following three aspects:  \n– The situation of ML reproducibility in different research fields (see Section 2) .  \n– Reproducibility issues that exist in research fields applying ML, and the barriers that cause these issues (see Section 3) .  \n– The drivers that support ML reproducibility, including different tools, practices, and interventions (see Section 4) .  \n1.1 Degrees of Reproducibility  \nAccording to [20], there are three degrees of reproducibility in ML, which can  \nbe seen in Table 1 .  \nTable 1: Different degrees of reproducibility according to [20]  \nType Requirement  \n(R1) Experiment Reproducibility The same implementation (including same software versions, hyperparamete","cbCaikjs0vaC05jG","https://ap.wps.com/l/cbCaikjs0vaC05jG","pdf",350496,1,15,"English","en",105,"# Introduction\n## Degrees of Reproducibility","[{\"question\":\"Why does a reproducibility crisis occur in machine learning and AI research?\",\"answer\":\"Many ML results are hard to reproduce due to unpublished data and/or source code, and sensitivity to ML training conditions.\"},{\"question\":\"What does the mini survey aim to accomplish?\",\"answer\":\"It reviews the literature to reflect the current situation across fields, identify reproducibility issues and barriers, and identify drivers such as tools, practices, and interventions.\"},{\"question\":\"What are the three degrees of reproducibility in ML?\",\"answer\":\"They are experiment reproducibility (same implementation produces exactly the same results), data reproducibility (alternative implementations produce almost the same results with the same data), and method reproducibility (alternative implementations on different data produce the same results or at least findings).\"}]","Reproducibility in Machine Learning-Driven Research | PDF",1785806185,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"reproducibility-in-machine-learning-driven-research","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/reproducibility-in-machine-learning-driven-research/121682/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does a reproducibility crisis occur in machine learning and AI research?","Question",{"text":75,"@type":76},"Many ML results are hard to reproduce due to unpublished data and/or source code, and sensitivity to ML training conditions.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What does the mini survey aim to accomplish?",{"text":80,"@type":76},"It reviews the literature to reflect the current situation across fields, identify reproducibility issues and barriers, and identify drivers such as tools, practices, and interventions.",{"name":82,"@type":73,"acceptedAnswer":83},"What are the three degrees of reproducibility in ML?",{"text":84,"@type":76},"They are experiment reproducibility (same implementation produces exactly the same results), data reproducibility (alternative implementations produce almost the same results with the same data), and method reproducibility (alternative implementations on different data produce the same results or at least findings).","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]