[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-122240-en":3,"doc-seo-122240-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},122240,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","What is Reproducibility in Artificial Intelligence and Machine Learning Research?","The rapid evolution of Artificial Intelligence (AI) and Machine Learning (ML) has intensified the reproducibility crisis, driven not only by complex validation practices but also by widespread confusion in validation terminology. To address this, the work proposes a structured framework that distinguishes repeatability, dependent and independent reproducibility, and direct and conceptual replicability. Clear definitions and roles support researchers in designing, conducting, and interpreting validation studies.","arXiv :2407 . 10239v2 [ cs .CY] 30 Mar 2025  \nWhat is Reproducibility in Artificial Intelligence and Machine  \nLearning Research?  \nAbhyuday Desai 1 , Mohamed Abdelhamid 1 , and Nakul R. Padalkar2  \n1 Ready Tensor, Inc.  \n2 Administrative Sciences, Metropolitan College, Boston University  \nAbstract  \nIn the rapidly evolving fields of Artificial Intelligence (AI) and Machine Learning (ML), the reproducibility crisis underscores the urgent need for clear validation methodologies to maintain scientific integrity and encourage advancement. The crisis is compounded by the prevalent confusion over validation terminology. In response to this challenge, we introduce a framework that clarifies the roles and definitions of key validation efforts: repeatability, dependent and independent reproducibility, and direct and conceptual replicability. This structured framework aims to provide AI/ML researchers with the necessary clarity on these essential concepts, facilitating the appropriate design, conduct, and interpretation of validation studies. By articulating the nuances and specific roles of each type of validation study, we aim to enhance the reliability and trustworthiness of research findings and support the community’s efforts to address reproducibility challenges effectively.  \n1 Introduction  \nThe AI/ML domain has witnessed explosive growth in research publications over the past few years. With major conferences receiving thousands of submissions annually, the sheer volume of research output has made it challenging to ensure consistent reproducibility. Conferences like NeurIPS, ICML, ICLR, and AAAI have seen a significant growth in paper submissions. The number of papers submitted to these four conferences has increased by 169% between 2018 and 2023 . Figure 1 shows the number of articles submitted and accepted at the top conferences.  \nAlthough the number of publications has increased, attempts to reproduce the findings of these papers have often been met with challenges. Numerous studies have highlighted this critical issue in recent years. For example, a survey of 1,576 researchers in the Nature journal indicated that more than 70% of researchers have tried and failed in their attempt to reproduce another researcher’s experiments, and more than half have failed to reproduce their own experiments (Baker, 2016) .  \nCollberg and Proebsting (2016) attempted to execute the code from 601 papers from computer systems research. This study only involved at-  \ntempts to execute the code, not verify the correctness of the published results. These attempts to re-execute the code were divided into three categories: category 1: time to examine a research artifact is limited and communicating with the author is not an option; category 2 involves the scenario where ample time is available, but the lead author is not available for consultation; category 3 represents the case where ample time is available, and the author is available to correspond. They achieved success in 32.3% in category 1 attempts, 48.3% in category 2, and 54 .0% in category 3 .  \nRaff (2019) attempted to reproduce results of 255 papers between 1984 and 2017 with a success rate of 63 .5% . This study involved independent reproduction attempts, where the original authors’code, even if available, was not used.  \nIslam et al. (2017) investigated the reproducibility of benchmarked deep reinforcement learning tasks and found a wide range of results reported in the literature for the same baseline algorithms, highlighting the difficulty of reproduction attempts. These variations were attributed to factors such as external randomness, under-reporting of hyperparameters, and a narrow range of tasks under the benchmark.  \nPham et al. (2020) revealed a striking statistic that highlights reproducibility challenges in deep  \nFigure 1: Total number of papers submitted and accepted at NeurIPS, ICML, ICLR and AAAI from 2018 to 2023  \nlearning research. They reported that for 16 identical ","cbCailcClzmto7mG","https://ap.wps.com/l/cbCailcClzmto7mG","pdf",415094,1,13,"English","en",105,"# Introduction\n## Reproducibility challenges in AI/ML research\n# Terminology Confusion\n## Repeatability, reproducibility, and replicability\n## Evidence from surveys and prior studies","[{\"question\":\"Why is reproducibility a critical problem in AI and machine learning research?\",\"answer\":\"The work highlights that increasing publication volume makes consistent validation harder, and reproduction attempts often fail due to challenges across experiments and reporting practices.\"},{\"question\":\"What framework is proposed to clarify validation concepts?\",\"answer\":\"It introduces a structured set of definitions covering repeatability, dependent and independent reproducibility, and direct and conceptual replicability.\"},{\"question\":\"What causes confusion about reproducibility-related terms?\",\"answer\":\"The document notes multiple existing definitions and inconsistent usage across research papers, including survey evidence showing many researchers cannot accurately differentiate related terms.\"}]","What is Reproducibility in Artificial Intelligence and Machine Learning Research? | PDF",1785809594,33,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"what-is-reproducibility-in-artificial-intelligence-and-machine-learning-research","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/what-is-reproducibility-in-artificial-intelligence-and-machine-learning-research/122240/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is reproducibility a critical problem in AI and machine learning research?","Question",{"text":75,"@type":76},"The work highlights that increasing publication volume makes consistent validation harder, and reproduction attempts often fail due to challenges across experiments and reporting practices.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What framework is proposed to clarify validation concepts?",{"text":80,"@type":76},"It introduces a structured set of definitions covering repeatability, dependent and independent reproducibility, and direct and conceptual replicability.",{"name":82,"@type":73,"acceptedAnswer":83},"What causes confusion about reproducibility-related terms?",{"text":84,"@type":76},"The document notes multiple existing definitions and inconsistent usage across research papers, including survey evidence showing many researchers cannot accurately differentiate related terms.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]