[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-290428-105":59,"doc-detail-290428-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","towards-inferential-reproducibility-of-machine-learning-research","TOWARDS INFERENTIAL REPRODUCIBILITY OF MACHINE LEARNING RESEARCH","","Machine learning evaluation reliability depends on the consistency of observed evaluation scores across replicated training runs, but nondeterminism introduces measurement-noise effects. Existing reproducibility efforts often remove noise while ignoring implementation-level nondeterminism and key interactions between algorithmic noise factors and data properties, limiting conclusions from such experiments. The work proposes incorporating multiple variance sources and their interactions with data properties into significance and reliability analysis. Linear mixed effects models and a generalized likelihood ratio test enable inference beyond particular trained models, while variance component analysis yields a reliability coefficient based on substantial versus total variance.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/towards-inferential-reproducibility-of-machine-learning-research/290428/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/towards-inferential-reproducibility-of-machine-learning-research/290428.png","ImageObject",300,407,{"name":92,"@type":93},"Ivy","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-27","2026-09-17",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",13,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What problem does the paper address in machine learning evaluation?","Question",{"text":113,"@type":114},"It addresses how nondeterminism affects the reliability of evaluation scores across replicated training runs.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"Why do the authors criticize current reproducibility practices?",{"text":118,"@type":114},"They argue that removing noise for reproducibility often overlooks irreducible implementation-level nondeterminism and ignores important interactions between algorithmic noise factors and data properties.",{"name":120,"@type":111,"acceptedAnswer":121},"How do the authors propose analyzing reliability instead of eliminating noise?",{"text":122,"@type":114},"They propose incorporating variance sources and their interactions with data properties into statistical analysis, using linear mixed effects models and a generalized likelihood ratio test, plus variance component analysis to compute reliability coefficients.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},290428,1789641671,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":145,"preview_page_count":4},549758252649,"https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819","TOWARDS INFERENTIAL REPRODUCIBILITY OF MACHINE LEARNING RESEARCH  \nMichael Hagmann 1 , Philipp Meier 1 , Stefan Riezler 1 ;2 Computational Linguistics 1 & IWR2  \nHeidelberg University, Germany  \nfhagmann,meier,riezlerg@cl.uni-heidelberg .de  \narXiv :2302 .04054v3 [ cs .LG] 16 Feb 2023  \nABSTRACT  \nReliability of machine learning evaluation—the consistency of observed evaluation scores across replicated model training runs—is affected by several sources of nondeterminism which can be regarded as measurement noise. Current tendencies to remove noise in order to enforce reproducibility of research results neglect inherent nondeterminism at the implementation level and disregard crucial interaction effects between algorithmic noise factors and data properties. This limits the scope of conclusions that can be drawn from such experiments. Instead of removing noise, we propose to incorporate several sources of variance, including their interaction with data properties, into an analysis of significance and reliability of machine learning evaluation, with the aim to draw inferences beyond particular instances of trained models. We show how to use linear mixed effects models (LMEMs) to analyze performance evaluation scores, and to conduct statistical inference with a generalized likelihood ratio test (GLRT) . This allows us to incorporate arbitrary sources of noise like meta-parameter variations into statistical significance testing, and to assess performance differences conditional on data properties. Furthermore, a variance component analysis (VCA) enables the analysis of the contribution of noise sources to overall variance and the computation of a reliability coefficient by the ratio of substantial to total variance.  \n1 INTRODUCTION  \nTraining of deep learning models utilizes randomness to improve generalization and training efficiency, thus causing an inherent nondeterminism that hampers the reliability of machine learning evaluation—the consistency of the measurement of evaluation scores across replicated training runs. Gundersen et al. (2022) list several sources of nondeterminism, e.g., implementation-level nondeterminism such as random ordering in floating-point accumulation in parallel GPU threads (Pham et al., 2021), algorithmic factors such as variations in meta-parameters and model architecture (Lucic et al., 2018; Henderson et al., 2018; D'Amour et al., 2020), or data-level factors such as variations in pre-processing and evaluation metrics (Post, 2018; Chen et al., 2022) or varying characteristics of data in different splits (Gorman & Bedrick, 2019; Søgaard et al., 2021) . Zhuang et al.(2022) show that implementation-level nondeterminism is partly irreducible, leading to variability in evaluation scores even for training runs on identical data, algorithmic settings and infrastructure. Furthermore, they point out strong effects of certain types of algorithm-level nondeterminism on certain subsets of the data.  \nRegarding the comparison of machine learning models, minor variations in these sources of nondeterminism can have huge impact on the resulting evaluation scores and sometimes even reverse the relation between optimal results for baseline and state-of-the-art (SOTA) model (Reimers & Gurevych, 2017; Melis et al., 2018) . This fact questions what can be validly learned from a typical machine learning experiment. One current answer is to foster training reproducibility 1 in the sense of an exact duplication of a state-of-the-art (SOTA) training result under exactly the same conditions. In this view, all sources of nondeterminism are regarded as noise or nuisance factors  \n1The term was coined by Leventi-Peetz & ¨Ostreich (2022) and corresponds to Drummond (2009)'s replicability.  \n(Forde & Paganini, 2019) that are independent of the learning signal and need to be removed (or at least reduced, even if incurring a cost in efficiency (Ahn et al., 2022)) . This goal is pursued by enforcing open-source program code, publicly avai","cbCaidlMwYKch88w","https://ap.wps.com/l/cbCaidlMwYKch88w","pdf",425743,15,"English","# Abstract\n# Introduction\n## Nondeterminism as a source of evaluation variance\n## Limits of strict training reproducibility\n## Inferential reproducibility and the proposed framework\n# Methods and contributions\n## Linear mixed effects models and GLRT\n## Variance component analysis and reliability coefficients","[{\"question\":\"What problem does the paper address in machine learning evaluation?\",\"answer\":\"It addresses how nondeterminism affects the reliability of evaluation scores across replicated training runs.\"},{\"question\":\"Why do the authors criticize current reproducibility practices?\",\"answer\":\"They argue that removing noise for reproducibility often overlooks irreducible implementation-level nondeterminism and ignores important interactions between algorithmic noise factors and data properties.\"},{\"question\":\"How do the authors propose analyzing reliability instead of eliminating noise?\",\"answer\":\"They propose incorporating variance sources and their interactions with data properties into statistical analysis, using linear mixed effects models and a generalized likelihood ratio test, plus variance component analysis to compute reliability coefficients.\"}]","TOWARDS INFERENTIAL REPRODUCIBILITY OF MACHINE LEARNING RESEARCH | PDF",38]