[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-124502-en":3,"doc-seo-124502-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},124502,13056703020460,"Valentina","https://ap-avatar.wpscdn.com/avatar/be000253dac470eee5d?_k=1778207105932848923",8,"Research & Report","An audit of machine learning experiments on software defect prediction","Machine learning methods are widely proposed for predicting software defects, yet the credibility of reported results depends heavily on experimental design and reporting quality. This study audits software defect prediction experiments from 2019–2023, evaluating experimental design, analysis, and reporting practices against accepted norms. The review assesses evaluation choices such as outcomes, out-of-sample validation regimes, and statistical inference, and measures reproducibility using a dedicated instrument.","An audit of machine learning experiments on software defect prediction  \nGiuseppe Destefanis1 · LeilaYousefi2 · Martin Shepperd2 · Allan Tucker2 · Stephen Swift2 · Steve Counsell2 · Mahir Arzoky2  \nReceived: 18 October 2024 / Accepted: 18 December 2025 © The Author(s) 2025  \nAbstract  \nBackground Machine learning algorithms are increasingly being proposed to solve the problem of predicting defect-prone software components. In this literature, computational experiments are the primary means of evaluating and comparing learners and the credibility of findings depends critically on their experimental design and reporting.  \nObjective This paper audits recent software defect prediction (SDP) experiments by assessing their experimental design, analysis and reporting practices against widely accepted norms from statistics, machine learning and empirical software engineering. Our aim is to characterise the current state of practice and evaluate the reproducibility of published findings. Method We undertook an audit of relevant studies published from the SCOPUS database (2019-2023) focusing on their experimental design and analysis choices e.g., the outcome variables such as F-measure and the type of out of sample (OOS) validation regime, e.g., cross-validation, plus the statistical analysis and inference mechanisms. In all, we evaluated nine different study issues. This was complemented by an assessment of reproducibility using the instrument proposed by González-Barahona and Robles.  \nResults Our search located approximately 1,585 experiments in SDP (2019-2023), a substantial body of work. From this, we randomly sampled 101 (≈ 6.4%) papers, 61 journal and 40 conference papers. Almost 50% are behind ‘paywalls’. We found considerable divergence in research practice. The number of datasets used ranged 1-365, the number of learners or learner variants evaluated from 1-34 and the number of performance metrics from 1 to 9. Approximately 45% of papers made use of formal statistical inference. We detected a total of 427 issues distributed across 101 papers (median=4) with only one paper being entirely issue-free. In terms of reproducibility, experiments ranged from near perfect to lacking almost all required information. We also found two examples of tortured phrasesand potential “paper mill” activity.  \nConclusions Approaches to designing and reporting computational experiments varied greatly, but almost half the studies provided insufficient information such that reproduction would be challenging. Overall, our audit suggests that as a research community, we have considerable scope for improvement. Fortunately, many improvements should be neither difficult nor costly to achieve.  \nCommunicated by: Leandro L. Minku.  \nExtended author information available on the last page of the article  \n1 3  \nKeywords Software defect · Machine learning · Audit · Research review  \n1 Introduction  \n“A variety of recent studies, primarily in the biomedical field, have revealed that an uncomfortably large number of research results found in the literature fail this [quality] test, because of sloppy experimental methods, flawed statistical analyses, or in rare cases, fraud.” ACM Artifact Review and Badging (2020) (ACM 2020) .  \nSadly, there is mounting evidence that software engineering is not immune from such quality problems of questionable research methods and poor reporting. This is compounded by growing activities of “paper mills” and other sources of fake papers (Candal-Pedreira et al. 2022) . While it might seem somewhat negative to focus on problems, we argue that this isan essential step in the journey to improve research practice.  \nAlthough machine learning algorithms are being widely touted as an effective means of classifying software components into those that are likely tobe defect-prone and those that are not, it is proving difficult to obtain an overall picture of what this large and rapidly growing body of research is actually telling us. For","cbCaiinYiS7REboU","https://ap.wps.com/l/cbCaiinYiS7REboU","pdf",2993351,1,36,"English","en",105,"# Abstract\n# Introduction\n## Research quality and reporting concerns\n## Motivation for the audit\n## Contributions and scope","[{\"question\":\"What is the main purpose of this audit?\",\"answer\":\"To assess recent software defect prediction experiments by comparing their experimental design, analysis, and reporting against widely accepted methodological norms and to evaluate reproducibility of published findings.\"},{\"question\":\"Which studies and time range does the audit cover?\",\"answer\":\"The audit covers relevant studies published in the SCOPUS database from 2019 to 2023, focusing on experimental design and analysis choices.\"},{\"question\":\"How is reproducibility evaluated in this work?\",\"answer\":\"Reproducibility is assessed using an instrument proposed by González-Barahona and Robles, examining how much required experimental information is provided.\"}]","An audit of machine learning experiments on software defect prediction | PDF",1785822792,91,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"an-audit-of-machine-learning-experiments-on-software-defect-prediction","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/an-audit-of-machine-learning-experiments-on-software-defect-prediction/124502/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main purpose of this audit?","Question",{"text":75,"@type":76},"To assess recent software defect prediction experiments by comparing their experimental design, analysis, and reporting against widely accepted methodological norms and to evaluate reproducibility of published findings.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which studies and time range does the audit cover?",{"text":80,"@type":76},"The audit covers relevant studies published in the SCOPUS database from 2019 to 2023, focusing on experimental design and analysis choices.",{"name":82,"@type":73,"acceptedAnswer":83},"How is reproducibility evaluated in this work?",{"text":84,"@type":76},"Reproducibility is assessed using an instrument proposed by González-Barahona and Robles, examining how much required experimental information is provided.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]