[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118773-en":3,"doc-seo-118773-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},118773,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697",8,"Research & Report","Quality Issues in Machine Learning Software Systems - Research on Real-World Quality Issues and Mitigation Strategies","Quality Issues in Machine Learning Software Systems addresses the need to ensure serving quality for deployed ML systems, where false or poor decisions can cause malfunction in other software, significant financial losses, or even threats to human life. The work investigates real-world quality issue characteristics from practitioners’ perspectives, using interviews and validating findings through a survey. The study compiles a catalog of recurring quality issues and corresponding mitigation strategies to support more effective quality assurance tools for ML models and MLSSs.","arXiv :2306 . 15007v1 [ cs . SE] 26 Jun 2023  \nEMSE 2023 manuscript No.  \n(will be inserted by the editor)  \nQuality Issues in Machine Learning Software Systems  \nPierre-Olivier C^ot􀀓e 􀀁 Amin Nikanjam 􀀁  \nRached Bouchoucha 􀀁 Ilan Basta 􀀁  \nMouna Abidi 􀀁 Foutse Khomh  \nReceived: date TODO / Accepted: date  \nAbstract Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are implemented as software components and deployed in Machine Learning Software Systems (MLSSs) . Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, signi􀀌cant 􀀌nancial losses, or even threats to human life. The quality assurance of MLSSs is considered a challenging task and currently is a hot research topic. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of quality issues in MLSSs. Method: We conduct a set of interviews with practitioners/experts, to gather insights about their experience and practices when dealing with quality issues. We validate the identi􀀌ed quality issues via a survey with ML practitioners. Results: Based on the content of 37 interviews, we identi􀀌ed 18 recurring quality issues and 24 strategies to mitigate them. For each identi􀀌ed issue, we describe the causes and consequences according to the practitioners'experience. Conclusion: We believe the catalog of issues developed in this study will allow the community to develop e􀀎cient quality assurance tools for ML models and MLSSs. A replication package of our study is available on our public GitHub repository 1 .  \nKeywords Machine Learning based Software Systems 􀀁 Quality Assurance 􀀁 Quality issues 􀀁 Interview 􀀁 MLOps 􀀁 Machine Learning  \nMathematics Subject Classi􀀌cation (2020) MSC 68T05  \nPierre-Olivier C^ot􀀓e 􀀁 Amin Nikanjam 􀀁 Rached Bouchoucha 􀀁 Ilan Basta 􀀁 Mouna Abidi 􀀁 Foutse Khomh  \nPolytechnique Montr􀀓eal, Qu􀀓ebec, Canada  \nE-mail: fpierre-olivier.cote, amin.nikanjam, rached.bouchoucha, ilan.basta, mouna.abidi, [foutse.khomh](foutse.khomhg@polymtl.ca)[g](foutse.khomhg@polymtl.ca)[@polymtl.ca](foutse.khomhg@polymtl.ca)  \nCorresponding author's e-mail address: [pierre-olivier.cote@polymtl.ca](pierre-olivier.cote@polymtl.ca)  \n1[https://github.com/poclecoqq/quality_issues_in_MLSSs](https://github.com/poclecoqq/quality_issues_in_MLSSs)  \n1 Introduction  \nNowadays, Machine Learning Software Systems (MLSSs) have become a part of our daily life (e.g. recommendation systems, speech recognition, face detection) . Increasing demand is observed in various companies to employ Machine Learning (ML) for solving problems in their business. Typically, an MLSS receives data as input and employs ML models to make intelligent decisions automatically based on learned patterns, associations, and knowledge from data (Marijan et al., 2019) . Therefore, ML models are implemented as software components integrated into other subsystems in MLSSs, and like other software systems, quality assurance is necessary. According to the growing importance of MLSSs in today's world, there is a strong need for ensuring their quality. False or poor decisions of such systems can lead to the malfunction of other systems, signi􀀌cant 􀀌nancial losses, or even threats to human life (Foidland Felderer, 2019) . Recently, an automated vehicle caused a crash of 8 vehicles, leading to 2 juveniles being transported to a hospital (Gua, 2022) . As reported by United States safety regulators, this is not a punctual event, since more than 400 car-crash with automated vehicles have been reported between July 2021 and May 2022 (Krisher, 2022) . Unquali􀀌ed MLSSs can be harmful to society in less obvious ways. For example, an experimental job recruitment tool developed by Amazon was shown to reject women's job applications more of","cbCaiqmj8ZLYWXIn","https://ap.wps.com/l/cbCaiqmj8ZLYWXIn","pdf",716674,1,52,"English","en",105,"# Introduction\n## Motivation and problem statement\n## Research gap and study intent\n# Method\n## Practitioner interviews\n## Survey validation","[{\"question\":\"Why is quality assurance critical for machine learning software systems (MLSSs)?\",\"answer\":\"MLSSs make automated decisions that can malfunction other systems and produce serious consequences, including financial losses and safety threats when decisions are false or poor.\"},{\"question\":\"How were the real-world quality issues identified in this study?\",\"answer\":\"The study conducts interviews with practitioners and experts to capture their experience and practices dealing with ML quality issues.\"},{\"question\":\"What outputs does the paper provide to help mitigate MLSS quality problems?\",\"answer\":\"It identifies recurring quality issues and corresponding mitigation strategies, and describes causes and consequences based on practitioners’ experience, supported by survey validation.\"}]","Quality Issues in Machine Learning Software Systems - Research on Real-World Quality Issues and Mitigation Strategies | PDF",1785720172,131,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"quality-issues-in-machine-learning-software-systems-research-on-real-world-quality-issues-and-mitigation-strategies","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/quality-issues-in-machine-learning-software-systems-research-on-real-world-quality-issues-and-mitigation-strategies/118773/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is quality assurance critical for machine learning software systems (MLSSs)?","Question",{"text":75,"@type":76},"MLSSs make automated decisions that can malfunction other systems and produce serious consequences, including financial losses and safety threats when decisions are false or poor.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How were the real-world quality issues identified in this study?",{"text":80,"@type":76},"The study conducts interviews with practitioners and experts to capture their experience and practices dealing with ML quality issues.",{"name":82,"@type":73,"acceptedAnswer":83},"What outputs does the paper provide to help mitigate MLSS quality problems?",{"text":84,"@type":76},"It identifies recurring quality issues and corresponding mitigation strategies, and describes causes and consequences based on practitioners’ experience, supported by survey validation.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]