Quality Issues in Machine Learning Software Systems - Research on Real-World Quality Issues and Mitigation Strategies

Aug 3, 2026

Upload by Himbo

Quality Issues in Machine Learning Software Systems addresses the need to ensure serving quality for deployed ML systems, where false or poor decisions can cause malfunction in other software, significant financial losses, or even threats to human life. The work investigates real-world quality issue characteristics from practitioners’ perspectives, using interviews and validating findings through a survey. The study compiles a catalog of recurring quality issues and corresponding mitigation strategies to support more effective quality assurance tools for ML models and MLSSs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages52

Reading Time2 h 11 min

AccessFree to download

Introduction

Motivation and problem statement

Research gap and study intent

Method

Practitioner interviews

Survey validation

arXiv :2306 . 15007v1 [ cs . SE] 26 Jun 2023 EMSE 2023 manuscript No. (will be inserted by the editor) Quality Issues in Machine Learning Software Systems Pierre-Olivier C^ot􀀓e 􀀁 Amin Nikanjam 􀀁 Rached Bouchoucha 􀀁 Ilan Basta 􀀁 Mouna Abidi 􀀁 Foutse Khomh Received: date TODO / Accepted: date Abstract Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are implemented as software components and deployed in Machine Learning Software Systems (MLSSs) . Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, signi􀀌cant 􀀌nancial losses, or even threats to human life. The quality assurance of MLSSs is considered a challenging task and currently is a hot research topic. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of quality issues in MLSSs. Method: We conduct a set of interviews with practitioners/experts, to gather insights about their experience and practices when dealing with quality issues. We validate the identi􀀌ed quality issues via a survey with ML practitioners. Results: Based on the content of 37 interviews, we identi􀀌ed 18 recurring quality issues and 24 strategies to mitigate them. For each identi􀀌ed issue, we describe the causes and consequences according to the practitioners'experience. Conclusion: We believe the catalog of issues developed in this study will allow the community to develop e􀀎cient quality assurance tools for ML models and MLSSs. A replication package of our study is available on our public GitHub repository 1 . Keywords Machine Learning based Software Systems 􀀁 Quality Assurance 􀀁 Quality issues 􀀁 Interview 􀀁 MLOps 􀀁 Machine Learning Mathematics Subject Classi􀀌cation (2020) MSC 68T05 Pierre-Olivier C^ot􀀓e 􀀁 Amin Nikanjam 􀀁 Rached Bouchoucha 􀀁 Ilan Basta 􀀁 Mouna Abidi 􀀁 Foutse Khomh Polytechnique Montr􀀓eal, Qu􀀓ebec, Canada E-mail: fpierre-olivier.cote, amin.nikanjam, rached.bouchoucha, ilan.basta, mouna.abidi, [foutse.khomh](foutse.khomhg@polymtl.ca)[g](foutse.khomhg@polymtl.ca)[@polymtl.ca](foutse.khomhg@polymtl.ca) Corresponding author's e-mail address: [pierre-olivier.cote@polymtl.ca](pierre-olivier.cote@polymtl.ca) 1[https://github.com/poclecoqq/quality_issues_in_MLSSs](https://github.com/poclecoqq/quality_issues_in_MLSSs) 1 Introduction Nowadays, Machine Learning Software Systems (MLSSs) have become a part of our daily life (e.g. recommendation systems, speech recognition, face detection) . Increasing demand is observed in various companies to employ Machine Learning (ML) for solving problems in their business. Typically, an MLSS receives data as input and employs ML models to make intelligent decisions automatically based on learned patterns, associations, and knowledge from data (Marijan et al., 2019) . Therefore, ML models are implemented as software components integrated into other subsystems in MLSSs, and like other software systems, quality assurance is necessary. According to the growing importance of MLSSs in today's world, there is a strong need for ensuring their quality. False or poor decisions of such systems can lead to the malfunction of other systems, signi􀀌cant 􀀌nancial losses, or even threats to human life (Foidland Felderer, 2019) . Recently, an automated vehicle caused a crash of 8 vehicles, leading to 2 juveniles being transported to a hospital (Gua, 2022) . As reported by United States safety regulators, this is not a punctual event, since more than 400 car-crash with automated vehicles have been reported between July 2021 and May 2022 (Krisher, 2022) . Unquali􀀌ed MLSSs can be harmful to society in less obvious ways. For example, an experimental job recruitment tool developed by Amazon was shown to reject women's job applications more of

FAQs

Why is quality assurance critical for machine learning software systems (MLSSs)?

MLSSs make automated decisions that can malfunction other systems and produce serious consequences, including financial losses and safety threats when decisions are false or poor.

How were the real-world quality issues identified in this study?

The study conducts interviews with practitioners and experts to capture their experience and practices dealing with ML quality issues.

What outputs does the paper provide to help mitigate MLSS quality problems?

It identifies recurring quality issues and corresponding mitigation strategies, and describes causes and consequences based on practitioners’ experience, supported by survey validation.

Aug 3, 2026

Upload by Himbo