Why do so? - A Practical Perspective on Machine Learning Security

Aug 2, 2026

Upload by Jiven

Despite extensive academic work on machine learning security, attack occurrences in real deployments remain poorly understood. This paper presents a quantitative study with 139 industrial practitioners to analyze how often attacks arise, how strongly practitioners perceive threats, and which factors shape threat exposure and defense practices. Organizational and individual patterns are examined, including how prior ML security knowledge influences perception. Detailed responses highlight concerns such as unreliable decision making, business information leakage, and bias introduction, informing regulation and auditing.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

AML threats studied in prior work

Motivation and survey design

Background

“Why do so?”—A Practical Perspective on Machine Learning Security Kathrin Grosse, Lukas Bieringer, Tarek R. Besold, Battista Biggio, Senior Member, IEEE, Katharina Krombholz arXiv :2207 .05 164v 1 [ cs .LG] 11 Jul 2022 Abstract—Despite the large body of academic work on machine learning security, little is known about the occurrence of attackson machine learning systems in the wild. In this paper, wereport on a quantitative study with 139 industrial practitioners. We analyze attack occurrence and concern and evaluate statistical hypotheses on factors inﬂuencing threat perception and exposure. Our results shed light on real-world attacks on deployed machine learning. On the organizational level, while we ﬁnd no predictors for threat exposure in our sample, the amount of implement defenses depends on exposure to threats or expected likelihood to become a target. We also provide a detailed analysis of practitioners' replies on the relevance of individual machine learning attacks, unveiling complex concerns like unreliable decision making, business information leakage, and bias introduction into models. Finally, we ﬁnd that on the individual level, prior knowledge about machine learning security inﬂuences threat perception. Our work paves the way for more research about adversarial machine learning in practice, but yields also insights for regulation and auditing. Index Terms—Adversarial Machine Learning, Machine Learning Security, Quantitative User Study. I. INTRODUCTION A large body of academic work focuses on machine learning security or adversarial machine learning (AML) [1]–[11] . These works investigate how machine learning (ML) can be circumvented and exploited by an attacker. For example, an attacker can tamper with the training data, yielding a model inferior in performance or that is sensitive to attacker speciﬁed, small parts of the input [4] . Alternatively, the attacker slightly alters tests data to change the output of an ML model [5],[10] . In addition, an ML model may leak the used training data [12] or can easily be copied when freely exposed [11] . Many of the settings studied AML can be criticized for being rather artiﬁcial. However, already these settings are hard to solve [4], [13] . One possible cause is that even though the current usage of ML in security and threat modelling have been criticised [14], [15], there is little work on ML security in the real world. The ﬁrst work in this direction, by Kumar et al. [16], investigated which AML threats are feared in practice and reported that industry is most concerned about poisoning attacks. Mirsky et al. [17] reported that organizations perceive 24 of 33 offensive AI techniques as a signiﬁcant threat. Moreover, Bieringer et al. [18] found evidence for rudimentary attacks on AI in the wild in their interviews among ML First two authors contributed equally. K. Grosse and B. Biggio are with the university of Cagliari. L. Bieringeris with QuantPi, T.R. Besold with Eindhoven University of Technology, K. Krombholz with CISPA Helmholtz Center of Information Security. B. Biggiois also with PluribusOne. Manuscript received April 19, 2005; revised August 26, 2015 . practitioners. Boenisch et al. [19] ﬁnd that security and ML security awareness of ML practitioners was overall low. To shed light on the state of AML in practice and the factors inﬂuencing organizations' approach to ML security, we conduct a quantitative survey among ML practitioners. Inspired by prior work [17], we investigate which threats are dreaded and why. Furthermore, we control for variables like application area [19], how long ML has been used in production, data type, and prior knowledge [18] . All these questions and variables form part of our anonymous questionnaire for ML-practitioners. Our 139 participants help us to shed light on the following topics: AML in practice. We ﬁnd that there are occurrences of AML attacks, more speciﬁcally evasion and poisoning, in practice. H

FAQs

What gap does the paper address in machine learning security research?

It highlights that, despite many academic studies on ML security, there is limited knowledge about how attacks actually occur in real-world deployments. The paper focuses on evidence from industrial practitioners rather than purely theoretical settings.

How was the study conducted to understand AML in practice?

The study uses a quantitative survey of 139 industrial machine learning practitioners. It collects responses about attack occurrences, threat concerns, and factors influencing threat perception and exposure.

Which factors influence threat perception and defense measures in the organizations studied?

The findings suggest no predictors for threat exposure in the sample at the organizational level, while the number of implemented defenses depends on exposure to threats or the expected likelihood of becoming a target. On the individual level, prior knowledge about ML security affects threat perception.

Aug 2, 2026

Upload by Jiven