TA3 - Testing Against Adversarial Attacks on Machine Learning Models - interactive system for HITL

Aug 4, 2026

Upload by Aurelia

Adversarial attacks pose major risks to deploying machine learning models across real-world applications. Testing models against such attacks is therefore essential for both evaluation and improvement. This paper presents the design and development of TA3, an interactive, human-in-the-loop system that enables human-steered attack simulation and visualization-assisted assessment of attack impact. The current version focuses on testing decision tree models using the One Pixel Attack method, highlighting HITL’s value and extending its potential to broader ML models and adversarial attack types.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

Motivation and differences from conventional software testing

Related defense and evaluation strategies

Related Work

Human-centered Artificial Intelligence

TA3: Testing Against Adversarial Attacks on Machine Learning Models Yuanzhe Jin Department of Engineering Science University of Oxford Oxford, United Kingdom [yuanzhe.jin@eng.ox.ac.uk](yuanzhe.jin@eng.ox.ac.uk) Min Chen Department of Engineering Science University of Oxford Oxford, United Kingdom [min.chen@oerc.ox.ac.uk](min.chen@oerc.ox.ac.uk) arXiv :2410 .05334v 1 [ cs .CR] 6 Oct 2024 Abstract—Adversarial attacks are major threats to the deployment of machine learning (ML) models in many applications. Testing ML models against such attacks is becoming an essential step for evaluating and improving ML models. In this paper, we report the design and development of an interactive system for aiding the workflow of Testing Against Adversarial Attacks (TA3). In particular, with TA3, human-in-the-loop (HITL) enables human-steered attack simulation and visualization-assisted attack impact evaluation. While the current version of TA3 focuses on testing decision tree models against adversarial attacks based on the One Pixel Attack Method, it demonstrates the importance of HITL in ML testing and the potential application of HITL to the ML testing workflows for other types of ML models and other types of adversarial attacks. Index Terms—Machine learning, decision tree, adversarial attack, visual analysis, model testing, interactive testing. I. INTRODUCTION The rapid development of machine learning (ML) technology has started to bring forth the deployment of ML models in our daily lives, e.g., face recognition, traffic management, unmanned supermarkets, and autonomous vehicles. At the same time, the quality of ML models has received increasing attention. Research on adversarial attacks has discovered a variety of vulnerabilities in ML models, ranging from getting a model to mistake a panda for a gibbon [1] to getting Tesla’s self-driving car to misjudge the speed limit sign [2] . In the field of ML, the current research effort on adversarial attacks places the main emphasis on discovering new adversarial attack methods and improving ML models’ resistance against specific types of adversarial attacks. The former has resulted in a large collection of attacking methods [3], while the latter typically focuses on the training processes, e.g., data distill [4] and adversarial training [5] . In this work, we focus on ML testing processes against adversarial attacks. There are differences as well as similarities between testing ML models and testing conventional software. The main differences include (i) deployable ML models are expected to make mistakes while deployable conventional software is expected to be bug-free; (ii) the inner workings of an ML model are expected to be difficult to comprehend, while a handcrafted program is expected to be fully understood. Because of these differences, the testing of ML models has relied primarily on statistical measures resulting from automated test runs, while the testing workflows for conventional software have relied extensively on human-in-the-loop (HITL) . Fundamentally, an ML model is also a program. Similar to a handcrafted program, the search space for a potential error in these programs is huge in that relying on a fully automated process to search for errors is mostly intractable computationally. Hence, in conventional software testing, HITL allows human experts to inject their knowledge to focus on highly probable parts of the search space. There is no reason why HITL cannot help human experts in testing ML models. This reasoning motivates us to develop an interactive software system to support ML testing workflows against adversarial attacks. There is a large collection of attacking methods, many of which apply to models trained with a specific algorithmic framework and specific types of training data [6] . There are also several strategies for defending against adversarial attacks, such as threat modeling, attack simulation, attack impact evaluation, countermeasure d

FAQs

What problem does TA3 address in machine learning security?

TA3 targets the need to test machine learning models against adversarial attacks before deployment, enabling evaluation and improvement of model robustness.

How does TA3 support testing workflows?

TA3 uses human-in-the-loop control to steer adversarial attack simulation and provides visualization to assess attack impact and model weaknesses beyond statistical summaries.

What does the current TA3 version focus on?

The current TA3 version tests decision tree models against adversarial attacks generated via the One Pixel Attack method.

Aug 4, 2026

Upload by Aurelia