A Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data - Research

Aug 4, 2026

Upload by Anda

Anomaly detection models commonly rely on residual-based learning from normal data and score unseen samples by dissimilarity to the learned normal regime. In real operations, training sets are often contaminated by a fraction of abnormal samples, which degrades residual-based anomaly detection performance. This paper proposes a generic, fully unsupervised refinement framework that removes candidate anomalies from contaminated training data in a single step. Experiments on multivariate time series datasets show clear improvements over training on contaminated data without refinement and strong competitiveness against an ideal anomaly-free reference.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

Residual-based anomaly detection and its assumptions

Limitations under contaminated training data

Fully unsupervised anomaly detection approaches

Proposed framework for refinement

Single-step refinement strategy

Application to residual-based models

Experimental evaluation

Industrial time series datasets

Comparison against naive and ideal baselines

A Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data arXiv :2308 . 13352v2 [ cs .LG] 7 Sep 2023 Markus Ulmer 1 , Jannik Zgraggen2 , and Lilach Goren Huber3 1,2,3 Zurich University of Applied Sciences, Winterthur 8401, Switzerland [markus.ulmer@zhaw.ch](markus.ulmer@zhaw.ch) [jannik.zgraggen@zhaw.ch](jannik.zgraggen@zhaw.ch) [lilach.gorenhuber@zhaw.ch](lilach.gorenhuber@zhaw.ch) ABSTRACT Anomaly detection (AD) tasks have been solved using machine learning algorithms in various domains and applications. The great majority of these algorithms use normal data to train a residual-based model, and assign anomaly scores to unseen samples based on their dissimilarity with the learned normal regime. The underlying assumption of these approaches is that anomaly-free data is available for training. This is, however, often not the case in real-world operational settings, where the training data may be contaminated with a certain fraction of abnormal samples. Training with contaminated data, in turn, inevitably leads to a deteriorated AD performance of the residual-based algorithms. In this paper we introduce a framework for a fully unsupervised refinement of contaminated training data for AD tasks. The framework is generic and can be applied to any residualbased machine learning model. We demonstrate the application of the framework to two public datasets of multivariate time series machine data from different application fields. We show its clear superiority over the naive approach of training with contaminated data without refinement. Moreover, we compare it to the ideal, unrealistic reference in which anomaly-free data would be available for training. Since the approach exploits information from the anomalies, and not only from the normal regime, it is comparable and often outperforms the ideal baseline as well. 1. INTRODUCTION Anomaly detection (AD) tasks are common in very diverse fields, including medical image processing, autonomous driving, fraud detection, and fault detection in industrial machines. An inherent property of AD tasks is that very few or no labelled examples of anomalous behavior are provided in advance. Therefore, the most common machine learning approaches to solve these tasks are based on using exclusively normal data to train a selected prediction algorithm, which is subsequently used to infer on unseen data. The hidden assumption here is that the algorithm’s prediction errors (residuals) will be higher whenever the input sample does not belong to the learned distribution. This family of models can be referred to as residual-based models, irrespective of whether they use regression or reconstruction residuals to detect anomalies, with the most common models being reconstruction models such as principal component analysis (PCA) or different types of autoencoder (AE) neural networks. It is worth noting that these models are often termed ”unsupervised”. However, in the context of AD, they should be referred to as ”semi supervised” since they assume the availability of labeled normal data for training. In fact, such models tend to perform rather poorly whenever contamination in the form of anomalous samples is introduced into the training data. In real world applications, however, the assumption of having anomaly-free training data does not always hold, as data contamination cannot be avoided. In this case, truly unsupervised methods, based on clustering or one-class classification (Schlkopf, Williamson, Smola, Shawe-Taylor, & Platt, 1999), are required. Recently there has been a growing effort to develop deep unsupervised algorithms for AD, whose performance is not severely damaged by data contamination (Munir, Siddiqui, Dengel, & Ahmed, 2018) . Despite the high practical relevance of the problem, systematic solutions for AD with contaminated training data are rather rare. Recently, several papers have suggested useful approaches based on d

FAQs

Why do residual-based anomaly detection methods often fail with contaminated training data?

They assume training data is normal and learn a residual pattern for the normal regime; abnormal contamination breaks this assumption and deteriorates anomaly scores.

What does the proposed framework do differently?

It performs a single-step refinement on contaminated training data by proposing candidate anomalies to remove, enabling fully unsupervised training for residual-based models.

How is the framework evaluated and what baselines are used?

It is tested on two multivariate time series datasets and compared against training a residual-based model on contaminated data without refinement, and against an ideal anomaly-free training reference.

Aug 4, 2026

Upload by Anda