Evaluating machine learning models in a SIEM environment - Creating a testbed for systematic evaluation of anomaly detection ML models in the ELK Stack

Sep 17, 2026

Uploaded by Eliana

This project designs and implements a reproducible testbed for detecting user behavior anomalies in web applications using machine learning within the Elastic Stack. It addresses limitations of rule-based SIEM detection by enabling consistent dataset creation, model training, deployment, and comparison under identical conditions. A synthetic data pipeline generates ECS-compliant logs from a custom student management platform. Ingest and transforms in the ELK pipeline create session-level features for supervision. Two evaluated approaches, Elastic DFA and a custom Random Forest in Python, are compared via an evaluation dashboard using standardized metrics and visualizations, supporting repeatable experimentation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages65

Reading Time2 h 44 min

AccessFree to download

Abstract

Management Summary

Initial Situation

Machine learning (ML) challenges

Objective

Approach and Technologies Used

Evaluating machine learning models in a SIEM environment Creating a testbed for systematic evaluation of anomaly detection ML models in the ELK Stack Print: 18.12.2025 Team / Authors: Martin Arendar und Matteo Mahler Advisor: Prof. Dr. Daniel Patrick Politze Abstract This project presents the design and implementation of a testbed for detecting user behavior anomalies in web applications using machine learning within the Elastic Stack. Traditional Security Information and Event Management (SIEM) systems rely largely on rule based detection, which offers limited flexibility and struggles with complex behavioral patterns. Modern machine learning approaches address these limitations but introduce new challenges, especially regarding data availability, model evaluation, and integration into operational monitoring systems. The goal of this work is therefore not to build a single high performing model, but to create an extensible and reproducible environment in which different machine learning models can be trained, deployed, and compared under identical conditions. Because no suitable dataset for user centric anomaly detection existed, we developed a synthetic data generation pipeline that simulates user interactions in a controlled web application. A custom student management platform was implemented to produce ECS compliant logs representing normal and anomalous behavior. A traffic generator produces user flows and defined anomaly patterns over configurable time ranges, allowing the creation of reproducible datasets. These logs are ingested through Filebeat and Logstash into Elasticsearch, enriched through ingest pipelines (e.g., GeoIP), and transformed into session level features using Elastic Transforms. This pivot from action level events to session level aggregates provides meaningful input for supervised machine learning models. Two machine learning approaches are evaluated: Elastics built in Data Frame Analytics (DFA) classifier and a custom Random Forest model trained in Python using scikit learn, imported into Elastic via eland. Both models operate on the same labelled session dataset, enabling direct comparison. A dedicated evaluation dashboard, the foundation of the testbed, visualizes confusion matrices, classification metrics, feature importance, and anomaly distributions. This dashboard allows users to compare models on standardized performance indicators and supports repeated experimentation with newly trained models. The results demonstrate that the system functions reliably as a testbed. Data generation, ingestion, transforms, and dashboards work as intended, and both models can be evaluated consistently. While Elastic’s DFA provides strong baseline performance with minimal configuration, the custom model highlights the challenges of feature engineering, data realism, and training strategy. Several findings indicate opportunities for improvement, particularly regarding model tuning, dataset variability, and automation of dashboard creation. The project concludes that a machine learning based anomaly detection workflow in Elastic is feasible and that the developed testbed offers a solid foundation for future research, such as improved model engineering, integration of real world data, or real time anomaly tracking. Management Summary Initial Situation Security Information and Event Management (SIEM) systems play a central role in modern cybersecurity by collecting, correlating, and analyzing log data from different systems in once centralized place. Traditional SIEM platforms rely mostly on rule based detection mechanisms. While these rules are easy to implement, they struggle with dynamic or previously unseen patterns and often generate false alarms. Machine learning (ML) offers an opportunity to improve anomaly detection by learning past patterns instead of relying on static rules. However, ML introduces new challenges, especially regarding the availability of suitable datasets, th

FAQs

Why is a testbed needed for evaluating ML-based anomaly detection in SIEM environments?

Traditional SIEM relies on rule-based detection that struggles with dynamic or previously unseen patterns. ML can improve anomaly detection but requires suitable datasets, reproducible evaluation, and operational integration, which the testbed standardizes.

How does the project generate data for user-centric anomaly detection?

It builds a synthetic data generation pipeline using a custom student management web application. A traffic generator produces user flows and configurable anomaly patterns, outputting ECS-compliant normal and anomalous logs.

Which machine learning models are evaluated and how are they compared?

The project evaluates Elastic’s Data Frame Analytics classifier and a custom Random Forest model trained in Python (scikit-learn) and imported into Elastic via eland. Both models run on the same labeled session dataset and are compared using a dedicated evaluation dashboard with metrics such as confusion matrices and feature importance.

Sep 17, 2026

Uploaded by Eliana