Practical Performance of a Distributed Processing Framework for Machine-Learning-based NIDS

Aug 3, 2026

Upload by Liam

Network Intrusion Detection Systems (NIDSs) identify intrusion attacks within network traffic and alert administrators, especially to support detection of unknown threats. This work addresses a gap in prior research by implementing a distributed processing framework for machine-learning-based NIDSs and evaluating end-to-end performance when common ML classifiers are deployed. Five representative classifiers—Decision Tree, Random Forest, Naive Bayes, SVM, and kNN—are built on the framework and benchmarked using throughput and latency measurements. Results reveal classifier-dependent processing behavior and pinpoint subsystem bottlenecks that limit pipeline speed.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Introduction

Background on NIDS and ML-based NIDS

Motivation and contributions

Related Work

Datasets for ML-based intrusion detection

Existing MLNIDS frameworks

Practical Performance of a Distributed Processing Framework for Machine-Learning-based NIDS Maho Kajiura Department of Computer Science and Engineering Toyohashi University of Technology Toyohashi, Japan [kmaho@dsl.cs.tut.ac.jp](kmaho@dsl.cs.tut.ac.jp) Junya Nakamura Information and Media Center Toyohashi University of Technology Toyohashi, Japan [junya@imc.tut.ac.jp](junya@imc.tut.ac.jp) arXiv :2405 . 13066v1 [ cs .CR] 20 May 2024 Abstract—Network Intrusion Detection Systems (NIDSs) detect intrusion attacks in network traffic. In particular, machinelearning-based NIDSs have attracted attention because of their high detection rates of unknown attacks. A distributed processing framework for machine-learning-based NIDSs employing a scalable distributed stream processing system has been proposed in the literature. However, its performance, when machine-learningbased classifiers are implemented has not been comprehensively evaluated. In this study, we implement five representative classifiers (Decision Tree, Random Forest, Naive Bayes, SVM, and kNN) based on this framework and evaluate their throughput and latency. By conducting the experimental measurements, we investigate the difference in the processing performance among these classifiers and the bottlenecks in the processing performance of the framework. Index Terms—machine-learning, network intrusion detection system, distributed processing, network security I. INTRODUCTION A network-based intrusion detection system (NIDS) detects intrusion attacks in network traffic and notifies the network administrator. NIDS is considered an effective defense mechanism against cyber attacks. Traditional NIDSs detect abnormal traffic that matches intrusion attack patterns, called signatures, stored in a system database. However, this approach cannot detect unknown attacks because their patterns do not match the signatures. To overcome this limitation, machine-learningbased NIDSs (MLNIDSs) have been proposed in recent years [1]–[3] . MLNIDSs detect both known and unknown attacks by building machine-learning models that include learned attack patterns based on known attacks. Several frameworks that assist in the implementation of MLNIDSs have been proposed [4], [5] . These frameworks implemented all the functions necessary for MLNIDS on scalable distributed processing systems to process network traffic efficiently. When the network traffic increases, the MLNIDS performance can be flexibly adapted by adding nodes. They demonstrated the effectiveness of the framework by building MLNIDSs and evaluating their performance in terms of throughput and latency. However, both authors in [4], [5] did not evaluate the effectiveness of the framework when a machine-learning classifier This work was supported by the Hibi Science Foundation, the Naito Science & Engineering Foundation, and Tokai Foundation for Technology. is implemented. Furthermore, existing MLNIDSs also often focus on classifier performance, with only a few focusing on processing speed. Consequently, the volume of traffic MLNIDS can process was not investigated. As a result, system sizing, which is necessary if the framework is actually used for MLNIDSs, becomes difficult. In this paper, we construct an MLNIDS by implementing five representative classifiers based on the framework proposed in [4] and evaluate their throughput and latency. Based on this evaluation, we identify the differences in the processing performance among the classifiers and the bottlenecks in the processing performance in the framework. The experimental results show that the processing speed and classifier performance are highly dependent on the typeof classifier. Using appropriate machine-learning algorithms, the load on the MLNIDS can be reduced while maintaining high classifier performance. We found that Zeek [6], which constructs sessions from the network traffic, Logstash [7], which performs the classification process using machinelea

FAQs

What problem does the study address in distributed ML-based NIDS frameworks?

Prior work proposed distributed processing frameworks but did not comprehensively evaluate performance when specific machine-learning classifiers are implemented. The study fills this gap by measuring throughput and latency across multiple representative classifiers.

Which classifiers are implemented and evaluated?

The framework is used to implement Decision Tree, Random Forest, Naive Bayes, SVM, and kNN classifiers. Their impact on processing throughput and latency is then benchmarked experimentally.

Where do the main processing bottlenecks occur?

The evaluation identifies bottlenecks in framework subsystems associated with Zeek session construction, Logstash classification, and Elasticsearch result storage.

Aug 3, 2026

Upload by Liam