Machine Learning for Malware and Intrusion Detection - Dataset Design, Cost-Aware Models, and Research Pitfalls

Aug 3, 2026

Upload by Ophelia

Information technology reduces constraints of physical distance and delays across sectors, yet it also enables smarter cyber threats from hackers and cybercriminals. This thesis shows how machine learning strengthens malware and intrusion detection, emphasizing that effective learning depends on high-quality datasets and careful study design. It introduces an eBPF-powered dynamic analysis pipeline and a labeled dataset for Linux malware detection, achieving strong F1 performance with Random Forest. It further studies intrusion detection dataset imbalance and proposes cost-aware modeling approaches that trade accuracy for major speed and memory gains, plus a checklist-driven analysis of common research pitfalls.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages128

Reading Time5 h 23 min

AccessFree to download

Abstract

Machine learning and dataset needs

eBPF-powered malware detection dataset

Intrusion detection: dataset imbalance and sampling

Cost-aware models and prediction cost functions

ML-based IDS power and research pitfalls

Best-practice checklists and contributions

Machine Learning for Malware and Intrusion Detection: Dataset Design, Cost-Aware Models, and Research Pitfalls by 􀀍c Javad Kamyabi A thesis submitted to the School of Graduate Studies in partial ful􀀌llment of the requirements for the degree of Master of Engineering. Faculty of Engineering and Applied Science Memorial University February 2026 St. John’s, Newfoundland and Labrador, Canada Abstract Information technology has reduced constraints of physical distance and delays associated with traditional methods in areas such as medicine, economy, industry, and beyond. However, it also presents potential threats such as hackers and cybercriminals. As information technology advances, threats become smarter and more complex, cat-and-mouse-game that continuously increases in complexity. Machine learning improves security tools such as malware or intrusion detection by taking advantage of past experiences. Machine learning requires high-quality datasets to create e􀀋ective models. The 􀀌rst paper in this thesis, eBPF-Powered Dynamic Analysis for Linux Malware Detection: A Dataset and Experimental Study, explores the application of machine learning to detect malware. The paper also introduces an automated eBPF-based data collection pipeline using Docker containers to generate labeled malware and clean environment traces. We construct a dataset of clean and infected Linux operating systems and use various machine learning techniques to identify patterns in Linux system calls that indicate whether the operating system is infected or not, achieving a detection F1-Score of up to 99% with Random Forest models. Machine learning can also be used to develop intrusion detection systems. Two critical components of such systems are the dataset and the models. However, popular network attack datasets su􀀋er from imbalances, with signi􀀌cant disparities in the number of instances between di􀀋erent classes (e.g., benign tra􀀎c can have thousands of samples, while rare attack types may have fewer than 50) . This imbalance can severely a􀀋ect model performance; for example, rare attack classes may be underrepresented by a ratio of 40:1 compared to benign tra􀀎c, which can signi􀀌cantly reduce recall for these classes. To address this issue, over-and undersampling methods balance datasets before feeding them into the algorithms. However, undersampling may overlook important data, whereas oversampling can introduce redundancy, ultimately weakening the model’s performance. Furthermore, the speed with which an intrusion detection tool makes decisions plays a vital role in its e􀀋ectiveness. The second paper in this thesis, titled Cost-Aware Machine Learning for Intrusion Detection: A Performance Trade-O􀀋 Study, demonstrates that by sacri􀀌cing an insigni􀀌 -cant amount of accuracy, it is possible to achieve models that are tens of times faster and signi􀀌cantly less memory-consuming, making them practical for real-time deployment. This is accomplished by exploring the combination of di􀀋erent deep learning and machine learning models, along with various over-and under-sampling methods. Furthermore, the paper proposes twelve prediction cost functions that integrate these trade-o􀀋s alongside traditional performance measures. A slow intrusion detection tool can otherwise become a bottleneck in a network, highlighting the need for models that balance accuracy and e􀀎ciency. The third paper, titled Power and Pitfalls of ML-Based Intrusion Detection Systems, examines key challenges in developing machine learning-based intrusion detection systems, with a focus on both dataset generation and model design. It highlights issues such as the lack of representative datasets and the limited generalizability of models. This paper examines ten signi􀀌cant research barriers and their interconnections, which means that a barrier may lead to one or more barriers. The study includes a statistical analysis of dozens of research papers, revealing the current stat

FAQs

Why does the thesis emphasize dataset quality for machine learning security tools?

Machine learning models depend on high-quality, representative datasets to learn reliable patterns. The work highlights that poor dataset design or imbalance can directly reduce detection performance and generalizability.

How does the thesis collect data for Linux malware detection?

It presents an eBPF-based automated data collection pipeline using Docker containers to generate labeled malware traces and clean environment traces, forming a dataset of clean and infected Linux systems.

What is the purpose of cost-aware intrusion detection models in this thesis?

The thesis aims to balance accuracy with real-time efficiency by exploring model combinations and sampling methods and by introducing twelve prediction cost functions that incorporate performance trade-offs.

Aug 3, 2026

Upload by Ophelia