Are Machine Learning Models for Malware Detection Ready for Prime Time? - Research-to-Deployment Reproducibility and Robustness

Aug 4, 2026

Upload by Logic

Machine learning models for malware detection achieve near-perfect results in academic settings but often fail to reproduce in real-world deployments. The paper examines why research evaluation conditions differ from practical deployment and how to design experiments that mimic operational scenarios. It also outlines ways to assess a model’s robustness over time and highlights pitfalls that hinder translating performance metrics into trustworthy, mainline defense tools.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

AccessFree to download

Introduction

Reproducibility gap in practice

Research setup and common pitfalls

Base rate fallacy and evaluation metrics

VU Research Portal Are Machine Learning Models for Malware Detection Ready for Prime Time? Cavallaro, Lorenzo; Kinder, Johannes; Pendlebury, Feargus; Pierazzi, Fabio; Massacci, Fabio; Bodden, Eric; Sabetta, Antonino published in IEEE Security and Privacy 2023 DOI (link to publisher) 10.1109/MSEC.2023.3236543 document version Publisher's PDF, also known as Version of record document license Article 25fa Dutch Copyright Act Link to publication in VU Research Portal citation for published version (APA) Cavallaro, L. , Kinder, J. , Pendlebury, F. , Pierazzi, F. , Massacci, F. , Bodden, E. , & Sabetta, A. (2023) . Are Machine Learning Models for Malware Detection Ready for Prime Time? IEEE Security and Privacy, 21(2), 53- 56. [https://doi.org/10.1109/MSEC.2023.3236543](https://doi.org/10.1109/MSEC.2023.3236543) General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. E-mail address: [vuresearchportal.ub@vu.nl](vuresearchportal.ub@vu.nl) [Download date: 03](Download date: 03) . Aug. 2026 |
BUILDING SECURITY IN | | --- | | Editors: Fabio Massacci, [fabio.massacci@ieee.org | Eric](fabio.massacci@ieee.org | Eric) Bodden, eric.bodden@uni-paderborn.de | Antonino Sabetta, [as@sabetta.com](as@sabetta.com) | Are Machine Learning Models for Malware Detection Ready for Prime Time? Lorenzo Cavallaro | University College London Johannes Kinder | Bundeswehr University Feargus Pendlebury | University College London Fabio Pierazzi | King’s College London We investigate why the performance of machine learning models for malware detection observed in alab setting often cannot be reproduced in practice. We discuss how to set up experiments mimicking a practical deployment and how to measure the robustness of a model over time. I n academic research on malware detection, machine learning-based techniques have become the de facto standard. Over the years, many papers havebeenpublishedonthetopic,using everything in the machine learning toolbox, from traditional models1, 2 to more recent neural network architectures.3, 4, 5 With published accuracies and other performance numbers regularly close to the 100% mark, you could not be blamed for thinking that the problem is practically solved. Why would anyone be interested in continuing to use malware signaturesand simple patterns? Cannot Reproduce? Yet, practitioners find themselves disillusioned when trying to put machine learning models from research into practice. In real-world deployments, machine learning-based malware classifiers are known to often be Digital Object Identifier 10.1109/MSEC.2023.3236543 Date of current version: 15 March 2023 unreliable and are not trusted to act asthe mainline of defense. Where does this discrepancy come from? It would be too easy toshrug this off as expected technical challenges in technology transfer; we must get to the bottom of the issue or risk exacerbating the reproducibility crisis decried in machine learning-based science.6 So, what’sthe deal? If the same model performs so differently in production, then the lab settings that yield near-perfect performance must not be representative of how such models are deployed. The typical research project on machine learning

FAQs

Why do malware-detection ML models perform differently in production than in lab settings?

The gap arises because lab evaluation conditions are often not representative of how models are deployed. Differences in experimental setup and assumptions can make reported performance non-transferable.

How should experiments be designed to better reflect practical deployment?

Experiments should mimic deployment conditions and evaluation procedures, so that measured performance aligns with operational realities. This includes using setups that better represent real-world conditions.

What is the base rate fallacy and how does it affect malware detection metrics?

Metrics can be misleading when the class ratio in the evaluation dataset differs from the real world. For example, a high precision or F1 can occur even when the classifier is effectively useless if malware dominates the test set.

Aug 4, 2026

Upload by Logic