ML-NIC - accelerating machine learning inference using smart network interface cards

Aug 3, 2026

Uploaded by Levi

Low-latency machine-learning inference is essential for mission-critical applications such as autonomous driving, defense target recognition, and network traffic analysis. To reduce inference delay, existing approaches offload computation to specialized hardware like GPUs, but mapping models onto programmable network devices remains limited. This work presents ML-NIC, a framework that deploys trained models directly into programmable network devices’ data-plane computational cores, leveraging device parallelism. Experiments show at least 6x lower latency and 16x higher throughput with minimal impact on model effectiveness, along with tighter latency bounds under concurrent traffic and reduced CPU and RAM usage.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Motivation for low-latency ML inference

Offloading inference to specialized hardware and its limits

Programmable data planes and the ML-NIC approach

TYPE Original Research PUBLISHED 06 January 2025 DOI 10. 3389/fcomp.2024.1493399 OPEN ACCESS EDITED BY Yong Wang, Guilin University of Electronic Technology, China REVIEWED BY Luca Deri, University of Pisa, Italy Sabina Rossi, Ca’ Foscari University of Venice, Italy *CORRESPONDENCE Sean Choi [sean.choi@scu.edu](sean.choi@scu.edu) RECEIVED 09 September 2024 ACCEPTED 04 December 2024 PUBLISHED 06 January 2025 CITATION Kapoor R, Anastasiu DC and Choi S (2025) ML-NIC: accelerating machine learning inference using smart network interface cards. Front. Comput. Sci. 6:1493399 . doi: 10.3389/fcomp.2024.1493399 COPYRIGHT © 2025 Kapoor, Anastasiu and Choi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. ML-NIC: accelerating machine learning inference using smart network interface cards Raghav Kapoor1 , David C. Anastasiu2 and Sean Choi1* 1 Cloud Lab, Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA, United States, 2Anastasiu Lab, Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA, United States Low-latency inference for machine learning models is increasingly becoming a necessary requirement, as these models are used in mission-critical applications such as autonomous driving, military defense (e.g., target recognition), and network tra􀀈c analysis. A widely studied and used technique to overcome this challenge is to o􀀉oad some or all parts of the inference tasks onto specialized hardware such as graphic processing units. More recently, o􀀉oading machine learning inference onto programmable network devices, such as programmable network interface cards or a programmable switch, is gaining interest from both industry and academia, especially due to the latency reduction and computational beneﬁts of performing inference directly on the data plane where the network packets are processed. Yet, current approaches are relatively limited in scope, and there is a need to develop more general approaches for mapping o􀀉oading machine learning models onto programmable network devices. Tofulﬁll such a need, this work introduces a novel framework, called ML-NIC, for deploying trained machine learning models onto programmable network devices’ data planes. ML-NIC deploys models directly into the computational cores of the devices to e􀀈ciently leverage the inherent parallelism capabilities of network devices, thus providing huge latency and throughput gains. Our experiments show that ML-NIC reduced inference latency by at least 6􀀂 on average and in the 99th percentile and increased throughput by at least 16x with little to no degradation in model e􀀀ectiveness compared to the existing CPU solutions. In addition, ML-NIC can provide tighter guaranteed latency boundsin the presence of other network tra􀀈c with shorter tail latencies. Furthermore, ML-NIC reduces CPU and host server RAM utilization by 6 .65% and 320 .80 MB. Finally, ML-NIC can handle machine learning models that are 2 .25 􀀂 larger than the current state-of-the-art network device o􀀉oading approaches. KEYWORDS machine learning, SmartNIC, Netronome, data plane, inference 1 Introduction Machine learning (ML) permeates a vast amount of everyday life, from personalized recommendations to stock market analysis and novel drug synthesis. While the machine learning models created to solve problems in these various 􀀂elds are proven to be highly e􀀓ective, these models often need large amount of time to make predictions (also referred to as model inference) on data instances. Often ti

FAQs

What problem does ML-NIC address?

ML-NIC targets the high latency of machine-learning inference for latency-critical applications by moving inference closer to where data packets are processed.

How does ML-NIC deploy machine learning models?

It deploys trained models directly into the computational cores of programmable network devices’ data planes to exploit inherent parallelism.

What performance improvements does ML-NIC report?

Experiments show at least 6x average latency reduction and latency improvement in the 99th percentile, at least 16x throughput increase, and reduced CPU and host server RAM usage, with little to no degradation in model effectiveness.

Aug 3, 2026

Uploaded by Levi