Neural Compression Inference Accelerator - A Cost and Energy-Effective Alternative to Conventional Machine Learning Inference Methods

Aug 3, 2026

Recent advances in machine learning and AI are driving demand for specialized cloud hardware, but GPUs and TPUs are costly to buy, expensive to rent, subject to pricing spikes, and highly energy intensive. The thesis investigates using field-programmable gate arrays (FPGAs) configured as machine learning accelerators. It proposes an FPGA architecture based on compressed neural networks and evaluates the design through architectural simulations and cloud deployment. Results show average 76.2% cost and 26.5% energy savings for neural inference workloads.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages100

Reading Time4 h 12 min

AccessFree to download

Chapter 1 Introduction

1.1 GPUs, FPGAs, the Cloud, and Machine Learning

1.2 Contribution and Thesis Outline

Chapter 2 Compressing Neural Networks

2.1 Pruning

2.2 Quantization and Weight Sharing

Chapter 3 Neural Compression Inference Accelerator

3.1 Hardware Parallelization

3.2 Hardware Load Balancing

Chapter 4 Cloud Application: Amazon Web Services

Chapter 5 Methodology

5.1 NCIA

5.2 V100 GPU

Chapter 6 Results

6.1 Performance

6.2 Power Efficiency

Chapter 7 Discussion and Future Work

7.1 Future of the NCIA

7.2 Edge Applications

Chapter 8 Related Work

8.1 Heterogeneous FPGAs in the Cloud

8.2 Accelerating Machine Learning on the Edge

Chapter 9 Concl

NEURAL COMPRESSION INFERENCE ACCELERATOR: A COST AND ENERGY-EFFECTIVE ALTERNATIVE TO CONVENTIONAL MACHINE LEARNING INFERENCE METHODS A Thesis presented to the Faculty of California Polytechnic State University, San Luis Obispo In Partial Fulfillment of the Requirements for the Degree Master of Science in Electrical Engineering by Matthew Wallace © 2023 Matthew Wallace ALL RIGHTS RESERVED ii COMMITTEE MEMBERSHIP TITLE: Neural Compression Inference Accelerator: A Cost and Energy-Effective Alternative to Conventional Machine Learning Inference Methods AUTHOR: Matthew Wallace DATE SUBMITTED: June 2023 COMMITTEE CHAIR: Joseph Callenes-Sloan, Ph.D. Professor of Computer Engineering COMMITTEE MEMBER: Andrew Danowitz, Ph.D. Professor of Computer Engineering COMMITTEE MEMBER: Helen Yu, Ph.D. Professor of Electrical Engineering ABSTRACT Neural Compression Inference Accelerator: A Cost and Energy-Effective Alternative to Conventional Machine Learning Inference Methods Matthew Wallace Recent developments in machine learning and artificial intelligence have sparked an influx of workloads that require specialized computer hardware for cloud services. The hardware running machine learning models predominantly consists of graphics processing units (GPUs) and tensor processing units (TPUs) . However, these components are expensive for cloud services to purchase, costly for customers to rent, prone to price spikes, and energy-intensive. In this research we show that both cloud services and customers would benefit from utilizing field programmable gate arrays (FPGAs) to alleviate the aforementioned challenges. An FPGA can be configured asa machine learning accelerator, operating similarly to a GPU or TPU. We propose an FPGA-based architecture that utilizes compressed neural networks. This provides an alternative hardware option for those running machine learning inference, resulting in savings in server construction costs, power consumption, workload distribution, and providing a more affordable option for customers. We evaluate the system using architectural simulations and cloud-based deployment and demonstrate 76.2% cost and 26.5% energy savings on average from using the Neural Compression Inference Accelerator (NCIA) . ACKNOWLEDGMENTS Thanks to: • Dr. Joseph Callenes-Sloan for advising me and guiding my research. I have learned so much about general computer engineering, modern technologies, how to implement novel systems, and how to effectively research directly from your classes and our meetings. I will have these new skills for life and am grateful to have learned them from someone so knowledgeable and experienced. • Dr. Andrew Danowitz and Dr. Xiao-Hua Yu for reviewing and providing feedback for my work. A great fraction of my foundational knowledge and inspiration for my research comes from taking your EE509, EE521, and EE542 classes. Your captivating lectures helped me identify my field of passion in the large realm of engineering. • My Mother, Father, and Sister for all of the love and support during my studies. I would not have been able to return to school and focus on learning without your support and guidance. I am forever grateful to have gone to graduate school to learn how to research and produce this Thesis. I am looking forward to celebrating and spending more time with you all soon. • Emma for the care and calmness you bring to balance my work with life. It was always great knowing I could look forward to spending some relaxing time together after studying. Thank you for all the meals and favors you have done for me while I was glued to my computer. TABLE OF CONTENTS Page LIST OF TABLES ................................. viii LIST OF FIGURES ................................ ix CHAPTER 1 Introduction ................................... 1 1.1 GPUs, FPGAs, the Cloud, and Machine Learning ........... 1 1.2 Contribution and Thesis Outline ...............

FAQs

Why consider FPGA-based inference instead of conventional GPU/TPU approaches in the cloud?

GPUs and TPUs are expensive to purchase, costly to rent, vulnerable to price spikes, and energy intensive. The thesis targets these cost and energy constraints for inference workloads.

What role does neural compression play in the proposed accelerator?

The FPGA architecture uses compressed neural networks, enabling an alternative inference path that reduces server construction costs, power consumption, and improves workload affordability.

How are the accelerator benefits evaluated in the thesis?

The work uses architectural simulations and cloud-based deployment to measure performance and quantify savings. The reported averages are 76.2% cost savings and 26.5% energy savings.

Aug 3, 2026

Upload by River Wang