From Micro-benchmarks to Machine Learning - Unveiling the Efficiency and Scalability of Hadoop and Spark

Aug 4, 2026

Upload by Liam

With the exponential growth of data, the demand for efficient and scalable data processing has become critical. The study evaluates Hadoop and Spark, key open-source Big Data components, through comprehensive performance analysis in virtualized environments using a suite of benchmarks. Workloads range from micro-benchmarks like Sort, WordCount, and TeraSort to web-search tasks such as PageRank and machine-learning methods including Naive Bayes and K-means. Results highlight Spark’s in-memory processing advantages across scenarios, while noting Hadoop’s potential edge under resource constraints and with large inputs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

Related Works

Performance Evaluation Methodology

Benchmarks and Workloads

Results and Discussion

Efficiency and Scalability Findings

Conclusion

JIM International Journal of Interactive Mobile Technologies [Onli](Online-Journals.org)[ne-Jo](Online-Journals.org)[urnals](Online-Journals.org)[.org](Online-Journals.org) iJIM | eISSN: 1865-7923 | Vol. 18 No. 17 (2024) | [https://doi.org/10.3991/ijim.v18i17.44555](https://doi.org/10.3991/ijim.v18i17.44555) PAPER From Micro-benchmarks to Machine Learning: Unveiling the Efficiency and Scalability of Hadoop and Spark Salah Eddine Hebabaze1(􀀍), Mohamed El Ghmary2, Hamid El Bouabidi1, Sara Maftah1, Mohamed Amnai1 1Ibn Tofaïl University, Kenitra, Morocco 2Sidi Mohamed Ben Abdellah University, Fez, Morocco Salaheddine.Hebabaze@ [uit.ac.ma](uit.ac.ma) ABSTRACT With the exponential growth of data, the demand for efficient and scalable data processing solutions has become paramount. Hadoop and Spark, pivotal components of the open-source Big Data landscape, have been put to the test in this study. We conducted a comprehensive performance analysis of Hadoop and Spark in virtualized environments, evaluating their prowess across a suite of benchmarks. The benchmarks encompassed a spectrum of workloads, from micro-benchmarks such as Sort, WordCount, and TeraSort to web search tasks such as PageRank and machine learning endeavors including Naive Bayes and K-means. The central focus was to gauge their performance, efficiency, and resource utilization. The findings ofthis study underscore the benefits of Spark’s in-memory processing, demonstrating its superiority over Hadoop in various scenarios. Spark excels in machine learning and web search applications, particularly when handling smaller inputs. Its efficient memory management and support for multiple iterations make it a strong choice. In resource-constrained environments or when dealing with large input files and limited memory, Hadoop may still hold an edge. The design and implementation of data processing solutions in virtualized environments should carefully consider the specific demands of each framework. This study not only presents a performance comparison of Hadoop and Spark across different benchmarks but also emphasizes the vital implications for designing and deploying data processing solutions in virtualized settings. It serves as a cornerstone for informed decision-making, paving the way for optimized algorithms and techniques in the dynamic landscape of big data processing. KEYWORDS big data, Hadoop, Apache Spark, MapReduce, HiBench benchmark, machine learning, memory resource limitations, data workloads 1 INTRODUCTION Nowadays, traditional data management systems can’t process the data’s complexity because of its size, structure, and limited processing time [1] . Massive and Hebabaze, S. E., El Ghmary, M., El Bouabidi, H., Maftah, S., Amnai, M. (2024) . From Micro-benchmarks to Machine Learning: Unveiling the Efficiency and Scalability of Hadoop and Spark. InternationalJournal of Interactive Mobile Technologies (iJIM), 18(17), pp. 46–60. [https://doi.org/10.3991/ijim.v18i17](https://doi.org/10.3991/ijim.v18i17) . 44555 Article submitted 2023-09-08. Revision uploaded 2024-06-24. Final acceptance 2024-06-29. © 2024 by the authors of this article. Published under CC-BY. 46 International Journal of Interactive Mobile Technologies (iJIM) iJIM | Vol. 18 No. 17 (2024) From Micro-benchmarks to Machine Learning: Unveiling the Efficiency and Scalability of Hadoop and Spark complex data, known as “Big Data,” requires new approaches such as Hadoop and Spark. Hadoop is batch-oriented, while Spark is in-memory and real-time, making it faster and more versatile [2] . This paper compares Hadoop and Spark, two popular data processing frameworks, highlighting their strengths and weaknesses. It emphasizes the importance of selecting the right framework for specific tasks. The paper also contributes to the ongoing discussion among professionals and organizations about improving web-based mobile learning in educational contexts by providing insight

FAQs

What benchmarks and workloads are used to compare Hadoop and Spark?

The study uses micro-benchmarks such as Sort, WordCount, and TeraSort, along with web search tasks like PageRank. It also evaluates machine learning workloads including Naive Bayes and K-means.

Why does Spark generally outperform Hadoop in the reported scenarios?

Spark’s in-memory processing is emphasized as a key advantage, supported by findings that show better efficiency and performance across multiple workloads.

When may Hadoop still be the better choice according to the paper?

The paper suggests Hadoop can hold an edge in resource-constrained environments or when dealing with large input files under limited memory conditions.

Aug 4, 2026

Upload by Liam