Flame-MR: An event-driven architecture for MapReduce applications

• Description of Flame-MR, a new MapReduce framework that improves the performance and resource efficiency of Hadoop.
• Flame-MR keeps Hadoop API compatibility in order to avoid source code modifications.
• Performance comparison with Hadoop-based frameworks using representative workloads on an HPC cluster and a cloud platform.
• Flame-MR reduces Hadoop execution times by up to 34% for the selected micro-benchmarks and 54% for the application benchmarks.

Nowadays, many organizations analyze their data with the MapReduce paradigm, most of them using the popular Apache Hadoop framework. As the data size managed by MapReduce applications is steadily increasing, the need for improving the Hadoop performance also grows. Existing modifications of Hadoop (e.g., Mellanox Unstructured Data Accelerator) attempt to improve performance by changing some of its underlying subsystems. However, they are not always capable to cope with all its performance bottlenecks or they hinder its portability. Furthermore, new frameworks like Apache Spark or DataMPI can achieve good performance improvements, but they do not keep compatibility with existing MapReduce applications. This paper proposes Flame-MR, a new event-driven MapReduce architecture that increases Hadoop performance by avoiding memory copies and pipelining data movements, without modifying the source code of the applications. The performance evaluation on two representative systems (an HPC cluster and a public cloud platform) has shown experimental evidence of significant performance increases, reducing the execution time by up to 54% on the Amazon EC2 cloud.