Making sense of performance in in-memory computing frameworks for scientific data analysis: A case study of the spark system

Over the last five years, Apache Spark has become a major software platform for in-memory data analysis. Acknowledging its widespread use, we present a comprehensive study of system characteristics of Spark targeting scientific data analytics performing large-scale matrix operations. We compare its performance to SciDB, a disk-based platform for array data analysis. A benchmark, ArrayBench, is developed to evaluate the performance of four analytics processing gene expression matrices using basic data operators of Spark and SciDB. It is applied to data from a real biological workflow whose data inputs are in matrix form. Herein, we report the findings, which shed light on the improvement of Spark and SciDB and the future development of data-intensive scientific data analytics using the in-memory computing frameworks.