SSFile: A novel column-store for efficient data analysis in Hadoop-based distributed systems

Recently, large-scale relational data analysis has gained much attention. Several Hadoop-based distributed systems have been proposed for scalable relational data analysis. Because the column-store approach is very suitable for analytic queries, many studies on column-oriented storage and query processing for Hadoop-based distributed systems have been conducted. However, two problems have arisen in existing studies, the first of which is that only a small amount of data is processed per task during distributed processing. Each task reads only the necessary data using the columnar structure. Because the task initialization in Hadoop requires a large overhead, it is inefficient that each task processes a small amount of data. The second problem is the lack of support for techniques that optimize columnar execution. Although many such techniques have been proposed for efficient columnar query execution, existing column-store methods for Hadoop-based distributed systems cannot support them efficiently. In this paper, we propose a novel column-store method called SSFile for Hadoop-based distributed systems. SSFile increases the actual amount of data processed per task and supports representative columnar execution techniques for efficient query processing. Through extensive experiments, we show that SSFile significantly improves the performance of distributed processing.