HaoLap: A Hadoop based OLAP system for big data

•We implement the Hadoop based multidimensional OLAP system for big data.•We propose dimension coding and traverse algorithm to achieve the roll up operation on hierarchy of dimension values; the chunk model and partition strategy to shard the cube; the linearization and reverse-linearization algorithm to store chunks and cells; the chunk selection algorithm to optimize OLAP performance.•We compared HaoLap performance with Hive, HadoopDB, HBaseLattice, and Olap4Cloud on several big datasets and OLAP applications.

In recent years, facing information explosion, industry and academia have adopted distributed file system and MapReduce programming model to address new challenges the big data has brought. Based on these technologies, this paper presents HaoLap (Hadoop based oLap), an OLAP (OnLine Analytical Processing) system for big data. Drawing on the experience of Multidimensional OLAP (MOLAP), HaoLap adopts the specified multidimensional model to map the dimensions and the measures; the dimension coding and traverse algorithm to achieve the roll up operation on dimension hierarchy; the partition and linearization algorithm to store dimensions and measures; the chunk selection algorithm to optimize OLAP performance; and MapReduce to execute OLAP. The paper illustrates the key techniques of HaoLap including system architecture, dimension definition, dimension coding and traversing, partition, data storage, OLAP and data loading algorithm. We evaluated HaoLap on a real application and compared it with Hive, HadoopDB, HBaseLattice, and Olap4Cloud. The experiment results show that HaoLap boost the efficiency of data loading, and has a great advantage in the OLAP performance of the data set size and query complexity, and meanwhile HaoLap also completely support dimension operations.