SERAC3: Smart and economical resource allocation for big data clusters in community clouds

Big data analysis jobs on clouds are gaining more and more popularity in recent years. It is critical but challenging to pick the right configuration for an incoming job, since the configuration space is too large, and the relationship between allocated resources and job performance is not deterministic. In this paper, we propose SERAC3 to allocate resources smartly and economically for big data clusters in community clouds. SERAC3 is a system that can automatically extract representative workloads from incoming big data analysis jobs, smartly decide an optimal configuration for each job, and adjust its assigning strategy in a quasi-realtime mode. With experiments on a community cloud built on OpenStack, we show that on average, SERAC3 can smartly select a configuration within 2.2% of the exact optimal one, while saving about 80.1% search cost compared to the exhaustive search.