A framework for joint resource allocation of MapReduce and web service applications in a shared cloud cluster

In this study, we take the perspective of an e-commerce site hosting its WS and MR applications on a fixed-size private cloud cluster. We assume Quality of Service (QoS) guarantees must be provided to end-users, represented by upper-bounds on the average response times of WS requests and on the MR jobs execution times, as MR applications can be interactive nowadays. We consider multiple MR and WS user classes with heterogeneous workload intensities and QoS requirements. Being the cluster capacity fixed, some requests may be rejected at heavy load, for which penalty costs are incurred. We propose a framework to jointly optimize resource allocation for WS and MR applications hosted in a private cloud with the aim to increase cluster utilization and reduce its operational and penalty costs. The optimization problem is formulated as a non linear mathematical programming model. Applying the KKT conditions, we derive an equivalent problem that can be solved efficiently by a greedy procedure. The proposed framework increases cluster utilization by up to 18% while cost savings go up to 50% compared to a priori partitioning the cluster resources between the two workload types.