iPACS: Power-aware covering sets for energy proportionality and performance in data parallel computing clusters

• Node set discovery algorithms that find an energy-optimized node set.
• An extended discovery algorithm providing any required degree of data availability.
• Mathematical analysis of minimal covering set size under the assumption.
• Extensive evaluation results with respect to energy consumption and performance.

Energy consumption in datacenters has recently become a major concern due to the rising operational costs and scalability issues. Recent solutions to this problem propose the principle of energy proportionality, i.e., the amount of energy consumed by the server nodes must be proportional to the amount of work performed. For data parallelism and fault tolerance purposes, most common file systems used in MapReduce-type clusters maintain a set of replicas for each data block. A covering subset is a group of nodes that together contain at least one replica of the data blocks needed for performing computing tasks. In this work, we develop and analyze algorithms to maintain energy proportionality by discovering a covering subset that minimizes energy consumption while placing the remaining nodes in low-power standby mode in a data parallel computing cluster. Our algorithms can also discover covering subset in heterogeneous   computing environments. In order to allow more data parallelism, we generalize our algorithms so that it can discover kk-covering subset, i.e., a set of nodes that contain at least kk replicas of the data blocks. Our experimental results show that we can achieve substantial energy saving without significant performance loss in diverse cluster configurations and working environments.