A Resource Co-Allocation method for load-balance scheduling over big data platforms

Recently, an increasingly large number of high-performance computing (HPC) applications have been developed and hosted in clouds. The cloud computing scheme provides a model of utility computing, which is often implemented in the form of big data centers. As HPC applications are usually resource-intensive, how to exploit economies of scale to harmonize the resource allocation among large-scale HPC applications under performance constraints often poses a significant challenge on the load-balance scheduling policies in big data platforms. In view of this challenge, in this paper we propose a Resource C o-Allocation method, named RCA, for load-balance scheduling. Technically, the method is promoted by four steps that are enacted in a hierarchical control framework. The effectiveness and efficiency of the proposed method are validated by extensive experiments.