| Article ID | Journal | Published Year | Pages | File Type |
|---|---|---|---|---|
| 425874 | Future Generation Computer Systems | 2014 | 14 Pages |
•We present a framework which deals with the issue of heterogeneity in clusters.•Mathematical Model for performance prediction and migration of job within clusters.•The framework is analyzed with comprehensive set of experiments (no simulations).•Sensitivity analysis is provided to determine the impact of various parameters.•Improve the throughput of a moderately heterogeneous compute farm by up to 25%.
In this paper we present ARRIVE-F, a novel open source framework which addresses the issue of heterogeneity in virtualized compute farms, such as those hosted by a cloud infrastructure provider. Unlike the previous attempts, our framework is not based on linear frequency models and does not require source code modifications or off-line profiling. The heterogeneous compute farm is first divided into a number of homogeneous sub-clusters. The framework then carries out a lightweight ‘online’ profiling of the CPU, communication and memory subsystems of all the active jobs in the compute farm. From this, it constructs a performance model to predict the execution times of each job on all the distinct sub-clusters in the compute farm. Based upon the predicted execution times, the framework is able to relocate the compute jobs to the currently best-suited hardware platforms such that the overall throughput of the compute farm is increased. We utilize the live migration feature of virtual machine monitors to migrate the job from one sub-cluster to another.The prediction accuracy of our performance estimation model is over 80%. The implementation of ARRIVE-F is lightweight, with an overhead of 3%. Experiments on a synthetic workload of scientific benchmarks show that we are able to improve the throughput of a moderately heterogeneous compute farm by up to 25%, with a time saving of up to 33%.
