An elasticity model for High Throughput Computing clusters

Different methods have been proposed to dynamically provide scientific applications with execution environments that hide the complexity of distributed infrastructures. Recently virtualization has emerged as a promising technology to provide such environments. In this work we present a generic cluster architecture that extends the classical benefits of virtual machines to the cluster level, so providing cluster consolidation, cluster partitioning and support for heterogeneous environments. Additionally the capacity of the virtual clusters can be supplemented with resources from a commercial cloud provider. The performance of this architecture has been evaluated in the execution of High Throughput Computing workloads. Results show that, in spite of the overhead induced by the virtualization and cloud layers, these virtual clusters constitute a feasible and performing HTC platform. Additionally, we propose a performance model to characterize these variable capacity (elastic) cluster environments. The model can be used to dynamically dimension the cluster using cloud resources, according to a fixed budget, or to estimate the cost of completing a given workload in a target time.

Research highlights
► Classical HTC clusters architectures can be used in Public and Hybrid Clouds.
► Virtualization and communication overheads can be neglected for HTC computations.
► Performance of HTC clusters grows linearly with the number of Cloud worker-nodes.
► Cloud Computing is a cost effective solution for HTC workloads.
► Clouds and virtualization delivers efficient, flexible and elastic HTC clusters.