Hybrid Computing—Where HPC meets grid and Cloud Computing

We introduce a hybrid High Performance Computing (HPC) infrastructure architecture that provides predictable execution of scientific applications, and scales from a single resource to multiple resources, with different ownership, policy, and geographic locations. We identify three paradigms in the evolution of HPC and high-throughput computing: owner-centric HPC (traditional), Grid computing, and Cloud computing. After analyzing the synergies among HPC, Grid and Cloud computing, we argue for an architecture that combines the benefits of these technologies. We call the building block of this architecture, Elastic Cluster. We describe the concept of Elastic Cluster and show how it can be used to achieve effective and predictable execution of HPC workloads. Then we discuss implementation aspects, and propose a new distributed information system design that combines features of distributed hash tables and relational databases.

Research highlights► A hybrid infrastructure for predictable execution of HPC workloads. ► An Elastic Cluster that seamlessly scales-up, scales-down, scales-out and scales-in. ► QoS via advance- or statistical reservation, and private virtual clusters. ► Information service design using distributed hash tables and relational databases.