Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds

•We define the problem of scheduling prioritized workflow ensembles on IaaS clouds.•We analyze and develop several dynamic (online) and static (offline) algorithms.•We address both problems of task scheduling and resource provisioning.•Our simulator models uncertainties in estimates, provisioning delays, and failures.•We use synthetic workflow ensembles based on important, real scientific applications.

Large-scale applications expressed as scientific workflows are often grouped into ensembles of inter-related workflows. In this paper, we address a new and important problem concerning the efficient management of such ensembles under budget and deadline constraints on Infrastructure as a Service (IaaS) clouds. IaaS clouds are characterized by on-demand resource provisioning capabilities and a pay-per-use model. We discuss, develop, and assess novel algorithms based on static and dynamic strategies for both task scheduling and resource provisioning. We perform the evaluation via simulation using a set of scientific workflow ensembles with a broad range of budget and deadline parameters, taking into account task granularity, uncertainties in task runtime estimations, provisioning delays, and failures. We find that the key factor determining the performance of an algorithm is its ability to decide which workflows in an ensemble to admit or reject for execution. Our results show that an admission procedure based on workflow structure and estimates of task runtimes can significantly improve the quality of solutions.