Pareto frontier for job execution and data transfer time in hybrid clouds

• Particle Swarm Optimization to find the Pareto frontier of data-aware job scheduling.
• Pareto frontier of execution of jobs vs. transfer time of their required data-files.
• Analysis of the influence of Big-data and/or Private-data presence in hybrid clouds.
• Significant outperformance in comparison with current algorithms.
• Fast convergence speed; usually a few minutes for typical hybrid clouds.

This paper proposes a solution to calculate the Pareto frontier for the execution of a batch of jobs versus data transfer time for hybrid clouds. Based on the nature of the cloud application, jobs are assumed to require a number of data-files from either public or private clouds. For example, gene probes can be used to identify various infection agents such as bacteria, viruses, etc. The heavy computational task of aligning probes of a patient’s DNA (private-data) with normal sequences (public-data) with various data sizes is the key to this process. Such files have different characteristics–depends on their nature–and could be either allowed for replication or not in the cloud. Files could be too big to replicate (big data), others might be small enough to be replicated but they cannot be replicated as they contain sensitive information (private data). To show the relationship between the execution time of a batch of jobs and the transfer time needed for their required data in hybrid cloud, we first model this problem as a bi-objective optimization problem, and then propose a Particle Swarm Optimization (PSO)-based approach, called here PSO-ParFnt, to find the relevant Pareto frontier. The results are promising and provide new insights into this complex problem.