Deadline scheduling algorithm for sustainable computing in Hadoop environment

Cloud computing is popular choice for processing and analyzing large amounts of data. Organizations can easily manage and deploy powerful clusters that run different software environments and enable distributed processing. Scheduling is an important part of distributed computing that allows users to leverage the available resources for a faster computation time. In this paper we propose a generic scheduling algorithm that takes deadline constraints into consideration. We develop a cost model that estimates the remaining work load which allows the scheduler to properly prioritize jobs according to their upcoming deadlines. The cost model works with generic abstract resources requests such as virtual cores, memory and containers and determines the remaining running time based on the completed tasks. We validate the cost model and measure the performance of the scheduler by running several experiments on a cluster on Amazon EC2 and our algorithm performs as expected under different scenarios.