Multi-policy-aware MapReduce resource allocation and scheduling for smart computing cluster

When a user submit a MapReduce job in the smart computing cluster, we first need to allocate cluster resource for the job. It is widely concerned that how to save time and resource costs to provide users with computing capacity and services. Here, we propose a multi-policy-aware Resources Allocation Algorithm that can allocate the appropriate amount of resources to the job to meet the execution deadline in private cloud and an extension in public cloud. Then when job running in the allocated cluster, in order to guarantee the performance of smart computing framework, we further propose a multi-policy-aware resource scheduling optimization model under YARN. If users use default policy, considering the difference of the heterogeneity of jobs and the resource request of tasks, we propose a global task dynamic resource scheduling algorithm-LRD algorithm, based on data locality, resource demand and task dependency. Further, if users set the deadline policy for jobs, we propose a DA (deadline-aware) scheduling algorithm based on performance prediction model. It can optimize the overall execution time of Hadoop jobs and improve the resource utilization of the entire cluster. Finally, we conduct different experiments to evaluate and verify the proposed models and algorithms in this paper.