COSHH: A classification and optimization based scheduler for heterogeneous Hadoop systems

•We propose a job scheduler to improve the mean completion time of jobs in a heterogeneous Hadoop cluster.•The scheduler considers heterogeneity at both the application and cluster levels of the system.•The scheduler is competitive with respect to other performance measures (fairness, locality and minimum share satisfaction).•Evaluation is performed by implementing the scheduler in MRSIM.•Production traces are used to evaluate performance.

A Hadoop system provides execution and multiplexing of many tasks in a common datacenter. There is a rising demand for sharing Hadoop clusters amongst various users, which leads to increasing system heterogeneity. However, heterogeneity is a neglected issue in most Hadoop schedulers. In this work we design and implement a new Hadoop scheduling system, named COSHH, which considers heterogeneity at both the application and cluster levels. The main objective of COSHH is to improve the mean completion time of jobs. However, as it is concerned with other key Hadoop performance metrics, our proposed scheduler also achieves competitive performance under minimum share satisfaction, fairness and locality metrics with respect to other well-known Hadoop schedulers.