InSTechAH: Cost-effectively autoscaling smart computing hadoop cluster in private cloud

Report shows that in many cloud data centers, computing resources are not used efficiently and thereby cause resource waste. In many cloud nodes, the utilization rate of CPU is only 20%. A large-scale multi-layer smart computing system (such as Deep Learning) require computing of a large amount of labeled data. To improve cost-effectiveness of resources in cloud data centers, running big data-based smart computing applications to utilize residual computing resource capacity is a feasible solution. However, performance loss brought by resource competition and interference between host application and smart computing application is the main challenge for us. In this paper, we design, implement and evaluate the InSTechAH: A big data-oriented cost-effective cluster autoscaling and scheduling scheme in private cloud, which improves the resource utilization as well as to maintain required quality of services by autoscaling and scheduling background smart computing analytics tasks.