Effective load balancing for cluster-based servers employing job preemption

A cluster-based server consists of a front-end dispatcher and multiple back-end servers. The dispatcher receives incoming jobs, and then decides how to assign them to back-end servers, which in turn serve the jobs according to some discipline. Cluster-based servers have been widely deployed, as they combine good performance with low costs.Several assignment policies have been proposed for cluster-based servers, most of which aim to balance the load among back-end servers. There are two main strategies for load balancing: The first aims to balance the amount of workload at back-end servers, while the second aims to balance the number of jobs assigned to back-end servers. Examples of policies using these strategies are Dynamic and LC (Least Connected), respectively.In this paper we propose a policy, called LC*, which combines the two aforementioned strategies. The paper shows experimentally that when preemption is admitted (i.e., when jobs execute concurrently on back-end servers), LC* substantially outperforms bothDynamic and LC in terms of response-time metrics. This improved performance is achieved by using only information readily available to the dispatcher, rendering LC* a practical policy to implement. Finally, we study a refinement, called ALC* (Adaptive LC*), which further improves on the response-time performance of LC* by adapting its actions to incoming traffic rates.