Article ID | Journal | Published Year | Pages | File Type |
---|---|---|---|---|
6872785 | Future Generation Computer Systems | 2018 | 13 Pages |
Abstract
Shuffle is a data exchanging phase that is always inserted between two adjacent computations to deliverintermediate results in data centers. It generates a burst of traffic that exhausts the network bandwidth, debasing the availability of the core layer facilities in fat-tree topologies. Previous researches follow either flow inhibition or infrastructural upgrading to achieve a high utilization of core network resources. However, dynamic pressure from shuffle burst introduces more unpredictable usage of core network that disturbs the global locality-based optimization on task schedule. In this work, we reduce the core bandwidth consumption by scheduling the location of adjacent computing workers based on our proposed distance model that uses a similarity-based distance to evaluate the dynamic distance between fat-tree leaf-nodes. Task assignment further utilizes this distance to schedule workers to avoid high intensity usage of core network resources. This design improves the performance of shuffle phase in popular on-data-center algorithms as well as maintains infrastructural inexpensiveness of their fat-tree topology. The proposed models are evaluated on a semi-physical simulation test platform and compared to state-of-the-art solutions, such as Space Shuffle and Scalable Shuffle. The results show that our design achieves an up to 18% speedup on shuffle procedure and a 23% extension of network capacity. In addition, a significant mitigation of congestion can be obtained on the bottleneck of core network.
Related Topics
Physical Sciences and Engineering
Computer Science
Computational Theory and Mathematics
Authors
Jihe Wang, Danghui Wang, Meikang Qiu, Yao Chen, Bing Guo,