Workload balancing and adaptive resource management for the swift storage system on cloud

• We propose a workload balancing and adaptive resource management framework for Swift.
• We implement optimization algorithms of dynamic workload balancing for Swift.
• We conduct an experiment to demonstrate the effectiveness of this framework.

The demand for big data storage and processing has become a challenge in today’s industry. To meet the challenge, there is an increasing number of enterprises adopting distributed storage systems. Frequently, in these systems, storage nodes intensively holding hotspot data could become system bottlenecks while storage nodes without hotspot data might result in low utilization of computing resource. This stems from the fact that almost all the typical distributed storage systems only provide data-amount-oriented balancing mechanisms without considering the different access load of data. To eliminate the system bottlenecks and optimize the resource utilization, there is a demand for such distributed storage systems to employ a workload balancing and adaptive resource management framework. In this paper, we propose a framework of workload balancing and resource management for Swift, a widely used and typical distributed storage system on cloud. In this framework, we design workload monitoring and analysis algorithms for discovering overloaded and underloaded nodes in the cluster. To balance the workload among those nodes, Split, Merge and Pair Algorithms are implemented to regulate physical machines while Resource Reallocate Algorithm is designed to regulate virtual machines on cloud. In addition, by leveraging the mature architecture of distributed storage systems, the framework resides in the hosts and operates through API interception. To demonstrate its effectiveness, we conduct experiments to evaluate it. And the experimental results show the framework can achieve its goals.