Allocation of partitioned data by using a neural network based approach

Secret sharing and erasure coding based approaches have been used in distributed storage systems to ensure confidentiality, integrity, and availability of critical information. However, these approaches introduce some access overhead since each access now needs to access multiple data sites to retrieve the original data.In this paper, we investigate allocation of data partitions in the network for data objects that are partitioned by using secret sharing schemes or erasure coding schemes. We define the problem as finding M nodes in the network to host the data partitions so that the total communication cost (the sum of the read communication cost and write communication cost in terms of communication distance) is minimized to read K partitions and update M partitions by all nodes in the network. This problem is NP-hard. We propose a self-organizing feature map (SOFM)-like algorithm and a heuristic algorithm for this problem. The numerical study shows both algorithms are effective for this problem.