PHash: A memory-efficient, high-performance key-value store for large-scale data-intensive applications

Large-scale data-intensive web services are evolving faster than ever, accelerating global growth in data usage and traffic at a rapid rate. This rapid growth is demanding the expansion of high-cost data infrastructures, which also underscores the industry's need for cost-effective, high-performance distributed key-value stores. Designing key-value stores often involves a trade-off between performance and memory usage. For example, many previous studies focusing on minimizing the memory usage have developed on-disk indexing schemes, leading to lower performance. An alternative design based on in-memory indexing allows better performance, but at the expense of greater memory usage. This paper proposes a novel key-value store called PHash (Packed Hash) based on an advanced design of index and data structures that ensures both high performance and small memory usage. These advantages make the proposed scheme a better fit for processing demanding workloads in large-scale data-intensive applications. Compared to the best-performing competitor, FAWN-DS, the proposed scheme significantly reduces the memory consumption (bytes per key-value) by 83% and improves the GET throughput by up to 27.3% while the PUT throughput decreases by 12.6%. In particular, the GET performance of the proposed scheme reaches up to 99.4% of the optimal performance of the raw SSD (Solid State Drive).