Storage-optimizing clustering algorithms for high-dimensional tick data

• We investigate tick data for applications that keep track of values changing rapidly over time, like stock market prices.
• We decompose tick data by clustering its attributes using Storage-Optimizing Hierarchical Agglomerative Clustering.
• We propose a method for speeding up the algorithm based on a new lower bounding technique.
• Experimental results show that the proposed approach compares favorably to several baselines in terms of compression.
• Additionally, the proposed approach can lead to significant speedup in terms of running time.

Tick data are used in several applications that need to keep track of values changing over time, like prices on the stock market or meteorological measurements. Due to the possibly very frequent changes, the size of tick data tends to increase rapidly. Therefore, it becomes of paramount importance to reduce the storage space of tick data while, at the same time, allowing queries to be executed efficiently. In this paper, we propose an approach to decompose the original tick data matrix by clustering their attributes using a new clustering algorithm called Storage-Optimizing Hierarchical Agglomerative Clustering (SOHAC). We additionally propose a method for speeding up SOHAC based on a new lower bounding technique that allows SOHAC to be applied to high-dimensional tick data. Our experimental evaluation shows that the proposed approach compares favorably to several baselines in terms of compression. Additionally, it can lead to significant speedup in terms of running time.