Cluster validity measure and merging system for hierarchical clustering considering outliers

•Cluster validity measure for arbitrary shaped clusters with outliers.•Cluster merging system grouping cluster cores based on the outliers׳ structure.•Truly hierarchical variants of support vector and Gaussian process clustering.•Benefits for unsupervised change detection applications are presented.

Clustering algorithms have evolved to handle more and more complex structures. However, the measures that allow to qualify the quality of such clustering partitions are rare and have been developed only for specific algorithms. In this work, we propose a new cluster validity measure (CVM) to quantify the clustering performance of hierarchical algorithms that handle overlapping clusters of any shape and in the presence of outliers. This work also introduces a cluster merging system (CMS) to group clusters that share outliers. When located in regions of cluster overlap, these outliers may be issued by a mixture of nearby cores. The proposed CVM and CMS are applied to hierarchical extensions of the Support Vector and Gaussian Process Clustering algorithms both in synthetic and real experiments. These results show that the proposed metrics help to select the appropriate level of hierarchy and the appropriate hyperparameters.