Cross-modal social image clustering and tag cleansing

• Semantic image clustering for exploit image topics of interest.
• Kernel combination for cross-modal contexts among weekly-tagged images.
• K-way min-max cut for social image clustering.

In this paper, a cross-modal approach is developed for social image clustering and tag cleansing. First, a semantic image clustering algorithm is developed for assigning large-scale weakly-tagged social images into a large number of image topics of interest. Spam tags are detected automatically via sentiment analysis and multiple synonymous tags are merged as one super-topic according to their inter-topic semantic similarity contexts. Second, multiple base kernels are seamlessly combined by maximizing the correlations between the visual similarity contexts and the semantic similarity context, which can achieve more precise characterization of cross-modal (semantic and visual) similarity contexts among weakly-tagged social images. Finally, a K-way min–max cut algorithm is developed for social image clustering by minimizing the cumulative inter-cluster cross-modal similarity contexts while maximizing the cumulative intra-cluster cross-modal similarity contexts. The optimal weights for base kernel combination are simultaneously determined by minimizing the cumulative within-cluster variances. The polysemous tags and their ambiguous images are further split into multiple sub-topics for reducing their within-topic visual diversity. Our experiments on large-scale weakly-tagged Flickr images have provided very positive results.