Web image concept annotation with better understanding of tags and visual features

This paper focuses on improving the semi-manual method for web image concept annotation. By sufficiently studying the characteristics of tag and visual feature, we propose the Grouping-Based-Precision & Recall-Aided (GBPRA) feature selection strategy for concept annotation. Specifically, for visual features, we construct a more robust middle level feature by concatenating the k-NN results for each type of visual feature. For tag, we construct a concept-tag co-occurrence matrix, based on which the probability of an image belonging to certain concept can be calculated. By understanding the tags’ quality and groupings’ semantic depth, we propose a grouping based feature selection method; by studying the tags’ distribution, we adopt Precision and Recall as a complementary indicator for feature selection. In this way, the advantages of both tags and visual features are boosted. Experimental results show our method can achieve very high Average Precision, which greatly facilitates the annotation of large-scale web image dataset.

Research highlights
► A K-Nearest Neighbor and multiple feature based mid-level feature is proposed.
► A concept-tag co-occurrence matrix is proposed for tag based concept prediction.
► We propose a grouping based and tag’s distribution aided feature selection method for web image annotation.