Learning descriptive visual representation for image classification and annotation

• We propose a novel method for learning descriptive visual representation.
• Our method leads to promising results in both image classification and annotation.
• Our method can readily be extended to other challenging tasks.

This paper presents a novel semantic regularized matrix factorization method for learning descriptive visual bag-of-words (BOW) representation. Although very influential in image classification, the traditional visual BOW representation has one distinct drawback. That is, for efficiency purposes, this visual representation is often generated by directly clustering the low-level visual feature vectors extracted from local keypoints or regions, without considering the high-level semantics of images. In other words, it still suffers from the semantic gap and may lead to significant performance degradation in more challenging tasks, e.g., image classification over social collections with large intra-class variations. To learn descriptive visual BOW representation for such image classification task, we develop a semantic regularized matrix factorization method by adding Laplacian regularization defined with the tags (easy to access) of social images into matrix factorization. Moreover, given that image annotation only provides the tags of training images in advance (while the tags of all social images are available), we can readily apply the proposed method to image annotation by first running a round of image annotation to predict the tags (maybe incorrect) of test images and thus obtaining the tags of all images. Experimental results show the promising performance of the proposed method.