A statistical design approach to unsupervised codeword selection in image retrieval

Recently, the bag-of-features (BoF) representation has attracted substantial interest in large scale image retrieval. In BoF representation, images are treated as collections of local invariant descriptors. A visual codebook is constructed by quantizing the local descriptors into clusters, whose centers are called codewords. Then each image could be represented by the frequency histogram over the codebook, and existing text retrieval techniques such as inverted indexing could be applied to image retrieval. It has been recognized that the quality of the codebook is critical to the performance of BoF-based image retrieval systems, and codeword selection is therefore a fundamental problem. Many of the existing approaches for codeword selection make use of the label information of the images. However, collecting the labels of a large amount of images is very expensive. In this paper, we investigate the problem of codeword selection in the absence of labels. Inspired from the techniques of statistical design, we propose two novel unsupervised learning algorithms to select the codewords for an image retrieval system. Specifically, we assume that the relationship between the relevance score and the BoF representation of an image could be expressed by a linear regression model, and select the codewords which can improve the regression model the most. In other words, if the selected codewords are used to train the regression model, the expected prediction error can be minimized. Extensive experimental results have demonstrated the effectiveness of our proposed methods.