A comprehensive study on learning to rank for content-based image retrieval

Recently, various learning to rank approaches have been proposed in the information retrieval realm, with their promising performance in general document and web page retrieval applications. Based on these achievements, in this paper, we investigate and discuss whether learning to rank approaches can be adapted to content-based image retrieval (CBIR). Given the complex structure of image representation, it is also challenging how to design visual features for learning to rank algorithms that not only scale up well, but also model various visual modalities and the spatial distributions of local features. We answer this question by introducing some scalable visual-based ranking features for learning to rank. Specifically, we firstly adopt several well performed ad hoc ranking models to generate the bag-of-visual-words-based ranking features. Besides, images are divided into different salient regions and spatial blocks, respectively, and ranking features are extracted from each region and block. Finally, image global features-based similarities are also concatenated with the existing ranking features. Extensive experiments with three state-of-the-art learning to rank algorithms are performed over four popular image retrieval databases, together with some insightful conclusions to facilitate the adaptation of learning to rank approaches to CBIR.

► The first comprehensive study of learning to rank approaches to content-based image retrieval. ► The first study of feature selection and dimension reduction for learning to rank in image retrieval. ► The first comprehensive study of visual ranking feature construction for image retrieval applications.