Mathematical aggregation operators in image retrieval: effect on retrieval performance and role in relevance feedback

We examine the effect of mathematical aggregation operators on the image retrieval performance, by empirically comparing 67 operators, applied to the problem of computing the overall image similarity, given a collection of individual feature similarities. While most of the existing image similarity models express the overall image similarity as an aggregation of multiple feature similarities, no study presents a comprehensive comparison of the different operators. For the comparison, we use a diverse test collection with around 2500 images in 62 semantic categories. Results show that the retrieval performance strongly depends on the mathematical aggregation operator(s) employed within the image similarity model-the difference in the average retrieval precision between the best performing and the worst performing of the 67 operators is over 40%. Based on this observation, we propose a genetic algorithm-based relevance feedback technique-called Local Aggregation Pattern (LAP)-which adapts the image similarity model to the user by modifying the combination of aggregation operators employed within the model to aggregate multiple feature similarities into the overall image similarity. Evaluated on the 2500 images test collection, the proposed LAP technique is shown to outperform the existing relevance feedback techniques-by over 5% higher average retrieval precision. Furthermore, by modifying the combination of aggregation operators rather than the relevance of image features, the proposed LAP technique is complementary to the majority of the existing relevance feedback techniques, with which it can be naturally coupled to further improve the image retrieval performance.