Local residual similarity for image re-ranking

Similarity measurement is an essential component in image retrieval systems. While previous work is focused on generic distance estimation, this paper investigates the problem of similarity estimation within a local neighborhood defined in the original feature space. Specifically, our method is characterized in two aspects, i.e., “local” and “residual”. First of all, we focus on a subset of the top-ranked relevant images to a query, with which anchors are discovered by methods such as averaging or clustering. The anchors are then subtracted from the neighborhood features, resulting in residual representations. The proposed Local Residual Similarity (LRS) homogenizes the feature distances within the local neighborhood. Effective and efficient image re-ranking is achieved by calculating LRS between the query and the top-ranked images. The method constrains that relevant images should appear similar in both original and local residual feature space. We evaluate the proposed method on two image retrieval benchmarks with global CNN representations, demonstrating a consistent improvement on performance with very limited extra computational cost.