Article ID | Journal | Published Year | Pages | File Type |
---|---|---|---|---|
527802 | Computer Vision and Image Understanding | 2013 | 17 Pages |
We present an image retrieval system for the interactive search of photo collections using free-hand sketches depicting shape. We describe Gradient Field HOG (GF-HOG); an adapted form of the HOG descriptor suitable for Sketch Based Image Retrieval (SBIR). We incorporate GF-HOG into a Bag of Visual Words (BoVW) retrieval framework, and demonstrate how this combination may be harnessed both for robust SBIR, and for localizing sketched objects within an image. We evaluate over a large Flickr sourced dataset comprising 33 shape categories, using queries from 10 non-expert sketchers. We compare GF-HOG against state-of-the-art descriptors with common distance measures and language models for image retrieval, and explore how affine deformation of the sketch impacts search performance. GF-HOG is shown to consistently outperform retrieval versus SIFT, multi-resolution HOG, Self Similarity, Shape Context and Structure Tensor. Further, we incorporate semantic keywords into our GF-HOG system to enable the use of annotated sketches for image search. A novel graph-based measure of semantic similarity is proposed and two applications explored: semantic sketch based image retrieval and a semantic photo montage.
► Comprehensive evaluation of new descriptor GF-HOG for Sketch Based Image Retrieval (SBIR). ► Compares accuracy, speed and affine invariance to six state of the art SBIR descriptors using several distance measures. ► New FlickR source annotated image dataset for SBIR. ► Fuses GF-HOG (shape based) retrieval with text keywords for semantic SBIR. ► All source code and data to be released upon publication.