Structural similarity for document image classification and retrieval

•We propose to capture structural similarity at different levels of abstractions.•SURF based spatial features achieve better accuracies with fewer codewords.•We show the effectiveness of retrieval with a limited number of training images.•We show in-class discrimination results on 53 classes of table images.•Our approach is effective when computational resources are limited.

This paper presents a novel approach to defining document image structural similarity for the applications of classification and retrieval. We first build a codebook of SURF descriptors extracted from a set of representative training images. We then encode each document and model the spatial relationships between them by recursively partitioning the image and computing histograms of codewords in each partition. A random forest classifier is trained with the resulting features, and used for classification and retrieval. We demonstrate the effectiveness of our approach on table and tax form retrieval, and show that the proposed method outperforms previous approaches even when the training data is limited.