Beyond visual word ambiguity: Weighted local feature encoding with governing region

•We propose a novel image classification method based on visual word’s governing regions.•We weight each local feature based on its discriminative power per classification task.•Weighted local features are encoded with visual words’ governing regions.•A weighted feature sign search algorithm is proposed to encode weighted local features.

Typically, k-means clustering or sparse coding is used for codebook generation in the bag-of-visual words (BoW) model. Local features are then encoded by calculating their similarities with visual words. However, some useful information is lost during this process. To make use of this information, in this paper, we propose a novel image representation method by going one step beyond visual word ambiguity and consider the governing regions of visual words. For each visual application, the weights of local features are determined by the corresponding visual application classifiers. Each weighted local feature is then encoded not only by considering its similarities with visual words, but also by visual words’ governing regions. Besides, locality constraint is also imposed for efficient encoding. A weighted feature sign search algorithm is proposed to solve the problem. We conduct image classification experiments on several public datasets to demonstrate the effectiveness of the proposed method.