Image automatic annotation via multi-view deep representation

•A deep neural network is used as a directed supervised classification for mapping image features to keywords.•Stacked auto-encoder is modified by sigmoid predictor and iteration algorithm.•Tag frequencies and properties should be paid attention to during tagging process.•Image sematic features will improve the image annotation performance.•Each keyword has been assigned to the appropriate visual descriptor.

The performance of text-based image retrieval is highly dependent on the tedious and inefficient manual work. For the purpose of realizing image keywords generated automatically, extensive work has been done in the area of image annotation. However, how to treat image diverse keywords and choose appropriate features are still two difficult problems. To address this challenge, we propose the multi-view stacked auto-encoder (MVSAE) framework to establish the correlations between the low-level visual features and high-level semantic information. In this paper, a new method, which incorporates the keyword frequencies and log-entropy, is presented to address the imbalanced distribution of keywords. In order to utilize the complementarities among diverse visual descriptors, we tactfully apply multi-view learning to search for the label-specific features. Thereafter, the image keywords are finally produced by appropriate features. Conducting extensive experiments on three popular data sets, we demonstrate that our proposed framework can achieve effective and favorable performance for image annotation.