Image-level classification by hierarchical structure learning with visual and semantic similarities

- We jointly model the visual and semantic information of images for classification.
- A hierarchical structure is automatically constructed to represent images discriminatively.
- the proposed method is independent of image representations which generalizes well.

Image classification methods often use class-level information without considering the distinctive character of each image. Images of the same class may have varied appearances. Besides, visually similar images may not be semantically correlated. To solve these problems, in this paper, we propose a novel image classification method by automatically learning the image-level hierarchical structure (ILHS) using both visual and semantic similarities. We try to generate new representations by exploring both visual and semantic similarities of images. Images are clustered hierarchically to explore their correlations. We then use them for image representations. The diversity of image classes within each cluster is used to re-weight visual similarities. The re-weighted similarities are aggregated to generate new image representations. We conduct image classification experiments on the Caltech-256 dataset, the PASCAL VOC 2007 dataset and the PASCAL VOC 2012 dataset. Experimental results demonstrate the effectiveness of the proposed method.