Efficient multi-modal fusion on supergraph for scalable image annotation

• We construct a supergraph to structurally combine various types of visual features.
• The main challenge of learning on supergraph is its large computational time.
• To reach scalability, we conduct learning on a small prototype graph in supergraph.
• Prototype graph is a good replacement for sample graph during label propagation.
• We achieve good performance by reconstructing labels of images from prototypes.

Different types of visual features provide multi-modal representation for images in the annotation task. Conventional graph-based image annotation methods integrate various features into a single descriptor and consider one node for each descriptor on the learning graph. However, this graph does not capture the information of individual features, making it unsuitable for propagating the labels of annotated images. In this paper, we address this issue by proposing an approach for fusing the visual features such that a specific subgraph is constructed for each visual modality and then subgraphs are connected to form a supergraph. As the size of supergraph grows linearly with the number of visual features, it is essential to handle large computational complexity of label propagation on the supergraph. To this end, we extract some prototypes from the feature vectors of images and incorporate them into the supergraph construction. The learning process is then conducted on the prototypes, instead of a large number of feature vectors, making the label inference scalable. The experiments on a wide range of standard datasets reveal that the proposed approach achieves scalable image annotation while having an acceptable level of performance.