Joint intermodal and intramodal correlation preservation for semi-paired learning

Canonical correlation analysis(CCA) and its variants have been playing an important role in dimension reduction and information fusion. Most of the existing CCA-related algorithms can only handle fully-paired scenarios, where all the samples from different views require to be rigorously paired. However, majority of the samples only provide partial pairwise correspondences in most cases. In such semi-paired scenarios, traditional methods often perform poorly due to overfitting. Furthermore, most CCA-related algorithms only focus on the correlation between different views but ignore preserving the within-view similarity which is helpful to make samples in the same class mapped more compactly onto canonical correlation space. In addition, since CCA is unable to exploit discriminative information effectively, improved performance is limited in many tasks. To solve these challenging problems, we propose a novel method named joint Intermodal and Intramodal Semi-paired Canonical Correlation Analysis (I2SCCA), which properly handle the semi-paired scenarios. At the same time, the method preserves the within-view similarity information by making two neighbor samples in the same view closer in the projected space. Different from traditional similarity matrix construction, we incorporate a rank constraint of Laplacian matrix into the graph-based clustering algorithms. With fewer parameters, a graph that has exactly c (the number of clusters) connected components is learned without requiring any post graph-cut processing. Based on the obtained graph, we estimate the cross-view correlation through similarity measure of the single view. Specially, taking advantage of clustering assumption, discriminative information is derived although there is no label available. Finally, we extend I2SCCA to multi-view and non-linear cases. Extensive experiments on several datasets demonstrate the effectiveness of I2SCCA.