Grassmann manifold for nearest points image set classification

•Modeling one image set as multiple overlapped local patches could capture more information.•Modeling local patches as subspaces can accommodate complex data variations.•Subspaces on Grassmann manifold can be mapped to a higher dimensional Hilbert space.•Nearest points on mapped Hilbert space can be found by kernel affine hull model.•Affine combinations of subspaces mapped on Hilbert space are beneficial for classification.

Image set classification has attracted increasing attention in recent years. How to effectively represent image sets is one key issue of set based classification. Subspaces form non-Euclidean Riemannian manifolds known as Grassmann manifolds, which allows an image set to be conveniently represented as a point on a Grassmann manifold is widely used in many visual classification tasks. Another issue is how to measure the distance/similarity between sets. Modeling image sets as hulls, and then finding distance of nearest points between sets as the set-to-set distance is a popular solution recently. In this paper, we propose a novel approach by exploiting the Projection kernel that explicitly maps the subspaces from the Grassmann manifold to a Reproducing Kernel Hilbert Space (RKHS) where the Euclidean geometry applies. And then, by modeling the points on RKHS as affine hulls, the Euclidean distance between the nearest points of two hulls can be used for classification. In order to obtain enough points for building the Grassmann affine hulls, we also develop a subspaces constructing method extended by K-means. Experiments are conducted on six datasets. Our proposed method achieves the best classification results on two multi-view object categorization datasets and one extreme illumination variation face recognition dataset.