Support image set machine: Jointly learning representation and classifier for image set classification

• We propose a joint method for image set representation and classification.
• Learning parameters of representation and classifier are unified in one objective.
• Learning of image set representation and classifier are regularized by each other.
• Our method reduces the time complexity of classifying set image set significantly.

In this paper, we investigate the problem of classifying an image set of an object, and develop a novel image set representation and classification algorithm. We propose to represent an image set by a joint representation method using both an affine hull of its image samples and a combination of its reference images, and further classify it by a linear classification function from its representation. A unified objective function is formulated to learn both the representation and classifier parameters. Similar to support vector machine, the hinge losses and the squared ℓ2ℓ2 norm of the image set classifier are minimized simultaneously in the objective. Moreover, the differences between the two different representations are also minimized. The objective function is optimized with respect to representation and classifier parameters alternately in an iterative algorithm. The proposed algorithm is named as support image set machine (SupISMac) because it takes advantage of support vector machine formulation to learn an image set classifier. The experiments on two different image set classification benchmark databases show that SupISMac not only outperforms the state-of-the-art image set classification methods, but also reduces the running time of test procedures significantly.