A new shape prior model with rotation invariance

•The shape priori with rotation invariance of arbitrary angle is proposed.•Using BP algorithm, the initial localization is not necessary for our model.•We generalize the classical distance transforms to calculate the beliefs efficiently.•Our model is capable of detection under both rotation and non-rotation situations.

Object detection methods based on keypoint localization are widely used, and the rotation invariance is one of the fundamental issues to consider. This paper proposes a novel shape prior model with rotation invariance. The proposed shape prior model discards all orientation-involved features and only uses the distance features among keypoints, hence it is competent to detect objects with a rotation of the arbitrary angle when combined with local appearance description with rotation invariance. In the stage of detection, belief propagation algorithm is employed, so that our method no longer needs the initial position of the keypoints. Furthermore, we generalize the classical distance transforms, the generalized distance transforms make the beliefs to be calculated in a nearly linear time. Experiments were carried out on face category and touring-bike category in the Caltech-256 database. The results demonstrated that the proposed method achieved a strong robustness of rotation.