L2SSP: Robust keypoint description using local second-order statistics with soft-pooling

In recent years, local image descriptors based on histograms of oriented gradients (e.g., SIFT, DAISY) and intensity orders (e.g., LBP, LIOP) have been popular for the keypoint matching. However, by relying on the dominant orientation estimation or several pixels based interaction to achieve rotation invariance, these descriptors tend to be error-prone or noise-sensitive. Moreover, they represent features as histograms which are restricted to the low-order statistics. In this paper, we propose to use local second-order statistics with soft-pooling (L2SSP) for robust keypoint description. To this end, a feature set is first designed by modeling each pixel as local spatial-frequency patterns and local extremum patterns. Such a feature set is rotationally invariant, highly discriminative and also robust to noise. Then, a soft spatial binning is introduced to encode the gradient information in a rotation invariant way. Finally, the descriptor is constructed by concatenating all sub-descriptors which are obtained by pooling local features within each spatial bin via the second-order statistics (covariance matrix). The proposed local descriptor, i.e., L2SSP, has been extensively evaluated on the Oxford dataset and some synthesized image pairs. Experimental results demonstrate the superior performance of L2SSP over state-of-the-art methods under a variety of image transformations. The source code of L2SSP is publicly available at https://pan.baidu.com/s/1pLIVwLH#path=/%252Fl2ssp.