Distinctive characteristics of a metric using deviations from Poisson for feature selection

In the previous paper (Ogura, H., Amano, H., & Kondo, M. (2009). Feature selection with a measure of deviations from Poisson in text categorization. Expert Systems with Applications, 36  , 6826–6832.), we proposed a new metric, χP2, for selecting features in text classification which estimates term importance based on how largely the probability distribution of a considered term deviates from the standard Poisson distribution. In this study, to establish the validity and advantage of using χP2, we conducted experiments of automatic text classification on 20 NewsGroups data collection with binary setting. In the experiments, other three metrics for feature selection, i.e., Gini index, χ2χ2 statistic and information gain, were also used for comparison. From the results, it was confirmed that χP2 and Gini index are much better than χ2χ2 statistic and information gain in terms of F1F1 performance when they handle imbalanced data set. Furthermore, through another experiment in which the degree of imbalance in class distribution was explicitly controlled, we clarified that the origin of the superiority of χP2 and Gini index is their ability to pick up suitable negative features in imbalanced data set. The ability of these two metrics to select suitable negative features is explained based on the analysis of their limiting behaviors at some extreme cases.