| Article ID | Journal | Published Year | Pages | File Type |
|---|---|---|---|---|
| 385962 | Expert Systems with Applications | 2011 | 6 Pages |
In this paper, we propose using chi-square statistics to measure similarities and chi-square tests to determine the homogeneity of two random samples of term vectors for text categorization. The properties of chi-square tests for text categorization are studied first. One of the advantages of chi-square test is that its significance level is similar to the miss rate that provides a foundation for theoretical performance (i.e. miss rate) guarantee. Generally a classifier using cosine similarities with TF ∗ IDF performs reasonably well in text categorization. However, its performance may fluctuate even near the optimal threshold value. To improve the limitation, we propose the combined usage of chi-square statistics and cosine similarities. Extensive experiment results verify properties of chi-square tests and performance of the combined usage.
Research highlights► For a text categorization task, chi-square statistics can be used to measure dissimilarities and chi-square tests can be used as classifiers. ► The significance level of a chi-square test and the miss rate for the corresponding text categorization task are completely positive correlated. ► A chi-square test can determine the homogeneity of two random samples of original TF vectors without difficulty. ► A classifier using both cosine similarities with TF ∗ IDF and chi-square statistics as its similarity measures performs in par or better than one using only cosine similarity with TF ∗ IDF in F1.
