A logistic regression-based smoothing method for Chinese text categorization

Automatic Chinese text classification is an important and a well-known technology in the field of machine learning. The first step for solving Chinese text categorization problems is to tokenize the Chinese words from a sequence of non-segmented sentences. However, previous literatures often employ a Chinese word tokenizer that was trained with different sources and then perform the conventional text classification approaches. However, these taggers are not perfect and often provide incorrect word boundary information. In this paper, we propose an N-gram-based language model which takes word relations into account for Chinese text categorization without Chinese word tokenizer. To prevent from out-of-vocabulary, we also propose a novel smoothing approach based on logistic regression to improve accuracy. The experimental result shows that our approach outperforms traditional methods at least 11% on micro-average F-measure.

Research highlights► An N-gram Language model is selected for Chinese text categorization. ► A novel smoothing method based on logistic regression is proposed. ► The chi-square value is used to exam the importance of N-gram for feature selection. ► Our approach outperforms traditional methods at least 11% on micro-average F-measure.