Article ID | Journal | Published Year | Pages | File Type |
---|---|---|---|---|
10481954 | Physica A: Statistical Mechanics and its Applications | 2013 | 9 Pages |
Abstract
This paper proposes a new metric to evaluate and rank the relevance of words in a text. The method uses the Shannon's entropy difference between the intrinsic and extrinsic mode, which refers to the fact that relevant words significantly reflect the author's writing intention, i.e., their occurrences are modulated by the author's purpose, while the irrelevant words are distributed randomly in the text. By using The Origin of Species by Charles Darwin as a representative text sample, the performance of our detector is demonstrated and compared to previous proposals. Since a reference text “corpus” is all of an author's writings, books, papers, etc. his collected works is not needed. Our approach is especially suitable for single documents of which there is no a priori information available.
Keywords
Related Topics
Physical Sciences and Engineering
Mathematics
Mathematical Physics
Authors
Zhen Yang, Jianjun Lei, Kefeng Fan, Yingxu Lai,