Article ID Journal Published Year Pages File Type
10481954 Physica A: Statistical Mechanics and its Applications 2013 9 Pages PDF
Abstract
This paper proposes a new metric to evaluate and rank the relevance of words in a text. The method uses the Shannon's entropy difference between the intrinsic and extrinsic mode, which refers to the fact that relevant words significantly reflect the author's writing intention, i.e., their occurrences are modulated by the author's purpose, while the irrelevant words are distributed randomly in the text. By using The Origin of Species by Charles Darwin as a representative text sample, the performance of our detector is demonstrated and compared to previous proposals. Since a reference text “corpus” is all of an author's writings, books, papers, etc. his collected works is not needed. Our approach is especially suitable for single documents of which there is no a priori information available.
Related Topics
Physical Sciences and Engineering Mathematics Mathematical Physics
Authors
, , , ,