Using local density information to improve IB algorithms

The Information Bottleneck principle provides a systematic method to extract relevant features from complex data sets, and it models features extraction as data compression and quantifies the relevance of extracted feature by how much information it preserved about a specified feature. How to construct an optimal solution to IB remains a problem. The current Information Bottleneck (IB) algorithms only utilize the information between element pairs, and ignore the information among the neighborhood of elements. This is one of the major reasons for most IB algorithms’ failure to preserve as much relative information as possible, which further limits IB applicability in many areas. In this paper, we present the concept of density connectivity component, by which the information loss among the neighbors of an element, rather than the information loss between paired elements, can be considered. Then, we introduce this concept into the current agglomerative IB algorithm (aIB) and sequential IB algorithm (sIB), and propose two density-based IB algorithms, DaIB and DsIB. The experiment results on the benchmark data sets indicate that the DaIB and DsIB algorithm can preserve more relevant information and achieve higher precision than the aIB and sIB algorithm, respectively.

Research highlights
► An improved solution to Information Bottleneck.
► Density Connectivity Component which considers the information loss in neighborhood.
► Improved IB algorithms by incorporating Density Connectivity Component.