Clustering of trending topics in microblogging posts: A graph-based approach

In this paper we will propose novel method that analyze a social network represented as a net of microblogging accounts. Based on hashtags in users posts we will detect not only the most popular topics users are giving opinions about but also what are the groups of people that talk about certain topics clusters. To solve this task we propose novel hashtags filtration model and community graph generation approach which is later used by community structures detection algorithm. We validate our approach on three very large real-life (not synthetic) datasets. Each of them contains more than 107 microblogging posts with about 106 distinct hashtags. We have also examined the scalability of the model checking how it will behave while applied to the analysis of limited number of randomly chosen subsets in comparison to full dataset. Our methodology is a nicely scalable filtering method that is capable to create graphs in which communities that share common interest might be detected. The important remark is that in our datasets the threshold on minimal random sample above which we can observe very similar distribution of vertices and edges weights is 10% of original followers. The manuscript has a dedicated section about data modeling and the appendix of our article contains the example implementation of vital fragments of our methodology.