Utilizing advances in correlation analysis for community structure detection

In the era of big data, some data records are interrelated with each other in many areas, such as marketing, management, health care, and education. These interrelated data can be more naturally represented as networks with nodes and edges. Inside this type of networks, there is usually a hidden community structure where each community represents a relatively independent functional module. Such hidden community structures are useful for many applications, such as word-of-mouth marketing, promoting decentralized social interactions inside organizations, and searching biological pathways related to various diseases. Therefore, how to detect hidden community structures becomes an important task with wide applications. Currently, modularity-based methods are widely-used among many existing community structure detection methods. They detect communities with more internal edges than expected under the null hypothesis of independence. Since research in correlation analysis also searches for patterns which occur more than expected under the null hypothesis of independence, this paper proposed a framework of changing the original modularity function according to different existing correlation functions in the correlation analysis research area. Such a framework can utilize not only the current but also the future potential research progresses in correlation analysis to advance community detection. In addition, a novel graphical analysis on different modified-modularity functions is conducted to analyze their different preferences, which are also validated by our evaluation on both real life and simulated networks. Our work to connect modularity-based methods with correlation analysis has several significant impacts on the community detection research and its applications to expert and intelligent systems. First, the research progress in correlation analysis can be utilized to define a more effective objective function in community detection for better detection results since different real-life applications might need communities with different resolutions. Second, any existing research progress for the modularity function, such as the Louvain method for speeding up the search and different extensions for overlapping community detection, can be applied in a similar way to the new objective function derived from existing correlation functions, because the new objective function is unified within one framework with the modularity function. Third, our framework opens a large unexplored area for the researchers interested in community detection. For example, what is the best heuristic search method for each different objective function? What are the characteristics of each objective function when applied to overlapping community detection? Among different extensions to overlapping community detection, which extension is better for each objective function?