Large-scale evaluation framework for local influence theories in Twitter

• We propose a formalized evaluation framework for topic-specific influence theories specialized in Twitter.
• We introduce a novel evaluation metric that assesses the aggregate sentiment of a group of user.
• We introduce a two-dimensional taxonomy for classifying influence theories.
• We use the framework to evaluate five existing theories from literature and assess their performance.

Influence theories constitute formal models that identify those individuals that are able to affect and guide their peers through their activity. There is a large body of work on developing such theories, as they have important applications in viral marketing, recommendations, as well as information retrieval. Influence theories are typically evaluated through a manual process that cannot scale to data voluminous enough to draw safe, representative conclusions. To overcome this issue, we introduce in this paper a formalized framework for large-scale, automatic evaluation of topic-specific influence theories that are specialized in Twitter. Basically, it consists of five conjunctive conditions that are indicative of real influence exertion: the first three determine which influence theories are compatible with our framework, while the other two estimate their relative effectiveness. At the core of these two conditions lies a novel metric that assesses the aggregate sentiment of a group of users and allows for estimating how close the behavior of influencers is to that of the entire community. We put our framework into practice using a large-scale test-bed with real data from 75 Twitter communities. In order to select the theories that can be employed in our analysis, we introduce a generic, two-dimensional taxonomy that elucidates their functionality. With its help, we ended up with five established topic-specific theories that are applicable to our settings. The outcomes of our analysis reveal significant differences in their performance. To explain them, we introduce a novel methodology for delving into the internal dynamics of the groups of influencers they define. We use it to analyze the implications of the selected theories and, based on the resulting evidence, we propose a novel partition of influence theories in three major categories with divergent performance.