A decentralized approach for mining event correlations in distributed system monitoring

Nowadays, there is an increasing demand to monitor, analyze, and control large scale distributed systems. Events detected during monitoring are temporally correlated, which is helpful to resource allocation, job scheduling, and failure prediction. To discover the correlations among detected events, many existing approaches concentrate detected events into an event database and perform data mining on it. We argue that these approaches are not scalable to large scale distributed systems as monitored events grow so fast that event correlation discovering can hardly be done with the power of a single computer. In this paper, we present a decentralized approach to efficiently detect events, filter irrelative events, and discover their temporal correlations. We propose a MapReduce-based algorithm, MapReduce-Apriori, to data mining event association rules, which utilizes the computational resource of multiple dedicated nodes of the system. Experimental results show that our decentralized event correlation mining algorithm achieves nearly ideal speedup compared to centralized mining approaches.

► Extend the event correlation rule mining problem to address temporal property of detected events.
► Propose an approach to efficiently filter irrelative events locally to reduce the number of events aggregated.
► Propose a MapReduce-based algorithm to mine events correlations in a set of dedicated nodes in parallel.