Decentralized multi-dimensional alert correlation for collaborative intrusion detection

The growth in coordinated network attacks such as scans, worms and distributed denial-of-service (DDoS) attacks is a profound threat to the security of the Internet. Collaborative intrusion detection systems (CIDSs) have the potential to detect these attacks, by enabling all the participating intrusion detection systems (IDSs) to share suspicious intelligence with each other to form a global view of the current security threats. Current correlation algorithms in CIDSs are either too simple to capture the important characteristics of attacks, or too computationally expensive to detect attacks in a timely manner. We propose a decentralized, multi-dimensional alert correlation algorithm for CIDSs to address these challenges. A multi-dimensional alert clustering algorithm is used to extract the significant intrusion patterns from raw intrusion alerts. A two-stage correlation algorithm is used, which first clusters alerts locally at each IDS, before reporting significant alert patterns to a global correlation stage. We introduce a probabilistic approach to decide when a pattern at the local stage is sufficiently significant to warrant correlation at the global stage. We then implement the proposed two-stage correlation algorithm in a fully distributed CIDS. Our experiments on a large real-world intrusion data set show that our approach can achieve a significant reduction in the number of alert messages generated by the local correlation stage with negligible false negatives compared to a centralized scheme. The proposed probabilistic threshold approach gains a significant improvement in detection accuracy in a stealthy attack scenario, compared to a naive scheme that uses the same threshold at the local and global stages. A large scale experiment on PlanetLab shows that our decentralized architecture is significantly more efficient than a centralized approach in terms of the time required to correlate alerts.