Network anomaly detection with incomplete audit data

With the ever increasing deployment and usage of gigabit networks, traditional network anomaly detection based Intrusion Detection Systems (IDS) have not scaled accordingly. Most, if not all IDS assume the availability of complete and clean audit data. We contend that this assumption is not valid. Factors like noise, mobility of the nodes and the large amount of network traffic make it difficult to build a traffic profile of the network that is complete and immaculate for the purpose of anomaly detection. In this paper, we attempt to address these issues by presenting an anomaly detection scheme, called SCAN (Stochastic Clustering Algorithm for Network Anomaly Detection), that has the capability to detect intrusions with high accuracy even with incomplete audit data. To address the threats posed by network-based denial-of-service attacks in high speed networks, SCAN consists of two modules: an anomaly detection module that is at the core of the design and an adaptive packet sampling scheme that intelligently samples packets to aid the anomaly detection module. The noteworthy features of SCAN include: (a) it intelligently samples the incoming network traffic to decrease the amount of audit data being sampled while retaining the intrinsic characteristics of the network traffic itself; (b) it computes the missing elements of the sampled audit data by utilizing an improved expectation–maximization (EM) algorithm-based clustering algorithm; and (c) it improves the speed of convergence of the clustering process by employing Bloom filters and data summaries.