A-GHSOM: An adaptive growing hierarchical self organizing map for network anomaly detection

The growing hierarchical self organizing map (GHSOM) has been shown to be an effective technique to facilitate anomaly detection. However, existing approaches based on GHSOM are not able to adapt online to the ever-changing anomaly detection. This results in low accuracy in identifying intrusions, particularly “unknown” attacks. In this paper, we propose an adaptive GHSOM based approach (A-GHSOM) to network anomaly detection. It consists of four significant enhancements: enhanced threshold-based training, dynamic input normalization, feedback-based quantization error threshold adaptation, and prediction confidence filtering and forwarding. We first evaluate the A-GHSOM approach for intrusion detection using the KDD’99 dataset. Extensive experimental results demonstrate that compared with eight representative intrusion detection approaches, A-GHSOM achieves significant overall accuracy improvement and significant improvement in identifying “unknown” attacks while maintaining low false-positive rates. It achieves an overall accuracy of 99.63%, and 94.04% accuracy in identifying “unknown” attacks while the false positive rate is 1.8%. To avoid drawing research results and conclusions solely based on experiments with the KDD dataset, we have also built a dataset (TD-Sim) that consists of a mixture of live trace data from the Lawrence Berkeley National Laboratory and simulated traffic based on our testbed network, ensuring adequate coverage of a variety of attacks. Performance evaluation with the TD-Sim dataset shows that A-GHSOM adapts to live traffic and achieves an overall accuracy rate of 97.12% while maintaining the false positive rate of 2.6%.

► Design of a novel machine learning based approach for network anormaly detection.
► Development of online adaptation capability for network anormaly detection.
► New approach achieves significantly better detection accuracy with low false positive rate.
► New approach effectively deals with unknown attacks compared to related techniques.
► Development of a new dataset (TD-Sim) for network anormaly detection testing.