Unsupervised learning clustering and self-organized agents applied to help network management

• Self-organized agents use multidimensional flow analysis to help network management.
• Traffic profiling and anomaly detection tasks are designed to operate autonomously.
• Reports are provided in real time to aid decision-making when anomalous events occur.
• A pattern matching technique calculates adaptive thresholds for anomaly detection.
• False alarm and accuracy rates are encouraging both in real and simulated traffic.

Traffic monitoring and anomaly detection are essential activities for computer network management, since they provide relevant information about its current performance and contribute to network control. Although there are several studies in this area, diagnosis and resolution of anomalies are still challenging issues. From an expert system point of view, current solutions have not been sufficient to meet the requirements demanded for use in large-scale network environments, and thus a significant portion of budgets on the workforce are spent to network management. Based on this context, the focus of this paper consists of the development of a system able to proactively monitor the network and detect anomalous events, reducing manual intervention and the probability of errors in decision-making, regarding network management. The proposed approach characterizes the normal pattern of the network traffic and detects anomalous behavior, outage events and attacks by deviations from this pattern. For this purpose, an unsupervised learning methodology is used to extract features of traffic through IP flows attributes, collected from a network structure. Aiming to improve its efficiency, a modification of the Ant Colony Optimization metaheuristic is proposed, which through self-organized agents optimizes the analysis of multidimensional flows attributes and allows it to be completed in time to mitigate the impact on large-scale networks. In addition to notify the network manager about the anomalies, the system provides necessary information to identify and take action against them. The resulting detection system was tested with real and simulated data, achieving high detection rates while the false alarm rate remains low.