An empirical comparison of botnet detection methods

•A comparison of three botnet detection methods using a real dataset.•A new, large and public dataset with background, normal and botnet labels.•A new performance metric for comparing botnet detection methods in real networks.•An analysis and insight view of the impact of botnet activities on the methods.•Each method is best for different botnet phases. The keys: a dataset and a metric.

The results of botnet detection methods are usually presented without any comparison. Although it is generally accepted that more comparisons with third-party methods may help to improve the area, few papers could do it. Among the factors that prevent a comparison are the difficulties to share a dataset, the lack of a good dataset, the absence of a proper description of the methods and the lack of a comparison methodology. This paper compares the output of three different botnet detection methods by executing them over a new, real, labeled and large botnet dataset. This dataset includes botnet, normal and background traffic. The results of our two methods (BClus and CAMNEP) and BotHunter were compared using a methodology and a novel error metric designed for botnet detections methods. We conclude that comparing methods indeed helps to better estimate how good the methods are, to improve the algorithms, to build better datasets and to build a comparison methodology.