Memory Management for Big Data Mining – Cache Hit Rate Estimation of LessFU

We have developed a network monitor which can find IP packets sent by Internet Virus from Internet backbone traffic. A data mining engine which can handle 10 M transactions per second is the main component of the monitor. Although the data mining engine have to analyze over 200G byte data in theory, a memory management strategy named LessFU removes non-essential data to realize efficient processing. Our past experiments which use real Internet traffic shows the advantage of our approach. However, there exits no method to evaluate the cache hit rate of LessFU. Since the cache hit rate results in serious consequences on the data mining results, this paper proposes a method to estimate the cache hit rate of LessFU. The experimental results which show the advantage of the proposed method are also reported in this paper.