Estimating evolution of freshness in Internet cache directories under the capture–recapture methodology

In this paper, we describe a new web sampling scheme for measuring the evolution of freshness in search engines. The methodology used is the capture–recapture, which is mainly applied for estimating evolution rates in wildlife biological studies. After modifications and amendments, necessary for web paradigm application, we conducted three capture–recapture experiments of different duration over the caches of Google and MSN. In parallel, we used a typical sampling scheme, similar to many other web sampling approaches used in the literature, to evaluate the robustness of our proposal. The paper provides the implementation details of a web-based capture–recapture model along with its assessment. The results show that through the capture–recapture methodology we are able not only to measure the freshness of the tested search services but also to monitor its evolution over time, with a substantially lower amount of required sampling instances. It was not our intention to compare the performance of Google and MSN. However, through our experiments, we observed that although one sometimes presents better refresh rates than the other, in general both search services have virtually equal capabilities in refreshing their directories and providing new and up-to-date results to their users.