Maintaining the search engine freshness using mobile agent

Search engines must keep an up-to-date image to all Web pages and other web resources hosted in web servers in their index and data repositories, to provide better and accurate results to its clients. The crawlers of these search engines have to retrieve the pages continuously to keep the index up-to-date. It is reported in the literature that 40% of the current Internet traffic and bandwidth consumption is due to these crawlers. So we are interested in detecting the significant changes in web pages which reflect effectively in search engine’s index and minimize the network load. In this paper, we suggest a document index based change detection technique and distributed indexing using mobile agents. The experimental results have shown that the proposed system can considerably reduce the network traffic and the computational load on the search engine side and keep its index up-to-date with significant changes.