Web Crawling Using Dynamic IP Address Using Single Server

Abstract: The major issue with web crawler is parallel execution of multiple requests using an individual IP address which can be easily traced by the website if multiple hits are observed from the same IP address. To overcome this, proxy servers are used which include discrete proxy IP address whenever the request is routed over the crawling server. Now multiple requests can be executed using same crawling server, to download the information from the same website without any blockage. Each request would contain one URL and one distinct proxy IP that routed to the crawling server by the request router. Now the routed request can fetch the information very fast with the parallel execution of requests by concealing identity from the tracking application. Data extracted from the website is stored into temporary database and Indexing is performed after the each URL seed receives complete response. The context of the documents collected by the crawling in the repository is extracted by the indexer using the context repository, thesaurus, repository, and documents are indexed according to their respective context.

Keywords: Crawling Servers, Discrete Proxy, Proxy Server, Web Crawler